Let's Encrypt DNS-01 Failure Traceback - LXC TK Core 18.0-1

[zndrr]

Hi Team,

Have been attempting to retrieve a cert via LE in confconsole and met with the following traceback.
Copied from confconsole UI, please excuse malformatting.

Hopefully I provide enough info and don't require too much back-and-forth:

Steps: confconsole > Lets encrypt > Get certificate > Accept TOS > dns-01

Traceback (most recent call last):
  File "/usr/bin/confconsole", line 719, in loop
    new_dialog = method()
                 ^^^^^^^^
  File "/usr/lib/confconsole/plugin.py", line 121, in run
    ret: Optional[str] = self.module.run()  
                         ^^^^^^^^^^^^^^^^^  
  File "/usr/lib/confconsole/plugins.d/Lets_Encrypt/get_certificate.py", line 153, in run  
    config = dns_01.load_config()  
             ^^^^^^^^^^^^^^^^^^  
AttributeError: 'NoneType' object has no attribute  
'load_config'

I'm kind of average at digging, but seems loosely that some configuration files might be missing ...

Particulars:

root@pve1a:~# pveversion
  pve-manager/8.0.4/d258a813cfa6b390 (running kernel: 6.2.16-12-pve)

CT Template:
  debian-12-turnkey-core_18.0-1_amd64.tar.gz

root@tkcore ~# turnkey-version
  turnkey-core-18.0-bookworm-amd64

root@tkcore ~# apt show confconsole | grep Vers
  Version: 2.1.1

- LXC IP and Domain are static on LXC. Privileged container.
- Domain is hosted on Cloudflare.
- Nameservers were Gateway and 1.1.1.1, but also tried only the latter and 8.8.8.8; resolution was fine from the start however.
- Have retrieved many certs via other methods (PVE, opnsense, certbot, Traefik etc etc)
- Tried reinstalling confconsole via apt as well as manually installing latest deb from GH (though it's older).
- Rebuilt again from template.

If any other info is wanted, please let me know.

Attached the apt output (probably not useful) and the files mentioned in the trace:
apt_show_confconsole.txt
trace_files.zip

[zndrr]

HTTP-01 method allows specifying domains and does go through some motions. Definitely progresses further.
Naturally this method isn't suitable and predictably fails.

Same outcome on the Bookstack 18.0-1 template with DNS-01; traceback is identical

[JedMeister]

Thanks so much for your excellent bug report.

I'll have to dig in and have a look. FWIW I haven't explicitly tested in an LXC container but AFAIK it should "just work", obviously that's not the case...

I've got a bit of a backlog of things that I need to follow up on, so it might be a day or 2 before I circle back to this. Assuming that I can reproduce the issue, I'll aim to at least post a workaround, if not a proper fixed package.

[zndrr]

@JedMeister thank you for the response. I'm happy to wait; not urgent.

I just tested TKL Core ISO in an attempt to help out.

Same traceback I'm afraid. Same hangups as the two LXC templates.

Hope that helps.

[JedMeister]

Thanks for your followup. On the plus side, I can reproduce the issue (simply trying to use DNS-01 gives me that same issue). I haven't yet had a closer look but I already have some ideas (I suspect it's as simple as the default conf file not being copied on install).

So hopefully I'll have a work around for you really soon and a proper fix (which I'll upload to the apt archive) soon after.

[OnGle]

Seems like it was caused by bit of an oversight on my behalf both on the plugin docs and implementation which lead to undefined behavior with this particular plugin (depending on the order in which plugins were loaded)

[JedMeister]

Ok @zndrr - apologies that this has taken a few days to get some action on, but I think we should be good now...

I've now tested locally and the fix(es) applied by @OnGle appear to have done the job. Although please note that I haven't actually tested getting a cert via the DNS-01 method. As such, there is a slim chance that there are still issues (sorry I don't have creds handy to do the full set up). Regardless, I'm pretty confident that this should fix the issue.

I've just uploaded it to the bookworm TurnKey apt repo. So this should do the trick:

apt update
apt install confconsole

If you continue to have issues, please post back.

[zndrr]

Thanks heaps for the turnaround. I have taken another dive and unfortunately run in to successive issues.
There were traces, but I was quite disorganised in both method and capture.

It certainly got further of course.

I attached a few traces that I dumped to an IDE (hopefully sanitised sufficiently). Apologies that the repro steps aren't alongside.

trace1-ui.txt
trace2-ui.txt
trace3-ui-with-cli-inputs.txt - throwing things at wall to see what sticks XD
trace4-ui.txt - domain instead of sub.domain (ruling out delegation?)
traceback_lexicon-cli.txt - For this Lexicon CLI trace, this was performing the actions noted at the top of the file; list and create. List threw a trace and then valid list, and create just a trace.

It may take me some spare time to eliminate user error... and to complicate things more, I likely won't have time to poke again until the weekend following :(

One thing in particular that wasn't super clear to me, was the input expected here.
I followed the URL and tried a few things but wasn't met with success ultimately

My general assumption was that I could use a Cloudflare API token with zone edit privilege which would create the acme challenge record for validation.

Their documentation wasn't super transparent on API global key vs Token and programmatic input assumptions; had to do some digging through Issues to clarify. Became a bit of a rabbit hole XD

Appreciate the product, so hopefully I can help resolve this in whatever manner you need.
Happy to follow direction too where time allows.

root@tkcore ~# apt show confconsole
Package: confconsole
Version: 2.1.2
<omit>

[zndrr]

Ah sorry just quick revelation RE: lexicon CLI:

• create did actually generate the record as specified; just found it in my CF dashboard
• delete on CLI also deleted it, despite the ominous traceback
• subdomain record creation/deletion also okay

Fresh trace of sub.domain TXT create/delete:
lexicon-create-delete-cli.txt

This output below the traces I'm assuming is the outcome heh

RESULT
------
True

[JedMeister]

Thanks heaps for the turnaround.

You're welcome. Although it probably would have been better if I'd fully tested it and actually ensured that the issue(s) were fixed...

I have taken another dive and unfortunately run in to successive issues.
There were traces, but I was quite disorganised in both method and capture.

No problem. What you've provided is still useful. It demonstrates the issue(s) and also suggests that we need better documentation...

I attached a few traces that I dumped to an IDE (hopefully sanitised sufficiently). Apologies that the repro steps aren't alongside.

No problem, your stacktraces are still very useful.

It may take me some spare time to eliminate user error... and to complicate things more, I likely won't have time to poke again until the weekend following :(

Considering your second post, it appears that there have been some changes in the Debian python lexicon library (bullseye vs bookworm) that we didn't acknowledge/consider. It's possibly irrelevant now, but FWIW what might tighten up the feedback loop for you (and avoid any remaining Confconsole bugs - perhaps isolate bugs elsewhere) would be to use our dehydrated-wrapper script directly. It's not in the $PATH so you'll need to call it via it's full path: /usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper. I.e.:

root@bookworm-test ~# /usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper -h

E.g.:

/usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper -r -c dns-01 -p cloudflare

Having said that, as demonstrated by your direct use of lexicon, I suspect that there would still be issues...

One thing in particular that wasn't super clear to me, was the input expected here.
I followed the URL and tried a few things but wasn't met with success ultimately

TBH, looking closer, you are absolutely correct! We definitely need to do a better job documenting this.

FWIW support for DNS-01 challenges was contributed by a user. I was pretty pumped about it (previously we only supported HTTP-01 challenges). I tested it (on TurnKey v17.x) at the time and it "just worked" so I was happy to merge. Although IIRC I only tested it in detail using AWS Route53 (which I was already using, had some understanding of and had handy).

However, following your feedback, a little testing and looking back over it all with a fresh (and perhaps more critical) eye, it's clear that I should tested more deeply on v18.0. I see a lot of shortcomings in the code, but even more so in the documentation... I don't have time or resources to rewrite the code right now, but obviously there isn't much value in providing something this broken...

My general assumption was that I could use a Cloudflare API token with zone edit privilege which would create the acme challenge record for validation.

I don't recall exactly, but my memory suggests it was essentially that easy when I tested (albeit via Route53 rather than Cloudflate). But I suspect that there was some Curse of knowledge going on and/or perhaps there are some bugs in the newer Debian package?.

Their documentation wasn't super transparent on API global key vs Token and programmatic input assumptions; had to do some digging through Issues to clarify. Became a bit of a rabbit hole XD

Definitely not very "turnkey"...

Appreciate the product, so hopefully I can help resolve this in whatever manner you need.
Happy to follow direction too where time allows.

And your appreciation is appreciated 🤣

My plan is to at least test, document and as need, debug usage with Route53. I imagine it won't be immediately transferable, but at least will demonstrate a "working" config. Hopefully it will also allow me to get an idea of what is actually going wrong here...

Ah sorry just quick revelation RE: lexicon CLI:

• create did actually generate the record as specified; just found it in my CF dashboard
• delete on CLI also deleted it, despite the ominous traceback
• subdomain record creation/deletion also okay

Thanks for posting the additional info. One thing it highlights is that the core issue is actually with the lexicon library (or perhaps just the way we're trying to use it).

I aim to prioritise working in this but it's Friday here (for you too assuming you're in NZ - I'm in Au) and I have an existing commitment so whilst I'll try to at least post an update, I may not have much progress until early next week.