Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JWS has no anti-replay nonce #684

Open
throw1008a opened this issue Oct 8, 2019 · 12 comments

Comments

@throw1008a
Copy link

commented Oct 8, 2019

So I needed to renew a cert on Monday, October 7, 2019, and got an error about "JWS has no anti-replay nonce"; this was with dehydrated 0.6.2 (it has been running fine for over a year). Doing some testing, I moved up to dehydrated 0.6.3, and got the following on a cert that needed renewal:

+ Requesting new certificate order from CA...
+ ERROR: An error occurred while sending post-request to https://acme-v02.api.letsencrypt.org/acme/new-order (Status 400)

Details:
HTTP/2.0 400
server:nginx
date:Tue, 08 Oct 2019 12:13:22 GMT
content-type:application/problem+json
content-length:112
boulder-requester:34257164
cache-control:public, max-age=0, no-cache
link:<https://acme-v02.api.letsencrypt.org/directory>;rel="index"
replay-nonce:0001 [...] V99NYWw

{
 "type": "urn:ietf:params:acme:error:badNonce",
 "detail": "JWS has no anti-replay nonce",
 "status": 400
}

Then in 0.6.4 and 0.6.5 I get:

+ Fetching account ID...
  + ERROR: An error occurred while sending post-request to https://acme-v02.api.letsencrypt.org/acme/new-acct (Status 400)

Details:
HTTP/2.0 400
server:nginx
date:Tue, 08 Oct 2019 12:16:36 GMT
content-type:application/problem+json
content-length:112
cache-control:public, max-age=0, no-cache
link:<https://acme-v02.api.letsencrypt.org/directory>;rel="index"
replay-nonce:0002 [...] RLKbE0UKE

{
  "type": "urn:ietf:params:acme:error:badNonce",
  "detail": "JWS has no anti-replay nonce",
  "status": 400
}

The version of cURL:
curl 7.47.1 (x86_64-redhat-linux-gnu) libcurl/7.47.1 OpenSSL/1.0.2p zlib/1.2.7 libidn/1.28 libssh2/1.4.3 nghttp2/1.31.1

After some playing around, I added an extra configuration file with the following contents:

CURL_OPTS="--http1.1"

No longer seem to get the “JWS has no anti-replay nonce” error.

See also this thread:

@throw1008a

This comment has been minimized.

Copy link
Author

commented Oct 8, 2019

Full output of cURL version:

curl 7.47.1 (x86_64-redhat-linux-gnu) libcurl/7.47.1 OpenSSL/1.0.2p zlib/1.2.7 libidn/1.28 libssh2/1.4.3 nghttp2/1.31.1
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets 

@ilude

This comment has been minimized.

Copy link

commented Oct 8, 2019

Master branch seems to fix this issue. Any chance we can get a v0.6.6 release tag?

@hefty-pty

This comment has been minimized.

Copy link

commented Oct 8, 2019

crossposting from https://community.letsencrypt.org/t/jws-has-no-anti-replay-nonce/103324/14
dehydrated output from where it goes wrong: https://pastebin.com/2VWLpgth

it ultimately seems to be an error with the letsencrypt infrastructure, but maybe dehydrated could handle an tls-connection error while trying to retrieve the nonce more graceful?

@hefty-pty

This comment has been minimized.

Copy link

commented Oct 8, 2019

also: shouldn't the curlret status in line 534 (master v0.6.5) be set before the touch command?

@lukas2511

This comment has been minimized.

Copy link
Owner

commented Oct 9, 2019

Master branch seems to fix this issue. Any chance we can get a v0.6.6 release tag?

Master branch currently is exactly at the state of v0.6.5. All other changes I'm working on are currently only on my local clone.

I've found what's going on here, it seems that fetching a nonce fails and for whatever reason (didn't have the time to debug it yet) dehydrated doesn't exit at that point but just keeps running with an empty nonce.

I'll have to implement a loooot of retry logic in the (very) near future as the availability of Let's Encrypts CDN seems to be getting worse by the day... All of the missing nonces and "error 35" messages etc... meh :-/

@lukas2511

This comment has been minimized.

Copy link
Owner

commented Oct 9, 2019

also: shouldn't the curlret status in line 534 (master v0.6.5) be set before the touch command?

Good find. That may actually be the logic error. I'll commit that to master branch. Would be great if somebody could test it.

@pR0Ps

This comment has been minimized.

Copy link

commented Oct 9, 2019

@lukas2511 Just pulled master and updated my certs - everything worked perfectly.

Before pulling master I was on v0.6.2 and seeing the anti-replay error. Might just be me getting lucky though.

@hefty-pty

This comment has been minimized.

Copy link

commented Oct 9, 2019

here a quickfix that implements while loops in http_request() for retry. patch is for v0.6.5, so it also fixes the curlret status issue already patched in master.
https://pastebin.com/igFHAp6D

@ilude

This comment has been minimized.

Copy link

commented Oct 9, 2019

Master branch currently is exactly at the state of v0.6.5.

Understood. I was just looking at SHA's yesterday morning when the issue occurred and master was different because of the merge. I was on the v0.6.5 tag and was getting the error and when I updated to master it went away and I have not seen it since. So my best assumption at this point is that I was getting the error due to something on LetsEncrypt's side of things, as you noted, and it was all just a timing issue.

@lukas2511

This comment has been minimized.

Copy link
Owner

commented Oct 9, 2019

Just to be clear: The underlying issue hasn't been completely fixed yet. This error occurs due to connection issues with the CA and there currently is no retry-logic in dehydrated. The only thing the last commit fixed should be the type of the error message, instead of failing with "no nonce" dehydrated should (at least in theory...) now notice the broken HEAD request and fail early with a correct error message, but I think there is another issue which again prevents this from working correctly. I'll have to do some work on it, until then you can always just rerun the script, at least with ACME it at least kinda keeps the state and doesn't have to rerun everything again.

@bonidier

This comment has been minimized.

Copy link

commented Oct 11, 2019

Hi,
FYI I've encountered this errors 2 days ago (curl error 35 + nonce) with a big SAN request using http-01, and it's OK today without any change (dehydrated 0.6.5).
This can also be due to requests limit from ACME servers.

Does playing with Curl options --retry / --retry-delay / --retry-max-time into CURL_OPTS may easily resolve retry logic (instead of reinventing the wheel) ?

@hefty-pty

This comment has been minimized.

Copy link

commented Oct 11, 2019

I looked at this, but you'll need at least curl 7.55.0 (which features --retry-connrefused) as earlier versions don't retry with connection errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.