Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[md:error] (13)Permission denied: AH10056: processing {DOMAIN} #117

Closed
jmwebservices opened this issue Mar 16, 2019 · 14 comments

Comments

@jmwebservices
Copy link

commented Mar 16, 2019

I have been using mod_md for a few months without issues. I have a handful of managed domains and the first one is up for renewal. Apache is throwing the following error:

[md:error] (13)Permission denied: AH10056: processing {DOMAIN}

I researched this error but found issues where other error messages accompanied this one. This is the only error being thrown. I ran a LogLevel md:trace1 and the output is below:

[Fri Mar 15 20:02:31 2019] [mpm_event:notice] AH00492: caught SIGWINCH, shutting down gracefully
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(1013): AH10070: initializing post config dry run
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(361): AH10037: server seems reachable via http: (port 80->80) and reachable via https: (port 443->443) 
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(221): AH10041: Server {DOMAIN}:0 matches md {DOMAIN} (config {DOMAIN}[default, default])
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(251): AH10043: Managed Domain {DOMAIN} applies to vhost {DOMAIN}:0
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(221): AH10041: Server {DOMAIN}:0 matches md {DOMAIN} (config {DOMAIN}[default, default])
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(251): AH10043: Managed Domain {DOMAIN} applies to vhost {DOMAIN}:0
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(281): AH10044: {DOMAIN}: added contact mailto:{EMAIL}
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(392): AH10039: Completed MD[{DOMAIN}, CA=https://acme-v01.api.letsencrypt.org/directory, Proto=ACME, Agreement=https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf, Drive=1, renew=2592000000000]
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(793): {DOMAIN}: update renew norm=7776000000000, window=2592000000000
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(488): update md {DOMAIN}
[Fri Mar 15 20:02:34 2019] [md:trace1] md_reg.c(528): update renew-window: {DOMAIN}
[Fri Mar 15 20:02:34 2019] [md:trace1] md_reg.c(549): update transitive: {DOMAIN}
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(830): md {DOMAIN} updated
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [md:info] AH10071: mod_md (v1.1.17), initializing...
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(361): AH10037: server seems reachable via http: (port 80->80) and reachable via https: (port 443->443) 
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(221): AH10041: Server {DOMAIN}:0 matches md {DOMAIN} (config {DOMAIN}[default, default])
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(251): AH10043: Managed Domain {DOMAIN} applies to vhost {DOMAIN}:0
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(221): AH10041: Server {DOMAIN}:0 matches md {DOMAIN} (config {DOMAIN}[default, default])
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(251): AH10043: Managed Domain {DOMAIN} applies to vhost {DOMAIN}:0
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(281): AH10044: {DOMAIN}: added contact mailto:{EMAIL}
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(392): AH10039: Completed MD[{DOMAIN}, CA=https://acme-v01.api.letsencrypt.org/directory, Proto=ACME, Agreement=https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf, Drive=1, renew=2592000000000]
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(793): {DOMAIN}: update renew norm=7776000000000, window=2592000000000
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(488): update md {DOMAIN}
[Fri Mar 15 20:02:34 2019] [md:trace1] md_reg.c(528): update renew-window: {DOMAIN}
[Fri Mar 15 20:02:34 2019] [md:trace1] md_reg.c(549): update transitive: {DOMAIN}
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(830): md {DOMAIN} updated
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [md:trace1] md_acme_drive.c(671): {DOMAIN}: init driver
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(974): {DOMAIN}: run load
[Fri Mar 15 20:02:34 2019] [md:debug] md_acme_drive.c(926): {DOMAIN}: preload start
[Fri Mar 15 20:02:34 2019] [md:debug] md_acme_drive.c(941): (2)No such file or directory: {DOMAIN}: loading staging private key
[Fri Mar 15 20:02:34 2019] [md:debug] md_acme_drive.c(1010): (2)No such file or directory: {DOMAIN}: ACME, ACME preload
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(996): (2)No such file or directory: {DOMAIN}: load done
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(927): AH10064: md({DOMAIN}): state=2, driving
[Fri Mar 15 20:02:34 2019] [md:debug] md_store_fs.c(688): purge staging/{DOMAIN} (/WAMP/apache/md/staging/{DOMAIN})
[Fri Mar 15 20:02:34 2019] [md:debug] md_store_fs.c(688): (2)No such file or directory: purge challenges/{DOMAIN} (/WAMP/apache/md/challenges/{DOMAIN})
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(959): AH10067: register md watchdog(_md_)
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [mpm_event:notice] AH00489: Apache/2.4.38 (Unix) OpenSSL/1.1.1 configured -- resuming normal operations
[Fri Mar 15 20:02:34 2019] [core:notice] AH00094: Command line: '/WAMP/apache/bin/httpd -D PROD'
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(749): AH10054: md watchdog start, auto drive 3 mds
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(761): AH10055: md watchdog run, auto drive 3 mds
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(690): AH10052: md({DOMAIN}): state=2, driving
[Fri Mar 15 20:02:34 2019] [md:trace1] md_acme_drive.c(671): {DOMAIN}: init driver
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(902): {DOMAIN}: run staging
[Fri Mar 15 20:02:34 2019] [md:debug] md_acme_drive.c(687): {DOMAIN}: staging started, state=2, can_http=1, can_https=1, challenges='http-01 tls-sni-01'
[Fri Mar 15 20:02:34 2019] [md:trace1] md_acme_drive.c(705): {DOMAIN}: checked staging area, will reset
[Fri Mar 15 20:02:34 2019] [md:debug] md_store_fs.c(688): (2)No such file or directory: purge staging/{DOMAIN} (/WAMP/apache/md/staging/{DOMAIN})
[Fri Mar 15 20:02:34 2019] [md:debug] md_acme.c(145): get directory from https://acme-v01.api.letsencrypt.org/directory
[Fri Mar 15 20:02:34 2019] [md:trace1] md_acme.c(488): add acme GET: https://acme-v01.api.letsencrypt.org/directory
[Fri Mar 15 20:02:34 2019] [md:debug] md_acme.c(422): req: POST https://acme-v01.api.letsencrypt.org/directory
[Fri Mar 15 20:02:34 2019] [md:trace1] md_curl.c(239): request 0 --> GET https://acme-v01.api.letsencrypt.org/directory
[Fri Mar 15 20:02:35 2019] [md:trace1] md_curl.c(255): request 0 <-- 200
[Fri Mar 15 20:02:35 2019] [md:trace1] md_acme.c(331): response: 200
[Fri Mar 15 20:02:35 2019] [md:trace2] md_acme.c(342): response: {\n  "Txe5rG-uWMc": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",\n  "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",\n  "meta": {\n    "caaIdentities": [\n      "letsencrypt.org"\n    ],\n    "terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",\n    "website": "https://letsencrypt.org"\n  },\n  "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",\n  "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",\n  "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",\n  "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"\n}
[Fri Mar 15 20:02:35 2019] [md:debug] md_acme.c(440): req sent
[Fri Mar 15 20:02:35 2019] [md:info] {DOMAIN}: setup staging
[Fri Mar 15 20:02:35 2019] [md:debug] md_store_fs.c(688): (2)No such file or directory: purge staging/{DOMAIN} (/WAMP/apache/md/staging/{DOMAIN})
[Fri Mar 15 20:02:35 2019] [md:debug] md_acme_drive.c(757): {DOMAIN}: save staged md
[Fri Mar 15 20:02:35 2019] [md:info] {DOMAIN}: need certificate
[Fri Mar 15 20:02:35 2019] [md:debug] md_acme_drive.c(95): re-use account 'ACME-.letsencrypt.org-0000'
[Fri Mar 15 20:02:35 2019] [md:debug] md_acme_acct.c(192): (13)Permission denied: error reading account: ACME-.letsencrypt.org-0000
[Fri Mar 15 20:02:35 2019] [md:debug] md_acme_drive.c(909): (13)Permission denied: {DOMAIN}: ACME, choose account
[Fri Mar 15 20:02:35 2019] [md:debug] md_reg.c(909): (13)Permission denied: {DOMAIN}: staging done
[Fri Mar 15 20:02:35 2019] [md:error] (13)Permission denied: AH10056: processing {DOMAIN}
[Fri Mar 15 20:02:35 2019] [md:info] AH10057: {DOMAIN}: encountered error for the 13. time, next run in  1:00:00 hours
[Fri Mar 15 20:02:35 2019] [md:trace1] mod_md.c(730): {DOMAIN}: saving job props
[Fri Mar 15 20:02:35 2019] [md:debug] mod_md.c(783): AH10107: next run in  0:59:59 hours

Any help would be appreciated.

@jmwebservices

This comment has been minimized.

Copy link
Author

commented Mar 16, 2019

Here is a summary of directory and file permissions:

Path Permission Owner/Group
md/accounts rwxr-xr-x root/root
md/accounts/ACME-.letsencrypt.org-0000 rwxr-xr-x root/root
md/accounts/ACME-.letsencrypt.org-0000/account.json rw-r---- root/root
md/accounts/ACME-.letsencrypt.org-0000/account.pem rw-r---- root/root

I changed account.json and account.pem to rw-r--r-- and the certificate was successfully renewed:

[md:notice] AH10051: {DOMAIN}: has been renewed successfully and should be activated at Sun, 17 Mar 2019 00:11:27 GMT (this requires a server restart latest in 22:59:58 hours)

So, the remaining question is why were the perms wrong to begin with?

@jmwebservices jmwebservices changed the title [md:error] (13)Permission denied: AH10056: processing [md:error] (13)Permission denied: AH10056: processing {DOMAIN} Mar 16, 2019

@jmwebservices

This comment has been minimized.

Copy link
Author

commented Mar 16, 2019

I believe I found the culprit.

In my /etc/init.d/ startup script I have the following line immediately before calling apachectl:

umask 007

I can't remember why but that mask has been there for ages. @icing, can (and should) mod_md break the mask and force file permissions? Or, can (and should) mod_md assign a different owner/group just like it does for the md/challenges and md/staging directories?

@icing

This comment has been minimized.

Copy link
Owner

commented Mar 16, 2019

@icing

This comment has been minimized.

Copy link
Owner

commented Mar 18, 2019

My test suite will not catch this as it does not run as root and httpd only switches users then.

I committed a change that hopefully addresses this issue and sets the file permissions explicitly. Are you able to clone the current master here, build and verify easily? I could also apply the change to the Apache subversion tree in case you want to avoid all the new things here.

@jmwebservices

This comment has been minimized.

Copy link
Author

commented Mar 18, 2019

Thanks for fixing this! As the readme states, patching mod_ssl is not my cup of tea. I would much rather run mod_md that is bundled with the Apache distro. I am currently running 2.4.38. What version will receive your fix, 2.4.39? Also, what permissions should I set to the existing managed domains?

Thanks again!

asfgit pushed a commit to apache/httpd that referenced this issue Mar 18, 2019

Stefan Eissing
*) mod_md: Explicitly setting file permissions to break out of umask…
…s. We want our

     non-privilegded apache user to be able to read them. See github issue
     <icing/mod_md#117>. [Stefan Eissing] 



git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1855777 13f79535-47bb-0310-9956-ffa450edef68

icing added a commit that referenced this issue Mar 18, 2019

*) Explicitly setting file permissions to break out of umasks. We wa…
…nt our

     non-privilegded apache user to be able to read them. See github issue
     <#117>.
  *) Merged some default store dir change from trunk.
  *) Merged unread var removal from trunk.
@icing

This comment has been minimized.

Copy link
Owner

commented Mar 18, 2019

Made you a v1.1.18 for testing.

@jmwebservices

This comment has been minimized.

Copy link
Author

commented Mar 18, 2019

I downloaded 1.1.18, unzipped it on the server and ran the following:
./configure.ac --with-apxs=/WAMP/apache/bin/apxs --enable-werror

Then, the following error was thrown:

./configure.ac: line 16: syntax error near unexpected token `[2.69]'
./configure.ac: line 16: `AC_PREREQ([2.69])'

A few things to note:

  1. Don't mind the reference to WAMP. The sever is actually running CentOS.
  2. ./configure did not work as written in the 2.4x installation section of the Wiki. So, I changed it to ./configure.ac.

As an alternative, I am willing to recompile Apache 2.4.38 with the new mod_md 1.1.18. Just tell me how.

@icing

This comment has been minimized.

Copy link
Owner

commented Mar 19, 2019

The configure.ac is the file for autoconf to generate the ./configure among other things. You cannot run it directly.

But autoconf/automake is not the lightest of topics. You can just take the *.c and *.h files from src, copy them into your 2.4.38 source at modules/mdand build and install.

@jmwebservices

This comment has been minimized.

Copy link
Author

commented Mar 19, 2019

mod_md 1.1.18 has been compiled as you suggested. Now, how do I test?

@icing

This comment has been minimized.

Copy link
Owner

commented Mar 20, 2019

Hmm, assuming you do not want to interrupt you working domains...take a another domain or subdomain and configure the staging environment of Lets Encrypt, like

<MDomainSet blabla.your-domain>
  MDDriveMode always
  MDCertificateAuthority https://acme-staging.api.letsencrypt.org/directory
</MDomainSet>

If you have no staging account yet, this should create a new one with the right permissions. If you have one, move it aside temporarily (or just remove it). A staging account should how in its JSON:

    "url": "https://acme-staging.api.letsencrypt.org/acme/reg/NNNNN"

where NNNNN is some number.

Hope this helps.

@jmwebservices

This comment has been minimized.

Copy link
Author

commented Mar 20, 2019

I followed your directions and it appears the permissions are still not being set correctly. Here is what I did:

  1. Added a new subdomain to my public DNS (test.MYDOMAIN.COM)
  2. Added the following to my main server config
<MDomainSet test.MYDOMAIN.COM>
  MDDriveMode always
  MDCertificateAuthority https://acme-staging.api.letsencrypt.org/directory
</MDomainSet>
  1. Executed /etc/init.d/apache graceful two times
  2. Observed the following permissions in the new /accounts/ directory (same as before):
Path Permission Owner/Group
md/accounts rwxr-xr-x root/root
md/accounts/ACME-.letsencrypt.org-0003 rwxr-xr-x root/root
md/accounts/ACME-.letsencrypt.org-0003/account.json rw-r---- root/root
md/accounts/ACME-.letsencrypt.org-0003/account.pem rw-r---- root/root

Wouldn't it be better to just change the group/owner just as done for /challenges and /staging?

icing added a commit that referenced this issue Mar 21, 2019

*) Another tackle at <#117>, explicitly setting
     specified permissions on all files created in md store.
@icing

This comment has been minimized.

Copy link
Owner

commented Mar 21, 2019

Wouldn't it be better to just change the group/owner just as done for /challenges and /staging?

No, the files really belong to the one starting the server and should be readable by everyone.

I made v1.1.19 with more permission setting goodness. Could you give this a try? (You can just remove all staging related information in the store and start over) Thanks!

@jmwebservices

This comment has been minimized.

Copy link
Author

commented Mar 21, 2019

Good job - that did the trick! Now, what httpd release will this patch be included?

@icing

This comment has been minimized.

Copy link
Owner

commented Mar 26, 2019

We are about to tag, I think. So, it will be too late for that. I will bring it into the one after that.

asfgit pushed a commit to apache/httpd that referenced this issue Mar 26, 2019

Stefan Eissing
*) mod_md: Store permissions are enforced on file creation, enforcin…
…g restrictions in

     spite of umask. Fixes <icing/mod_md#117>. [Stefan Eissing]



git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1856297 13f79535-47bb-0310-9956-ffa450edef68

asfgit pushed a commit to apache/httpd that referenced this issue Apr 4, 2019

Stefan Eissing
Merged /httpd/httpd/trunk:r1856297
  *) mod_md: Store permissions are enforced on file creation, enforcing restrictions in
     spite of umask. Fixes <icing/mod_md#117>. [Stefan Eissing]



git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x@1856935 13f79535-47bb-0310-9956-ffa450edef68

icing added a commit that referenced this issue May 13, 2019

v1.99.8
----------------------------------------------------------------------------------------------------
 * Added MD section to Apache's "server-status" page listing all managed domains and their
   settings, renewal status and error count.
 * Fix for #117, explicitly set file permissions to work around umask defaults.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.