Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[md:error] (13)Permission denied: AH10056: processing {DOMAIN} #117

Closed
jmwebservices opened this issue Mar 16, 2019 · 14 comments
Closed

[md:error] (13)Permission denied: AH10056: processing {DOMAIN} #117

jmwebservices opened this issue Mar 16, 2019 · 14 comments

Comments

@jmwebservices
Copy link

@jmwebservices jmwebservices commented Mar 16, 2019

I have been using mod_md for a few months without issues. I have a handful of managed domains and the first one is up for renewal. Apache is throwing the following error:

[md:error] (13)Permission denied: AH10056: processing {DOMAIN}

I researched this error but found issues where other error messages accompanied this one. This is the only error being thrown. I ran a LogLevel md:trace1 and the output is below:

[Fri Mar 15 20:02:31 2019] [mpm_event:notice] AH00492: caught SIGWINCH, shutting down gracefully
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(1013): AH10070: initializing post config dry run
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(361): AH10037: server seems reachable via http: (port 80->80) and reachable via https: (port 443->443) 
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(221): AH10041: Server {DOMAIN}:0 matches md {DOMAIN} (config {DOMAIN}[default, default])
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(251): AH10043: Managed Domain {DOMAIN} applies to vhost {DOMAIN}:0
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(221): AH10041: Server {DOMAIN}:0 matches md {DOMAIN} (config {DOMAIN}[default, default])
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(251): AH10043: Managed Domain {DOMAIN} applies to vhost {DOMAIN}:0
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(281): AH10044: {DOMAIN}: added contact mailto:{EMAIL}
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(392): AH10039: Completed MD[{DOMAIN}, CA=https://acme-v01.api.letsencrypt.org/directory, Proto=ACME, Agreement=https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf, Drive=1, renew=2592000000000]
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(793): {DOMAIN}: update renew norm=7776000000000, window=2592000000000
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(488): update md {DOMAIN}
[Fri Mar 15 20:02:34 2019] [md:trace1] md_reg.c(528): update renew-window: {DOMAIN}
[Fri Mar 15 20:02:34 2019] [md:trace1] md_reg.c(549): update transitive: {DOMAIN}
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(830): md {DOMAIN} updated
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [md:info] AH10071: mod_md (v1.1.17), initializing...
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(361): AH10037: server seems reachable via http: (port 80->80) and reachable via https: (port 443->443) 
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(221): AH10041: Server {DOMAIN}:0 matches md {DOMAIN} (config {DOMAIN}[default, default])
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(251): AH10043: Managed Domain {DOMAIN} applies to vhost {DOMAIN}:0
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(221): AH10041: Server {DOMAIN}:0 matches md {DOMAIN} (config {DOMAIN}[default, default])
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(251): AH10043: Managed Domain {DOMAIN} applies to vhost {DOMAIN}:0
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(281): AH10044: {DOMAIN}: added contact mailto:{EMAIL}
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(392): AH10039: Completed MD[{DOMAIN}, CA=https://acme-v01.api.letsencrypt.org/directory, Proto=ACME, Agreement=https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf, Drive=1, renew=2592000000000]
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(793): {DOMAIN}: update renew norm=7776000000000, window=2592000000000
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(488): update md {DOMAIN}
[Fri Mar 15 20:02:34 2019] [md:trace1] md_reg.c(528): update renew-window: {DOMAIN}
[Fri Mar 15 20:02:34 2019] [md:trace1] md_reg.c(549): update transitive: {DOMAIN}
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(830): md {DOMAIN} updated
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [md:trace1] md_acme_drive.c(671): {DOMAIN}: init driver
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(974): {DOMAIN}: run load
[Fri Mar 15 20:02:34 2019] [md:debug] md_acme_drive.c(926): {DOMAIN}: preload start
[Fri Mar 15 20:02:34 2019] [md:debug] md_acme_drive.c(941): (2)No such file or directory: {DOMAIN}: loading staging private key
[Fri Mar 15 20:02:34 2019] [md:debug] md_acme_drive.c(1010): (2)No such file or directory: {DOMAIN}: ACME, ACME preload
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(996): (2)No such file or directory: {DOMAIN}: load done
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(927): AH10064: md({DOMAIN}): state=2, driving
[Fri Mar 15 20:02:34 2019] [md:debug] md_store_fs.c(688): purge staging/{DOMAIN} (/WAMP/apache/md/staging/{DOMAIN})
[Fri Mar 15 20:02:34 2019] [md:debug] md_store_fs.c(688): (2)No such file or directory: purge challenges/{DOMAIN} (/WAMP/apache/md/challenges/{DOMAIN})
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(959): AH10067: register md watchdog(_md_)
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(245): md{{DOMAIN}}: is complete
[Fri Mar 15 20:02:34 2019] [mpm_event:notice] AH00489: Apache/2.4.38 (Unix) OpenSSL/1.1.1 configured -- resuming normal operations
[Fri Mar 15 20:02:34 2019] [core:notice] AH00094: Command line: '/WAMP/apache/bin/httpd -D PROD'
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(749): AH10054: md watchdog start, auto drive 3 mds
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(761): AH10055: md watchdog run, auto drive 3 mds
[Fri Mar 15 20:02:34 2019] [md:debug] mod_md.c(690): AH10052: md({DOMAIN}): state=2, driving
[Fri Mar 15 20:02:34 2019] [md:trace1] md_acme_drive.c(671): {DOMAIN}: init driver
[Fri Mar 15 20:02:34 2019] [md:debug] md_reg.c(902): {DOMAIN}: run staging
[Fri Mar 15 20:02:34 2019] [md:debug] md_acme_drive.c(687): {DOMAIN}: staging started, state=2, can_http=1, can_https=1, challenges='http-01 tls-sni-01'
[Fri Mar 15 20:02:34 2019] [md:trace1] md_acme_drive.c(705): {DOMAIN}: checked staging area, will reset
[Fri Mar 15 20:02:34 2019] [md:debug] md_store_fs.c(688): (2)No such file or directory: purge staging/{DOMAIN} (/WAMP/apache/md/staging/{DOMAIN})
[Fri Mar 15 20:02:34 2019] [md:debug] md_acme.c(145): get directory from https://acme-v01.api.letsencrypt.org/directory
[Fri Mar 15 20:02:34 2019] [md:trace1] md_acme.c(488): add acme GET: https://acme-v01.api.letsencrypt.org/directory
[Fri Mar 15 20:02:34 2019] [md:debug] md_acme.c(422): req: POST https://acme-v01.api.letsencrypt.org/directory
[Fri Mar 15 20:02:34 2019] [md:trace1] md_curl.c(239): request 0 --> GET https://acme-v01.api.letsencrypt.org/directory
[Fri Mar 15 20:02:35 2019] [md:trace1] md_curl.c(255): request 0 <-- 200
[Fri Mar 15 20:02:35 2019] [md:trace1] md_acme.c(331): response: 200
[Fri Mar 15 20:02:35 2019] [md:trace2] md_acme.c(342): response: {\n  "Txe5rG-uWMc": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",\n  "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",\n  "meta": {\n    "caaIdentities": [\n      "letsencrypt.org"\n    ],\n    "terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",\n    "website": "https://letsencrypt.org"\n  },\n  "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",\n  "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",\n  "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",\n  "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"\n}
[Fri Mar 15 20:02:35 2019] [md:debug] md_acme.c(440): req sent
[Fri Mar 15 20:02:35 2019] [md:info] {DOMAIN}: setup staging
[Fri Mar 15 20:02:35 2019] [md:debug] md_store_fs.c(688): (2)No such file or directory: purge staging/{DOMAIN} (/WAMP/apache/md/staging/{DOMAIN})
[Fri Mar 15 20:02:35 2019] [md:debug] md_acme_drive.c(757): {DOMAIN}: save staged md
[Fri Mar 15 20:02:35 2019] [md:info] {DOMAIN}: need certificate
[Fri Mar 15 20:02:35 2019] [md:debug] md_acme_drive.c(95): re-use account 'ACME-.letsencrypt.org-0000'
[Fri Mar 15 20:02:35 2019] [md:debug] md_acme_acct.c(192): (13)Permission denied: error reading account: ACME-.letsencrypt.org-0000
[Fri Mar 15 20:02:35 2019] [md:debug] md_acme_drive.c(909): (13)Permission denied: {DOMAIN}: ACME, choose account
[Fri Mar 15 20:02:35 2019] [md:debug] md_reg.c(909): (13)Permission denied: {DOMAIN}: staging done
[Fri Mar 15 20:02:35 2019] [md:error] (13)Permission denied: AH10056: processing {DOMAIN}
[Fri Mar 15 20:02:35 2019] [md:info] AH10057: {DOMAIN}: encountered error for the 13. time, next run in  1:00:00 hours
[Fri Mar 15 20:02:35 2019] [md:trace1] mod_md.c(730): {DOMAIN}: saving job props
[Fri Mar 15 20:02:35 2019] [md:debug] mod_md.c(783): AH10107: next run in  0:59:59 hours

Any help would be appreciated.

@jmwebservices
Copy link
Author

@jmwebservices jmwebservices commented Mar 16, 2019

Here is a summary of directory and file permissions:

Path Permission Owner/Group
md/accounts rwxr-xr-x root/root
md/accounts/ACME-.letsencrypt.org-0000 rwxr-xr-x root/root
md/accounts/ACME-.letsencrypt.org-0000/account.json rw-r---- root/root
md/accounts/ACME-.letsencrypt.org-0000/account.pem rw-r---- root/root

I changed account.json and account.pem to rw-r--r-- and the certificate was successfully renewed:

[md:notice] AH10051: {DOMAIN}: has been renewed successfully and should be activated at Sun, 17 Mar 2019 00:11:27 GMT (this requires a server restart latest in 22:59:58 hours)

So, the remaining question is why were the perms wrong to begin with?

Loading

@jmwebservices jmwebservices changed the title [md:error] (13)Permission denied: AH10056: processing [md:error] (13)Permission denied: AH10056: processing {DOMAIN} Mar 16, 2019
@jmwebservices
Copy link
Author

@jmwebservices jmwebservices commented Mar 16, 2019

I believe I found the culprit.

In my /etc/init.d/ startup script I have the following line immediately before calling apachectl:

umask 007

I can't remember why but that mask has been there for ages. @icing, can (and should) mod_md break the mask and force file permissions? Or, can (and should) mod_md assign a different owner/group just like it does for the md/challenges and md/staging directories?

Loading

@icing
Copy link
Owner

@icing icing commented Mar 16, 2019

Loading

@icing
Copy link
Owner

@icing icing commented Mar 18, 2019

My test suite will not catch this as it does not run as root and httpd only switches users then.

I committed a change that hopefully addresses this issue and sets the file permissions explicitly. Are you able to clone the current master here, build and verify easily? I could also apply the change to the Apache subversion tree in case you want to avoid all the new things here.

Loading

@jmwebservices
Copy link
Author

@jmwebservices jmwebservices commented Mar 18, 2019

Thanks for fixing this! As the readme states, patching mod_ssl is not my cup of tea. I would much rather run mod_md that is bundled with the Apache distro. I am currently running 2.4.38. What version will receive your fix, 2.4.39? Also, what permissions should I set to the existing managed domains?

Thanks again!

Loading

asfgit pushed a commit to apache/httpd that referenced this issue Mar 18, 2019
…s. We want our

     non-privilegded apache user to be able to read them. See github issue
     <icing/mod_md#117>. [Stefan Eissing] 



git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1855777 13f79535-47bb-0310-9956-ffa450edef68
icing added a commit that referenced this issue Mar 18, 2019
…nt our

     non-privilegded apache user to be able to read them. See github issue
     <#117>.
  *) Merged some default store dir change from trunk.
  *) Merged unread var removal from trunk.
@icing
Copy link
Owner

@icing icing commented Mar 18, 2019

Made you a v1.1.18 for testing.

Loading

@jmwebservices
Copy link
Author

@jmwebservices jmwebservices commented Mar 18, 2019

I downloaded 1.1.18, unzipped it on the server and ran the following:
./configure.ac --with-apxs=/WAMP/apache/bin/apxs --enable-werror

Then, the following error was thrown:

./configure.ac: line 16: syntax error near unexpected token `[2.69]'
./configure.ac: line 16: `AC_PREREQ([2.69])'

A few things to note:

  1. Don't mind the reference to WAMP. The sever is actually running CentOS.
  2. ./configure did not work as written in the 2.4x installation section of the Wiki. So, I changed it to ./configure.ac.

As an alternative, I am willing to recompile Apache 2.4.38 with the new mod_md 1.1.18. Just tell me how.

Loading

@icing
Copy link
Owner

@icing icing commented Mar 19, 2019

The configure.ac is the file for autoconf to generate the ./configure among other things. You cannot run it directly.

But autoconf/automake is not the lightest of topics. You can just take the *.c and *.h files from src, copy them into your 2.4.38 source at modules/mdand build and install.

Loading

@jmwebservices
Copy link
Author

@jmwebservices jmwebservices commented Mar 19, 2019

mod_md 1.1.18 has been compiled as you suggested. Now, how do I test?

Loading

@icing
Copy link
Owner

@icing icing commented Mar 20, 2019

Hmm, assuming you do not want to interrupt you working domains...take a another domain or subdomain and configure the staging environment of Lets Encrypt, like

<MDomainSet blabla.your-domain>
  MDDriveMode always
  MDCertificateAuthority https://acme-staging.api.letsencrypt.org/directory
</MDomainSet>

If you have no staging account yet, this should create a new one with the right permissions. If you have one, move it aside temporarily (or just remove it). A staging account should how in its JSON:

    "url": "https://acme-staging.api.letsencrypt.org/acme/reg/NNNNN"

where NNNNN is some number.

Hope this helps.

Loading

@jmwebservices
Copy link
Author

@jmwebservices jmwebservices commented Mar 20, 2019

I followed your directions and it appears the permissions are still not being set correctly. Here is what I did:

  1. Added a new subdomain to my public DNS (test.MYDOMAIN.COM)
  2. Added the following to my main server config
<MDomainSet test.MYDOMAIN.COM>
  MDDriveMode always
  MDCertificateAuthority https://acme-staging.api.letsencrypt.org/directory
</MDomainSet>
  1. Executed /etc/init.d/apache graceful two times
  2. Observed the following permissions in the new /accounts/ directory (same as before):
Path Permission Owner/Group
md/accounts rwxr-xr-x root/root
md/accounts/ACME-.letsencrypt.org-0003 rwxr-xr-x root/root
md/accounts/ACME-.letsencrypt.org-0003/account.json rw-r---- root/root
md/accounts/ACME-.letsencrypt.org-0003/account.pem rw-r---- root/root

Wouldn't it be better to just change the group/owner just as done for /challenges and /staging?

Loading

icing added a commit that referenced this issue Mar 21, 2019
     specified permissions on all files created in md store.
@icing
Copy link
Owner

@icing icing commented Mar 21, 2019

Wouldn't it be better to just change the group/owner just as done for /challenges and /staging?

No, the files really belong to the one starting the server and should be readable by everyone.

I made v1.1.19 with more permission setting goodness. Could you give this a try? (You can just remove all staging related information in the store and start over) Thanks!

Loading

@jmwebservices
Copy link
Author

@jmwebservices jmwebservices commented Mar 21, 2019

Good job - that did the trick! Now, what httpd release will this patch be included?

Loading

@icing
Copy link
Owner

@icing icing commented Mar 26, 2019

We are about to tag, I think. So, it will be too late for that. I will bring it into the one after that.

Loading

asfgit pushed a commit to apache/httpd that referenced this issue Mar 26, 2019
…g restrictions in

     spite of umask. Fixes <icing/mod_md#117>. [Stefan Eissing]



git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1856297 13f79535-47bb-0310-9956-ffa450edef68
asfgit pushed a commit to apache/httpd that referenced this issue Apr 4, 2019
  *) mod_md: Store permissions are enforced on file creation, enforcing restrictions in
     spite of umask. Fixes <icing/mod_md#117>. [Stefan Eissing]



git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x@1856935 13f79535-47bb-0310-9956-ffa450edef68
icing added a commit that referenced this issue May 13, 2019
----------------------------------------------------------------------------------------------------
 * Added MD section to Apache's "server-status" page listing all managed domains and their
   settings, renewal status and error count.
 * Fix for #117, explicitly set file permissions to work around umask defaults.
asfgit pushed a commit to apache/httpd that referenced this issue Oct 5, 2019
  *) mod_md: Store permissions are enforced on file creation, enforcing restrictions in
     spite of umask. Fixes <icing/mod_md#117>. [Stefan Eissing]



git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x@1856935 13f79535-47bb-0310-9956-ffa450edef68
asfgit pushed a commit to apache/httpd that referenced this issue Oct 5, 2019
…s. We want our

     non-privilegded apache user to be able to read them. See github issue
     <icing/mod_md#117>. [Stefan Eissing] 



git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1855777 13f79535-47bb-0310-9956-ffa450edef68
asfgit pushed a commit to apache/httpd that referenced this issue Oct 5, 2019
…g restrictions in

     spite of umask. Fixes <icing/mod_md#117>. [Stefan Eissing]



git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1856297 13f79535-47bb-0310-9956-ffa450edef68
tlhackque pushed a commit to tlhackque/mod_md that referenced this issue Mar 6, 2020
tlhackque pushed a commit to tlhackque/mod_md that referenced this issue Mar 6, 2020
…nt our

     non-privilegded apache user to be able to read them. See github issue
     <icing#117>.
  *) Merged some default store dir change from trunk.
  *) Merged unread var removal from trunk.
tlhackque pushed a commit to tlhackque/mod_md that referenced this issue Mar 6, 2020
     specified permissions on all files created in md store.
tlhackque pushed a commit to tlhackque/mod_md that referenced this issue Mar 6, 2020
----------------------------------------------------------------------------------------------------
 * Added MD section to Apache's "server-status" page listing all managed domains and their
   settings, renewal status and error count.
 * Fix for icing#117, explicitly set file permissions to work around umask defaults.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants