New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Augeas fails to parse RewriteRule and FilesMatch regexps (incl. Drupal .htaccess files) #1531

Closed
kristofferwiklund opened this Issue Nov 17, 2015 · 23 comments

Comments

Projects
None yet
6 participants
@kristofferwiklund

kristofferwiklund commented Nov 17, 2015

I am trying running the following command:
./letsencrypt-auto --server https://acme-v01.api.letsencrypt.org/directory --help

And I get:
Updating letsencrypt and virtual environment dependencies.......
Running with virtualenv: sudo /home/koffe/.local/share/letsencrypt/bin/letsencrypt --apache --server https://acme-v01.api.letsencrypt.org/directory --agree-dev-preview
The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/var/aegir/platforms/20151024/.htaccess', u'Syntax error'),)

With a verbose output:
Requested authenticator apache and installer apache
Other error:(PluginEntryPoint#apache): ('There has been an error in parsing the file (%s): %s', u'/var/aegir/platforms/20151024/.htaccess', u'Syntax error')
Traceback (most recent call last):
File "/home/koffe/.local/share/letsencrypt/local/lib/python2.7/site-packages/letsencrypt/plugins/disco.py", line 103, in prepare self._initialized.prepare()
File "/home/koffe/.local/share/letsencrypt/local/lib/python2.7/site-packages/letsencrypt_apache/configurator.py", line 152, in prepare self.check_parsing_errors("httpd.aug")
File "/home/koffe/.local/share/letsencrypt/local/lib/python2.7/site-packages/letsencrypt_apache/augeas_configurator.py", line 68, in check_parsing_errors raise errors.PluginError(msg)
PluginError: ('There has been an error in parsing the file (%s): %s', u'/var/aegir/platforms/20151024/.htaccess', u'Syntax error')

Here is the .htaccess file
htaccess.txt

@bmw bmw added the area: apache label Nov 18, 2015

@anarcat

This comment has been minimized.

Show comment
Hide comment
@anarcat

anarcat Nov 18, 2015

this is likely #1294 - i had the same issue and in there they say it was fixed in master. yet i have tried to run the dev version as explained in:

https://letsencrypt.readthedocs.org/en/latest/contributing.html#hacking

.. but i got the same error. either the bug is not fixed or the install instructions are too complicated or it's time to make a release (or all of the above).

anarcat commented Nov 18, 2015

this is likely #1294 - i had the same issue and in there they say it was fixed in master. yet i have tried to run the dev version as explained in:

https://letsencrypt.readthedocs.org/en/latest/contributing.html#hacking

.. but i got the same error. either the bug is not fixed or the install instructions are too complicated or it's time to make a release (or all of the above).

@anarcat

This comment has been minimized.

Show comment
Hide comment
@anarcat

anarcat Nov 18, 2015

yep - i tried flushing the venv and re-running the above instructions, i still get the bug:

http://paste.debian.net/333539/

$ git show
commit fc07238804314ae3b29d915e40756e5d3e4d0908
Merge: f265179 ec58ad2
Author: Peter Eckersley <pde@users.noreply.github.com>
Date:   Mon Nov 16 12:19:55 2015 -0800

    Merge pull request #1516 from henrychen95/master

    Fix Amazon Linux bootstrapping error.

anarcat commented Nov 18, 2015

yep - i tried flushing the venv and re-running the above instructions, i still get the bug:

http://paste.debian.net/333539/

$ git show
commit fc07238804314ae3b29d915e40756e5d3e4d0908
Merge: f265179 ec58ad2
Author: Peter Eckersley <pde@users.noreply.github.com>
Date:   Mon Nov 16 12:19:55 2015 -0800

    Merge pull request #1516 from henrychen95/master

    Fix Amazon Linux bootstrapping error.

@kristofferwiklund

This comment has been minimized.

Show comment
Hide comment
@kristofferwiklund

kristofferwiklund Nov 18, 2015

I am running on master already. Doing a "git pull" in ~/letsencrypt.
I am on this commit: 457be44

Still same problem.

kristofferwiklund commented Nov 18, 2015

I am running on master already. Doing a "git pull" in ~/letsencrypt.
I am on this commit: 457be44

Still same problem.

@anarcat

This comment has been minimized.

Show comment
Hide comment
@anarcat

anarcat Nov 19, 2015

i wonder if this is not because of RewriteRules, which are both in Drupal's .htaccess and my specific config. This is my config:

# Apache config file for anarcat.wiki.orangeseeds.org
# Automatically generated by ikisite; do not modify directly.

<VirtualHost *:80>
        ServerAdmin root@localhost
        ServerName anarcat.wiki.orangeseeds.org:80


        SuexecUserGroup w-anarcat w-anarcat


        UserDir disabled

        RewriteEngine On
        RewriteRule ^/(.*) http\:\/\/anarc\.at\/$1 [L,R,NE]

        ErrorLog /var/log/ikiwiki-hosting/w-anarcat/error.log
        LogLevel warn
        CustomLog /var/log/ikiwiki-hosting/w-anarcat/access.log combined
</VirtualHost>

i am also on 457be44 now.

anarcat commented Nov 19, 2015

i wonder if this is not because of RewriteRules, which are both in Drupal's .htaccess and my specific config. This is my config:

# Apache config file for anarcat.wiki.orangeseeds.org
# Automatically generated by ikisite; do not modify directly.

<VirtualHost *:80>
        ServerAdmin root@localhost
        ServerName anarcat.wiki.orangeseeds.org:80


        SuexecUserGroup w-anarcat w-anarcat


        UserDir disabled

        RewriteEngine On
        RewriteRule ^/(.*) http\:\/\/anarc\.at\/$1 [L,R,NE]

        ErrorLog /var/log/ikiwiki-hosting/w-anarcat/error.log
        LogLevel warn
        CustomLog /var/log/ikiwiki-hosting/w-anarcat/access.log combined
</VirtualHost>

i am also on 457be44 now.

@pde pde added this to the Nice for 1.0 milestone Nov 21, 2015

@pde

This comment has been minimized.

Show comment
Hide comment
@pde

pde Nov 21, 2015

Member

@domcleal Is this an augeas lens bug?

Member

pde commented Nov 21, 2015

@domcleal Is this an augeas lens bug?

@pde

This comment has been minimized.

Show comment
Hide comment
@pde

pde Nov 21, 2015

Member

Probably an instance of hercules-team/augeas#307

Member

pde commented Nov 21, 2015

Probably an instance of hercules-team/augeas#307

@pde pde changed the title from Syntax error on Drupal .htaccess to Augeas fails to parse RewriteRule and FilesMatch regexps (incl. Drupal .htaccess files) Nov 22, 2015

@domcleal

This comment has been minimized.

Show comment
Hide comment
@domcleal

domcleal Nov 23, 2015

Collaborator

Yep, it's that.

Collaborator

domcleal commented Nov 23, 2015

Yep, it's that.

@pde pde modified the milestones: 1.0 for launch, Nice for 1.0 Nov 24, 2015

@pde

This comment has been minimized.

Show comment
Hide comment
@pde

pde Nov 27, 2015

Member

The Drupal .htaccess case is hercules-team/augeas/issues/324.

Member

pde commented Nov 27, 2015

The Drupal .htaccess case is hercules-team/augeas/issues/324.

@pde

This comment has been minimized.

Show comment
Hide comment
@pde

pde Dec 2, 2015

Member

We're waiting for the augeas team to review their PR for this; will ship it when it's reviewed. Meanwhile, kicking the remainder of this issue out of 1.0 for launch.

Member

pde commented Dec 2, 2015

We're waiting for the augeas team to review their PR for this; will ship it when it's reviewed. Meanwhile, kicking the remainder of this issue out of 1.0 for launch.

@pde pde modified the milestones: Nice for 1.0, 1.0 for launch Dec 2, 2015

@domcleal

This comment has been minimized.

Show comment
Hide comment
@domcleal

domcleal Dec 2, 2015

Collaborator

@pde sorry about that, merged as hercules-team/augeas@cb85b59.

Collaborator

domcleal commented Dec 2, 2015

@pde sorry about that, merged as hercules-team/augeas@cb85b59.

@kristofferwiklund

This comment has been minimized.

Show comment
Hide comment
@kristofferwiklund

kristofferwiklund Dec 8, 2015

Running ./letsencrypt-auto --apache with the latest version of letsencrypt (commit 3838ea4)

thecomputer ~/letsencrypt (master): ./letsencrypt-auto --apache
Updating letsencrypt and virtual environment dependencies.......
Running with virtualenv: sudo /home/theuser/.local/share/letsencrypt/bin/letsencrypt --apache
The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/var/aegir/platforms/20151024/.htaccess', u'Syntax error'),)

So it is not working. This is still a issue.

kristofferwiklund commented Dec 8, 2015

Running ./letsencrypt-auto --apache with the latest version of letsencrypt (commit 3838ea4)

thecomputer ~/letsencrypt (master): ./letsencrypt-auto --apache
Updating letsencrypt and virtual environment dependencies.......
Running with virtualenv: sudo /home/theuser/.local/share/letsencrypt/bin/letsencrypt --apache
The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/var/aegir/platforms/20151024/.htaccess', u'Syntax error'),)

So it is not working. This is still a issue.

@domcleal

This comment has been minimized.

Show comment
Hide comment
@domcleal

domcleal Dec 8, 2015

Collaborator

While I've not tried the LE client, the lens currently in 3838ea4 seems to be parsing the htaccess file linked in the description just fine, and correctly fails on the previous commit.

Collaborator

domcleal commented Dec 8, 2015

While I've not tried the LE client, the lens currently in 3838ea4 seems to be parsing the htaccess file linked in the description just fine, and correctly fails on the previous commit.

@acrollet

This comment has been minimized.

Show comment
Hide comment
@acrollet

acrollet Dec 8, 2015

@kristofferwiklund - I had trouble with this too - you need to be sure that letsencrypt-apache/letsencrypt_apache/augeas_lens/httpd.aug has been updated.

acrollet commented Dec 8, 2015

@kristofferwiklund - I had trouble with this too - you need to be sure that letsencrypt-apache/letsencrypt_apache/augeas_lens/httpd.aug has been updated.

@pde pde modified the milestones: 0.1.1, Nice for 1.0 Dec 8, 2015

@kristofferwiklund

This comment has been minimized.

Show comment
Hide comment
@kristofferwiklund

kristofferwiklund Dec 9, 2015

httpd.aug had been update yesterday. But still problems running commit ce14851

kristofferwiklund commented Dec 9, 2015

httpd.aug had been update yesterday. But still problems running commit ce14851

@domcleal

This comment has been minimized.

Show comment
Hide comment
@domcleal

domcleal Dec 9, 2015

Collaborator

Can you pastebin the exact file so I can double check it's parsing with the lens itself? We may be able to narrow down then whether it's another issue in the lens or if it's a problem in how it's being run.

If you can follow these steps too, we can check it's parsing on your computer by doing:

  1. download augcheck from https://raw.githubusercontent.com/raphink/augeas-sandbox/master/augcheck
  2. ensure you have augparse installed (augeas-tools package on Debian/Ubuntu, augeas on RPM distros)
  3. from the LE directory, run: AUGEAS_LENS_LIB=letsencrypt-apache/letsencrypt_apache/augeas_lens/ augcheck /var/aegir/platforms/20151024/.htaccess Httpd
Collaborator

domcleal commented Dec 9, 2015

Can you pastebin the exact file so I can double check it's parsing with the lens itself? We may be able to narrow down then whether it's another issue in the lens or if it's a problem in how it's being run.

If you can follow these steps too, we can check it's parsing on your computer by doing:

  1. download augcheck from https://raw.githubusercontent.com/raphink/augeas-sandbox/master/augcheck
  2. ensure you have augparse installed (augeas-tools package on Debian/Ubuntu, augeas on RPM distros)
  3. from the LE directory, run: AUGEAS_LENS_LIB=letsencrypt-apache/letsencrypt_apache/augeas_lens/ augcheck /var/aegir/platforms/20151024/.htaccess Httpd
@kristofferwiklund

This comment has been minimized.

Show comment
Hide comment
@kristofferwiklund

kristofferwiklund Dec 9, 2015

The .htaccess file is the same as the reported (https://github.com/letsencrypt/letsencrypt/files/36394/htaccess.txt)

I have runned the commands provide:
augcheck.result.txt

Seems to be working standalone.

kristofferwiklund commented Dec 9, 2015

The .htaccess file is the same as the reported (https://github.com/letsencrypt/letsencrypt/files/36394/htaccess.txt)

I have runned the commands provide:
augcheck.result.txt

Seems to be working standalone.

@kristofferwiklund

This comment has been minimized.

Show comment
Hide comment
@kristofferwiklund

kristofferwiklund Dec 9, 2015

Log files says:

2015-12-09 21:57:32,099:DEBUG:letsencrypt.cli:Root logging level set at 30
2015-12-09 21:57:32,100:INFO:letsencrypt.cli:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2015-12-09 21:57:32,100:DEBUG:letsencrypt.cli:letsencrypt version: 0.1.0
2015-12-09 21:57:32,100:DEBUG:letsencrypt.cli:Arguments: ['--apache']
2015-12-09 21:57:32,100:DEBUG:letsencrypt.cli:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#webroot,PluginEntryPoint#null,Plug
2015-12-09 21:57:32,103:DEBUG:letsencrypt.cli:Requested authenticator apache and installer apache
2015-12-09 21:57:32,341:DEBUG:letsencrypt.plugins.disco:Other error:(PluginEntryPoint#apache): ('There has been an error in parsing the file (%s): %s', u'/va
Traceback (most recent call last):
File "/home/koffe/.local/share/letsencrypt/local/lib/python2.7/site-packages/letsencrypt/plugins/disco.py", line 103, in prepare
self._initialized.prepare()
File "/home/koffe/.local/share/letsencrypt/local/lib/python2.7/site-packages/letsencrypt_apache/configurator.py", line 150, in prepare
self.check_parsing_errors("httpd.aug")
File "/home/koffe/.local/share/letsencrypt/local/lib/python2.7/site-packages/letsencrypt_apache/augeas_configurator.py", line 68, in check_parsing_errors
raise errors.PluginError(msg)
PluginError: ('There has been an error in parsing the file (%s): %s', u'/var/aegir/platforms/20151024/.htaccess', u'Syntax error')
2015-12-09 21:57:32,341:DEBUG:letsencrypt.display.ops:No candidate plugin
2015-12-09 21:57:32,341:DEBUG:letsencrypt.cli:Selected authenticator None and installer None

kristofferwiklund commented Dec 9, 2015

Log files says:

2015-12-09 21:57:32,099:DEBUG:letsencrypt.cli:Root logging level set at 30
2015-12-09 21:57:32,100:INFO:letsencrypt.cli:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2015-12-09 21:57:32,100:DEBUG:letsencrypt.cli:letsencrypt version: 0.1.0
2015-12-09 21:57:32,100:DEBUG:letsencrypt.cli:Arguments: ['--apache']
2015-12-09 21:57:32,100:DEBUG:letsencrypt.cli:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#webroot,PluginEntryPoint#null,Plug
2015-12-09 21:57:32,103:DEBUG:letsencrypt.cli:Requested authenticator apache and installer apache
2015-12-09 21:57:32,341:DEBUG:letsencrypt.plugins.disco:Other error:(PluginEntryPoint#apache): ('There has been an error in parsing the file (%s): %s', u'/va
Traceback (most recent call last):
File "/home/koffe/.local/share/letsencrypt/local/lib/python2.7/site-packages/letsencrypt/plugins/disco.py", line 103, in prepare
self._initialized.prepare()
File "/home/koffe/.local/share/letsencrypt/local/lib/python2.7/site-packages/letsencrypt_apache/configurator.py", line 150, in prepare
self.check_parsing_errors("httpd.aug")
File "/home/koffe/.local/share/letsencrypt/local/lib/python2.7/site-packages/letsencrypt_apache/augeas_configurator.py", line 68, in check_parsing_errors
raise errors.PluginError(msg)
PluginError: ('There has been an error in parsing the file (%s): %s', u'/var/aegir/platforms/20151024/.htaccess', u'Syntax error')
2015-12-09 21:57:32,341:DEBUG:letsencrypt.display.ops:No candidate plugin
2015-12-09 21:57:32,341:DEBUG:letsencrypt.cli:Selected authenticator None and installer None

@kristofferwiklund

This comment has been minimized.

Show comment
Hide comment
@kristofferwiklund

kristofferwiklund Dec 9, 2015

I am testing to copy the .htaccess file to another server, that is not running Drupal sites (and the Aegir setup). And there the parsing works, just get some other error
But that is not related to this. Its about the verification and I get the error: The client lacks sufficient authorization :: Correct zName not found for TLS SNI challenge.

So there is something in the setup of Apache that triggers a Syntax error for the htaccess file.

kristofferwiklund commented Dec 9, 2015

I am testing to copy the .htaccess file to another server, that is not running Drupal sites (and the Aegir setup). And there the parsing works, just get some other error
But that is not related to this. Its about the verification and I get the error: The client lacks sufficient authorization :: Correct zName not found for TLS SNI challenge.

So there is something in the setup of Apache that triggers a Syntax error for the htaccess file.

pde added a commit that referenced this issue Dec 10, 2015

@domcleal

This comment has been minimized.

Show comment
Hide comment
@domcleal

domcleal Dec 10, 2015

Collaborator

Are /home/koffe/.local/share/letsencrypt/local/lib/python2.7/site-packages/letsencrypt_apache/augeas_lens/httpd.aug and letsencrypt-apache/letsencrypt_apache/augeas_lens/httpd.aug identical (compare the md5sum)?

A quick test here shows that it might not get updated in ~/.local when you run ./letsencrypt-auto from the git checkout.

Collaborator

domcleal commented Dec 10, 2015

Are /home/koffe/.local/share/letsencrypt/local/lib/python2.7/site-packages/letsencrypt_apache/augeas_lens/httpd.aug and letsencrypt-apache/letsencrypt_apache/augeas_lens/httpd.aug identical (compare the md5sum)?

A quick test here shows that it might not get updated in ~/.local when you run ./letsencrypt-auto from the git checkout.

@kristofferwiklund

This comment has been minimized.

Show comment
Hide comment
@kristofferwiklund

kristofferwiklund Dec 10, 2015

They aren't.. :)

Should I just delete the .local folder and hope letsencrypt recreates it. Or is the some update command that can be run for ./letsencrypt-auto

kristofferwiklund commented Dec 10, 2015

They aren't.. :)

Should I just delete the .local folder and hope letsencrypt recreates it. Or is the some update command that can be run for ./letsencrypt-auto

@domcleal

This comment has been minimized.

Show comment
Hide comment
@domcleal

domcleal Dec 10, 2015

Collaborator

It should be updating already, so it's a bug somewhere (it looks like it's running pip install to update it, but perhaps not registering that the file changed). I'd probably just copy the file for now as .local may contain things other than LE.

Edit: correction, you could probably delete ~/.local/share/letsencrypt safely and it'll reinstall everything.

It appears to me that letsencrypt-auto is only designed to install the latest released version, not the latest contents of a git checkout.

Collaborator

domcleal commented Dec 10, 2015

It should be updating already, so it's a bug somewhere (it looks like it's running pip install to update it, but perhaps not registering that the file changed). I'd probably just copy the file for now as .local may contain things other than LE.

Edit: correction, you could probably delete ~/.local/share/letsencrypt safely and it'll reinstall everything.

It appears to me that letsencrypt-auto is only designed to install the latest released version, not the latest contents of a git checkout.

@kristofferwiklund

This comment has been minimized.

Show comment
Hide comment
@kristofferwiklund

kristofferwiklund Dec 10, 2015

Now that is working. But it found some more problems with a Drupal 6 site. But for this I can understand it.

Here is the Drupal 6.37 .htaccess file: http://cgit.drupalcode.org/drupal/tree/.htaccess?h=6.37
On line 22 the ending " is missing. Some type of syntax hack for Apache 1.3

Drupal 6 is EOL in February 2016 so people still using it have to patch their on .htaccess file.

Adding in the " and letsencrypt is proceeding. Now I just get "Correct zName not found for TLS SNI challenge.". But that is for an other issue, if Google is not helping me.

kristofferwiklund commented Dec 10, 2015

Now that is working. But it found some more problems with a Drupal 6 site. But for this I can understand it.

Here is the Drupal 6.37 .htaccess file: http://cgit.drupalcode.org/drupal/tree/.htaccess?h=6.37
On line 22 the ending " is missing. Some type of syntax hack for Apache 1.3

Drupal 6 is EOL in February 2016 so people still using it have to patch their on .htaccess file.

Adding in the " and letsencrypt is proceeding. Now I just get "Correct zName not found for TLS SNI challenge.". But that is for an other issue, if Google is not helping me.

@domcleal

This comment has been minimized.

Show comment
Hide comment
@domcleal

domcleal Dec 10, 2015

Collaborator

Thanks, the missing double quote is bug #1724.

Collaborator

domcleal commented Dec 10, 2015

Thanks, the missing double quote is bug #1724.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment