New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MentosError header errors #45

Open
aprescott opened this Issue Oct 18, 2012 · 33 comments

Comments

Projects
None yet
@aprescott
1.9.3p286 :001 > require "pygments"
 => true 
1.9.3p286 :002 > Pygments.highlight("foo")
MentosError: Failed to get header.
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:351:in `rescue in get_header'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:332:in `get_header'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:229:in `block in mentos'
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:203:in `mentos'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:186:in `highlight'
    from (irb):2
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/bin/irb:16:in `<main>'

As far as I can tell, this is happening on a clean gem environment.

Curiously, installing python-pygments and calling Pygmentize.start("pygmentize") first gives, for one single subsequent call to highlight a slightly different message:

1.9.3p286 :003 > Pygments.start("pygmentize"); Pygments.highlight("foo")
MentosError: No header received back.
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:295:in `handle_header_and_return'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:232:in `block in mentos'
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:203:in `mentos'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:186:in `highlight'
    from (irb):3
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/bin/irb:16:in `<main>'

1.9.3p286 :004 > Pygments.highlight("foo")
MentosError: Failed to get header.
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:351:in `rescue in get_header'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:332:in `get_header'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:229:in `block in mentos'
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:203:in `mentos'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:186:in `highlight'
    from (irb):4
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/bin/irb:16:in `<main>'

@tnm

This comment has been minimized.

Show comment
Hide comment
@tnm

tnm Oct 20, 2012

Collaborator

Can you let me know what version of Python you are running on this machine?

Collaborator

tnm commented Oct 20, 2012

Can you let me know what version of Python you are running on this machine?

@aprescott

This comment has been minimized.

Show comment
Hide comment
@aprescott

aprescott Oct 20, 2012

python --version is 3.3.0.

python --version is 3.3.0.

@DirtYiCE

This comment has been minimized.

Show comment
Hide comment
@DirtYiCE

DirtYiCE Oct 20, 2012

I can confirm the same bug with Python 3.2.3, however it works fine with Python 2.7.3 (both on Gentoo Linux). Probably a Python 3 issue.

I can confirm the same bug with Python 3.2.3, however it works fine with Python 2.7.3 (both on Gentoo Linux). Probably a Python 3 issue.

@aprescott

This comment has been minimized.

Show comment
Hide comment
@aprescott

aprescott Oct 20, 2012

What's the best way for me to specify a different executable to pygments.rb in the meantime? I have python2 on my PATH.

What's the best way for me to specify a different executable to pygments.rb in the meantime? I have python2 on my PATH.

@aprescott

This comment has been minimized.

Show comment
Hide comment
@aprescott

aprescott Oct 20, 2012

(The README used to mention, e.g., RubyPython.configure :python_exe => 'python2.7', but it seems 0.3 has moved to a different IPC system.)

(The README used to mention, e.g., RubyPython.configure :python_exe => 'python2.7', but it seems 0.3 has moved to a different IPC system.)

@tnm

This comment has been minimized.

Show comment
Hide comment
@tnm

tnm Dec 7, 2012

Collaborator

We're officially only supporting 2.5, 2.6, and 2.7 for now. I'm planning support for Python 3, and patches are welcome for now.

Collaborator

tnm commented Dec 7, 2012

We're officially only supporting 2.5, 2.6, and 2.7 for now. I'm planning support for Python 3, and patches are welcome for now.

@aprescott

This comment has been minimized.

Show comment
Hide comment
@aprescott

aprescott Dec 7, 2012

Good to know, thanks. My ~/bin/python + PATH workaround is working for now.

Good to know, thanks. My ~/bin/python + PATH workaround is working for now.

@matthiasbeyer

This comment has been minimized.

Show comment
Hide comment
@matthiasbeyer

matthiasbeyer Dec 16, 2012

Having the same issue here! python2 and python2.7 are in my path! What to do?

Having the same issue here! python2 and python2.7 are in my path! What to do?

@Ivoz

This comment has been minimized.

Show comment
Hide comment
@Ivoz

Ivoz Jan 21, 2013

pygments should not assume that the system default python is going to be v2.x and not v3.x

It should be easy enough to find if a python2 exists on the path, and use that instead, no? Given that v2.x is now pretty much at EOL.

Ivoz commented Jan 21, 2013

pygments should not assume that the system default python is going to be v2.x and not v3.x

It should be easy enough to find if a python2 exists on the path, and use that instead, no? Given that v2.x is now pretty much at EOL.

@obsoke

This comment has been minimized.

Show comment
Hide comment
@obsoke

obsoke Jan 24, 2013

I've been getting this error as well in Jekyll. Using pygments 0.3.7. I have 'python2' aliased to python & in my path.

Has anyone managed to resolve this?

edit: Editing lib/pygments/mentos.py's first line from 'python' to 'python2' seemed to fix it for me.

obsoke commented Jan 24, 2013

I've been getting this error as well in Jekyll. Using pygments 0.3.7. I have 'python2' aliased to python & in my path.

Has anyone managed to resolve this?

edit: Editing lib/pygments/mentos.py's first line from 'python' to 'python2' seemed to fix it for me.

@aprescott

This comment has been minimized.

Show comment
Hide comment
@aprescott

aprescott Jan 25, 2013

Has anyone managed to resolve this?

I just created ~/bin/python as a symlink to python2. ~/bin is in my PATH. I didn't need to edit anything in mentos.py.

Has anyone managed to resolve this?

I just created ~/bin/python as a symlink to python2. ~/bin is in my PATH. I didn't need to edit anything in mentos.py.

@mattwildig

This comment has been minimized.

Show comment
Hide comment
@mattwildig

mattwildig Mar 28, 2013

I’ve got the same error, but I’m using Python 2.7. As far as I know I don’t have Python 3 installed on my machine at all (Mac 10.6, Python installed via Homebrew):

1.9.3-p385 :001 > require 'pygments'
 => true 
1.9.3-p385 :002 > Pygments.highlight 'foo'
MentosError: Failed to get header.
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:357:in `rescue in get_header'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:338:in `get_header'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:235:in `block in mentos'
        from /Users/matt/.rvm/rubies/ruby-1.9.3-p385/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:209:in `mentos'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:192:in `highlight'
        from (irb):2
        from /Users/matt/.rvm/rubies/ruby-1.9.3-p385/bin/irb:16:in `<main>'

In order to try and track the issue down I tried running the mentos.py script independently. It failed with the error Illegal instruction, which seems to be produced by the line os.close(fd). Commenting out this line (actually the whole for block so the indentation is right) clears the error, and commenting the same block in the gem itself seems to fix it:

1.9.3-p385 :001 > require 'pygments'
 => true 
1.9.3-p385 :002 > Pygments.highlight 'foo'
 => "<div class=\"highlight\"><pre><span class=\"n\">foo</span>\n</pre></div>" 

Can anyone cast any light on this?

I’ve got the same error, but I’m using Python 2.7. As far as I know I don’t have Python 3 installed on my machine at all (Mac 10.6, Python installed via Homebrew):

1.9.3-p385 :001 > require 'pygments'
 => true 
1.9.3-p385 :002 > Pygments.highlight 'foo'
MentosError: Failed to get header.
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:357:in `rescue in get_header'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:338:in `get_header'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:235:in `block in mentos'
        from /Users/matt/.rvm/rubies/ruby-1.9.3-p385/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:209:in `mentos'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:192:in `highlight'
        from (irb):2
        from /Users/matt/.rvm/rubies/ruby-1.9.3-p385/bin/irb:16:in `<main>'

In order to try and track the issue down I tried running the mentos.py script independently. It failed with the error Illegal instruction, which seems to be produced by the line os.close(fd). Commenting out this line (actually the whole for block so the indentation is right) clears the error, and commenting the same block in the gem itself seems to fix it:

1.9.3-p385 :001 > require 'pygments'
 => true 
1.9.3-p385 :002 > Pygments.highlight 'foo'
 => "<div class=\"highlight\"><pre><span class=\"n\">foo</span>\n</pre></div>" 

Can anyone cast any light on this?

@tnm

This comment has been minimized.

Show comment
Hide comment
@tnm

tnm Mar 28, 2013

Collaborator

@mattwildig Thanks for that. That can be fixed. I'll investigate.

Collaborator

tnm commented Mar 28, 2013

@mattwildig Thanks for that. That can be fixed. I'll investigate.

@mattwildig

This comment has been minimized.

Show comment
Hide comment
@mattwildig

mattwildig Apr 19, 2013

@tnm It appears that this error has fixed itself. Whilst I could reliably reproduce it when I reported it, it is no longer appearing. Unfortunately I don’t know what changed between than and now that could affect this. I did restart my machine recently (after installing updates) so that might have cleared something in the OS.

@tnm It appears that this error has fixed itself. Whilst I could reliably reproduce it when I reported it, it is no longer appearing. Unfortunately I don’t know what changed between than and now that could affect this. I did restart my machine recently (after installing updates) so that might have cleared something in the OS.

@ekampp

This comment has been minimized.

Show comment
Hide comment
@ekampp

ekampp May 28, 2013

I have this problem appear intermittent on Heroku, when attempting to parse some markdown. The problem seems to be spread across all OS, browsers and countries, and there is no discernible pattern. Here is an example error:

MentosError: File "/app/vendor/bundle/ruby/2.0.0/gems/pygments.rb-0.5.0/lib/pygments/mentos.py", line 282, 
in start header = json.loads(line) File "/usr/local/lib/python2.7/json/__init__.py", line 326, in loads return 
_default_decoder.decode(s) File "/usr/local/lib/python2.7/json/decoder.py", line 369, in decode raise 
ValueError(errmsg("Extra data", s, end, len(s))) ValueError: Extra data: line 1 column 1 - line 1 column 129 
(char 1 - 129)

ekampp commented May 28, 2013

I have this problem appear intermittent on Heroku, when attempting to parse some markdown. The problem seems to be spread across all OS, browsers and countries, and there is no discernible pattern. Here is an example error:

MentosError: File "/app/vendor/bundle/ruby/2.0.0/gems/pygments.rb-0.5.0/lib/pygments/mentos.py", line 282, 
in start header = json.loads(line) File "/usr/local/lib/python2.7/json/__init__.py", line 326, in loads return 
_default_decoder.decode(s) File "/usr/local/lib/python2.7/json/decoder.py", line 369, in decode raise 
ValueError(errmsg("Extra data", s, end, len(s))) ValueError: Extra data: line 1 column 1 - line 1 column 129 
(char 1 - 129)
@radum

This comment has been minimized.

Show comment
Hide comment
@radum

radum Jun 13, 2013

For Windows users, Python can run with multiple versions without any problem, so just install Python 2 and 3 and just change the env path. I know it is not really useful but could get you started at least.

radum commented Jun 13, 2013

For Windows users, Python can run with multiple versions without any problem, so just install Python 2 and 3 and just change the env path. I know it is not really useful but could get you started at least.

@tschaub

This comment has been minimized.

Show comment
Hide comment
@tschaub

tschaub Jun 25, 2013

@mattwildig - thanks for posting your fix here. I was getting the same with Ruby 1.8, Pygments 0.5.0, and Python 2.6.1. Commenting out the relevant block in mentos.py fixed the issue for me (and now Jekyll works again - see jekyll/jekyll#1181).

tschaub commented Jun 25, 2013

@mattwildig - thanks for posting your fix here. I was getting the same with Ruby 1.8, Pygments 0.5.0, and Python 2.6.1. Commenting out the relevant block in mentos.py fixed the issue for me (and now Jekyll works again - see jekyll/jekyll#1181).

@milinda

This comment has been minimized.

Show comment
Hide comment
@milinda

milinda Jul 6, 2013

I got the same errror in Mountain Lion with Ruby 2.0.0p0 and Python 2.7.2. But it was not coming from the code block mentioned above. It was coming from ruby-2.0.0-p0/gems/pygments.rb-0.5.1/lib/pygments/popen.rb:366.

It says "in `rescue in get_header': Failed to get header. (MentosError)". Please let me know if any of you have any idea about this.

milinda commented Jul 6, 2013

I got the same errror in Mountain Lion with Ruby 2.0.0p0 and Python 2.7.2. But it was not coming from the code block mentioned above. It was coming from ruby-2.0.0-p0/gems/pygments.rb-0.5.1/lib/pygments/popen.rb:366.

It says "in `rescue in get_header': Failed to get header. (MentosError)". Please let me know if any of you have any idea about this.

@milinda

This comment has been minimized.

Show comment
Hide comment
@milinda

milinda Jul 6, 2013

In my case popen.rb's get_header method's size[0..-2] failed saying

"
undefined method `[]' for nil:NilClass
"

It looks like size = @out.read(33) returns a null. This may be caused by a different error.

milinda commented Jul 6, 2013

In my case popen.rb's get_header method's size[0..-2] failed saying

"
undefined method `[]' for nil:NilClass
"

It looks like size = @out.read(33) returns a null. This may be caused by a different error.

@milinda

This comment has been minimized.

Show comment
Hide comment
@milinda

milinda Jul 9, 2013

Finally I found the cause and a solution for my issue. It was due to corrupted python installation on Mac OS X Mountain Lion. pygmentize was throwing errors due to a permission issue.

milinda commented Jul 9, 2013

Finally I found the cause and a solution for my issue. It was due to corrupted python installation on Mac OS X Mountain Lion. pygmentize was throwing errors due to a permission issue.

@millisami

This comment has been minimized.

Show comment
Hide comment
@millisami

millisami Jul 14, 2013

@milinda So wats the fix?

@milinda So wats the fix?

@vikstrous

This comment has been minimized.

Show comment
Hide comment
@vikstrous

vikstrous Aug 30, 2013

I ran into this too. This guy's fix worked temporarily, but it would be great if there was a real fix ;).
http://www.nonsenseby.me/blog/2013/04/13/arch-linux/

I ran into this too. This guy's fix worked temporarily, but it would be great if there was a real fix ;).
http://www.nonsenseby.me/blog/2013/04/13/arch-linux/

@Ivoz

This comment has been minimized.

Show comment
Hide comment
@Ivoz

Ivoz Aug 30, 2013

You can also run the tool in a python virtualenv that uses python2, for a workaround.

Ivoz commented Aug 30, 2013

You can also run the tool in a python virtualenv that uses python2, for a workaround.

@cdgz

This comment has been minimized.

Show comment
Hide comment
@cdgz

cdgz Nov 22, 2013

Just to confirm that issue persists in 0.5.4 with ruby 1.8.7 and python 2.7.2 on SuSE Linux (3.0.13-0.27)

With python 2.6.7 Ok.

cdgz commented Nov 22, 2013

Just to confirm that issue persists in 0.5.4 with ruby 1.8.7 and python 2.7.2 on SuSE Linux (3.0.13-0.27)

With python 2.6.7 Ok.

@phansch

This comment has been minimized.

Show comment
Hide comment
@phansch

phansch Dec 7, 2013

I had the same error but it seems for me the cause was something different? Running mentos.py gave me the following exception:

$ python mentos.py
Traceback (most recent call last):
  File "mentos.py", line 17, in <module>
    from pygments import lexers, formatters, styles, filters
  File "/home/philipp/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/gems/2.0.0/gems/pygments.rb-0.5.0/vendor/pygments-main/pygments/lexers/__init__.py", line 18, in <module>
    from pygments.plugin import find_plugin_lexers
  File "/home/philipp/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/gems/2.0.0/gems/pygments.rb-0.5.0/vendor/pygments-main/pygments/plugin.py", line 39, in <module>
    import pkg_resources
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2823, in <module>
    add_activation_listener(lambda dist: dist.activate())
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 710, in subscribe
    callback(dist)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2823, in <lambda>
    add_activation_listener(lambda dist: dist.activate())
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2255, in activate
    self.insert_on(path)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2362, in insert_on
    self.check_version_conflict()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2401, in check_version_conflict
    for modname in self._get_metadata('top_level.txt'):
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2249, in _get_metadata
    for line in self.get_metadata_lines(name):
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1219, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1211, in get_metadata
    return self._get(self._fn(self.egg_info,name))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1326, in _get
    stream = open(path, 'rb')
IOError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/oauthlib-0.6.0-py2.7.egg/EGG-INFO/top_level.txt'

Some time ago I installed a package via sudo easy_install. I don't use the python setuptools very often so maybe I did something wrong there.
For some reason mentos.py insisted on using the oauth package in /usr/local/lib/python2.7/dist-packages/... I removed that (sudo rm -rf /usr/local/lib/python2.7/dist-packages/oauthlib-0.6.0-py2.7.egg) and now jekyll is running again. Now, this is just a workaround because that probably breaks the python package, right?

phansch commented Dec 7, 2013

I had the same error but it seems for me the cause was something different? Running mentos.py gave me the following exception:

$ python mentos.py
Traceback (most recent call last):
  File "mentos.py", line 17, in <module>
    from pygments import lexers, formatters, styles, filters
  File "/home/philipp/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/gems/2.0.0/gems/pygments.rb-0.5.0/vendor/pygments-main/pygments/lexers/__init__.py", line 18, in <module>
    from pygments.plugin import find_plugin_lexers
  File "/home/philipp/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/gems/2.0.0/gems/pygments.rb-0.5.0/vendor/pygments-main/pygments/plugin.py", line 39, in <module>
    import pkg_resources
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2823, in <module>
    add_activation_listener(lambda dist: dist.activate())
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 710, in subscribe
    callback(dist)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2823, in <lambda>
    add_activation_listener(lambda dist: dist.activate())
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2255, in activate
    self.insert_on(path)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2362, in insert_on
    self.check_version_conflict()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2401, in check_version_conflict
    for modname in self._get_metadata('top_level.txt'):
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2249, in _get_metadata
    for line in self.get_metadata_lines(name):
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1219, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1211, in get_metadata
    return self._get(self._fn(self.egg_info,name))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1326, in _get
    stream = open(path, 'rb')
IOError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/oauthlib-0.6.0-py2.7.egg/EGG-INFO/top_level.txt'

Some time ago I installed a package via sudo easy_install. I don't use the python setuptools very often so maybe I did something wrong there.
For some reason mentos.py insisted on using the oauth package in /usr/local/lib/python2.7/dist-packages/... I removed that (sudo rm -rf /usr/local/lib/python2.7/dist-packages/oauthlib-0.6.0-py2.7.egg) and now jekyll is running again. Now, this is just a workaround because that probably breaks the python package, right?

@Ivoz

This comment has been minimized.

Show comment
Hide comment
@Ivoz

Ivoz Dec 7, 2013

I'm not sure without knowing why you have a python 2.7 in both /usr and /usr/local

Ivoz commented Dec 7, 2013

I'm not sure without knowing why you have a python 2.7 in both /usr and /usr/local

@aleiphoenix

This comment has been minimized.

Show comment
Hide comment
@aleiphoenix

aleiphoenix Dec 11, 2013

(The README used to mention, e.g., RubyPython.configure :python_exe => 'python2.7', but it seems 0.3 has moved to a different IPC system.)

Actually, pygments already support python 3 now, but it should install with python setup.py install or other tool, it will use 2to3 to turn some 2 specified syntax into 3-compatible. (such as turning except Error, err to except Error as err ). The original source code is just python 2 compatible. (see here)

mentos.py in pygments.rb just modifies sys.path in order to import pygments. I think maybe pygments.rb should hack with mentos.py, modify sys.path accroding to python executable version, for instance, add another python 3 compatible copy of pygments code

pygments-main/
pygments-main3/
base_dir = dirname(dirname(dirname(os.path.abspath(__file__))))
sys.path.append(base_dir + "/vendor")
if sys.version_info[0] > 2:
    sys.path.append(base_dir + "/vendor/pygments-main3")
else:
    sys.path.append(base_dir + "/vendor/pygments-main")
sys.path.append(base_dir + "/vendor/simplejson")

(The README used to mention, e.g., RubyPython.configure :python_exe => 'python2.7', but it seems 0.3 has moved to a different IPC system.)

Actually, pygments already support python 3 now, but it should install with python setup.py install or other tool, it will use 2to3 to turn some 2 specified syntax into 3-compatible. (such as turning except Error, err to except Error as err ). The original source code is just python 2 compatible. (see here)

mentos.py in pygments.rb just modifies sys.path in order to import pygments. I think maybe pygments.rb should hack with mentos.py, modify sys.path accroding to python executable version, for instance, add another python 3 compatible copy of pygments code

pygments-main/
pygments-main3/
base_dir = dirname(dirname(dirname(os.path.abspath(__file__))))
sys.path.append(base_dir + "/vendor")
if sys.version_info[0] > 2:
    sys.path.append(base_dir + "/vendor/pygments-main3")
else:
    sys.path.append(base_dir + "/vendor/pygments-main")
sys.path.append(base_dir + "/vendor/simplejson")
@flashlib

This comment has been minimized.

Show comment
Hide comment
@flashlib

flashlib Feb 25, 2014

I fixed by re-install python 2.7.6. Refer to here:
jekyll/jekyll#1181 (comment)

I fixed by re-install python 2.7.6. Refer to here:
jekyll/jekyll#1181 (comment)

@mattwildig

This comment has been minimized.

Show comment
Hide comment
@mattwildig

mattwildig Apr 28, 2014

This has reared its head again for me, so I’ve had another look.

I’m adding my comment here since it continues from my previous one in this thread but arguably could warrant its own issue, and doesn’t apply to everyone seeing this error (e.g. this isn’t related to the Python2/Python3 problem, except that the error is the same). The exception is raised when there is an issue communicating with Python and could result from a number of underlying causes.

My problem is the same as before, an Illegal instruction caused by the line os.close(fd) in mentos.py. Digging a bit further I came up with this Python script that reproduces the problem on my machine:

import os

# this is from plugin.py
import pkg_resources

maxfd = 65536

# this is the loop from mentos.py
for fd in range(3, maxfd):
    try:
        os.close(fd)
    except:
        pass

The import pkg_resources is from plugin.py within Pygments, and changing that try...except block to just do pkg_resources = None (i.e. as if the package wasn’t found) fixes the problem when calling from Ruby.

With a little more digging I can actually get the same issue with this smaller snippet:

import os, time

import pkg_resources

# note no try...except needed, this is actually closing an fd
os.close(4)

time.sleep(10)

What seems to be happening is Pygments is importing pkg_resources, which is opening a file or doing something else to get a file descriptor. mentos.py is then closing that fd, assuming it has been inherited from Ruby. After it has been closed something happens in Pygments or pkg_resources that expects the fd to be open, and that causes the Illegal instruction. What that something is, what the file being opened is, why it always seems to be fd 4, whether it will always be fd 4 and why it seemed to be working for a while on my machine are all things I don’t know.

Is this something that can be fixed in pygments.rb? Looking at the code it seems it will only affect people with pkg_resources which might be why it hasn’t been a bigger problem. I should point out I’m still on Snow Leopard (I should get round to upgrading) – I don’t know if that would make a difference. Can anyone reproduce with my snippets on a later Mac version or different OS?

This has reared its head again for me, so I’ve had another look.

I’m adding my comment here since it continues from my previous one in this thread but arguably could warrant its own issue, and doesn’t apply to everyone seeing this error (e.g. this isn’t related to the Python2/Python3 problem, except that the error is the same). The exception is raised when there is an issue communicating with Python and could result from a number of underlying causes.

My problem is the same as before, an Illegal instruction caused by the line os.close(fd) in mentos.py. Digging a bit further I came up with this Python script that reproduces the problem on my machine:

import os

# this is from plugin.py
import pkg_resources

maxfd = 65536

# this is the loop from mentos.py
for fd in range(3, maxfd):
    try:
        os.close(fd)
    except:
        pass

The import pkg_resources is from plugin.py within Pygments, and changing that try...except block to just do pkg_resources = None (i.e. as if the package wasn’t found) fixes the problem when calling from Ruby.

With a little more digging I can actually get the same issue with this smaller snippet:

import os, time

import pkg_resources

# note no try...except needed, this is actually closing an fd
os.close(4)

time.sleep(10)

What seems to be happening is Pygments is importing pkg_resources, which is opening a file or doing something else to get a file descriptor. mentos.py is then closing that fd, assuming it has been inherited from Ruby. After it has been closed something happens in Pygments or pkg_resources that expects the fd to be open, and that causes the Illegal instruction. What that something is, what the file being opened is, why it always seems to be fd 4, whether it will always be fd 4 and why it seemed to be working for a while on my machine are all things I don’t know.

Is this something that can be fixed in pygments.rb? Looking at the code it seems it will only affect people with pkg_resources which might be why it hasn’t been a bigger problem. I should point out I’m still on Snow Leopard (I should get round to upgrading) – I don’t know if that would make a difference. Can anyone reproduce with my snippets on a later Mac version or different OS?

@Ivoz

This comment has been minimized.

Show comment
Hide comment
@Ivoz

Ivoz Apr 28, 2014

you could use lsof while the script is open (do a sleep first, before the close) to check what file is being open that's causing the issue

Ivoz commented Apr 28, 2014

you could use lsof while the script is open (do a sleep first, before the close) to check what file is being open that's causing the issue

@mattwildig

This comment has been minimized.

Show comment
Hide comment
@mattwildig

mattwildig Apr 28, 2014

Thanks @Ivoz. lsof produces this line which looks like the culprit:

python  41894 matt    4u  KQUEUE                                count=1, state=0x2

Thanks @Ivoz. lsof produces this line which looks like the culprit:

python  41894 matt    4u  KQUEUE                                count=1, state=0x2
@paulfariello

This comment has been minimized.

Show comment
Hide comment
@paulfariello

paulfariello Jan 23, 2015

For the record, I had the same issue on OpenBSD with:

  • python 2.7.6
  • ruby 2.1.0
  • posix-spawn 0.3.9
  • pygments.rb 0.6.0

Running

python /usr/local/lib/ruby/gems/2.1/gems/pygments.rb-0.5.0/lib/pygments/mentos.py

was just fine (no output, waiting for some input).

The problem was with posix-spawn. Sorry but no more log about that.
Doing

gem install posix-spawn --version "=0.3.8"
gem uninstall posix-spawn --version ">0.3.8"

solved the issue.

For the record, I had the same issue on OpenBSD with:

  • python 2.7.6
  • ruby 2.1.0
  • posix-spawn 0.3.9
  • pygments.rb 0.6.0

Running

python /usr/local/lib/ruby/gems/2.1/gems/pygments.rb-0.5.0/lib/pygments/mentos.py

was just fine (no output, waiting for some input).

The problem was with posix-spawn. Sorry but no more log about that.
Doing

gem install posix-spawn --version "=0.3.8"
gem uninstall posix-spawn --version ">0.3.8"

solved the issue.

@frenkel

This comment has been minimized.

Show comment
Hide comment
@frenkel

frenkel Jan 25, 2015

Thanks @paulfariello I had the same problem on OpenBSD!

frenkel commented Jan 25, 2015

Thanks @paulfariello I had the same problem on OpenBSD!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment