MentosError header errors #45

Open
aprescott opened this Issue Oct 18, 2012 · 33 comments

Projects

None yet
@aprescott
1.9.3p286 :001 > require "pygments"
 => true 
1.9.3p286 :002 > Pygments.highlight("foo")
MentosError: Failed to get header.
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:351:in `rescue in get_header'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:332:in `get_header'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:229:in `block in mentos'
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:203:in `mentos'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:186:in `highlight'
    from (irb):2
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/bin/irb:16:in `<main>'

As far as I can tell, this is happening on a clean gem environment.

Curiously, installing python-pygments and calling Pygmentize.start("pygmentize") first gives, for one single subsequent call to highlight a slightly different message:

1.9.3p286 :003 > Pygments.start("pygmentize"); Pygments.highlight("foo")
MentosError: No header received back.
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:295:in `handle_header_and_return'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:232:in `block in mentos'
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:203:in `mentos'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:186:in `highlight'
    from (irb):3
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/bin/irb:16:in `<main>'

1.9.3p286 :004 > Pygments.highlight("foo")
MentosError: Failed to get header.
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:351:in `rescue in get_header'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:332:in `get_header'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:229:in `block in mentos'
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:203:in `mentos'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:186:in `highlight'
    from (irb):4
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/bin/irb:16:in `<main>'

@tnm
Collaborator
tnm commented Oct 20, 2012

Can you let me know what version of Python you are running on this machine?

@aprescott

python --version is 3.3.0.

@DirtYiCE

I can confirm the same bug with Python 3.2.3, however it works fine with Python 2.7.3 (both on Gentoo Linux). Probably a Python 3 issue.

@aprescott

What's the best way for me to specify a different executable to pygments.rb in the meantime? I have python2 on my PATH.

@aprescott

(The README used to mention, e.g., RubyPython.configure :python_exe => 'python2.7', but it seems 0.3 has moved to a different IPC system.)

@tnm
Collaborator
tnm commented Dec 7, 2012

We're officially only supporting 2.5, 2.6, and 2.7 for now. I'm planning support for Python 3, and patches are welcome for now.

@aprescott

Good to know, thanks. My ~/bin/python + PATH workaround is working for now.

@matthiasbeyer

Having the same issue here! python2 and python2.7 are in my path! What to do?

@Ivoz
Ivoz commented Jan 21, 2013

pygments should not assume that the system default python is going to be v2.x and not v3.x

It should be easy enough to find if a python2 exists on the path, and use that instead, no? Given that v2.x is now pretty much at EOL.

@obsoke
obsoke commented Jan 24, 2013

I've been getting this error as well in Jekyll. Using pygments 0.3.7. I have 'python2' aliased to python & in my path.

Has anyone managed to resolve this?

edit: Editing lib/pygments/mentos.py's first line from 'python' to 'python2' seemed to fix it for me.

@aprescott

Has anyone managed to resolve this?

I just created ~/bin/python as a symlink to python2. ~/bin is in my PATH. I didn't need to edit anything in mentos.py.

@mattwildig

I’ve got the same error, but I’m using Python 2.7. As far as I know I don’t have Python 3 installed on my machine at all (Mac 10.6, Python installed via Homebrew):

1.9.3-p385 :001 > require 'pygments'
 => true 
1.9.3-p385 :002 > Pygments.highlight 'foo'
MentosError: Failed to get header.
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:357:in `rescue in get_header'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:338:in `get_header'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:235:in `block in mentos'
        from /Users/matt/.rvm/rubies/ruby-1.9.3-p385/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:209:in `mentos'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:192:in `highlight'
        from (irb):2
        from /Users/matt/.rvm/rubies/ruby-1.9.3-p385/bin/irb:16:in `<main>'

In order to try and track the issue down I tried running the mentos.py script independently. It failed with the error Illegal instruction, which seems to be produced by the line os.close(fd). Commenting out this line (actually the whole for block so the indentation is right) clears the error, and commenting the same block in the gem itself seems to fix it:

1.9.3-p385 :001 > require 'pygments'
 => true 
1.9.3-p385 :002 > Pygments.highlight 'foo'
 => "<div class=\"highlight\"><pre><span class=\"n\">foo</span>\n</pre></div>" 

Can anyone cast any light on this?

@tnm
Collaborator
tnm commented Mar 28, 2013

@mattwildig Thanks for that. That can be fixed. I'll investigate.

@mattwildig

@tnm It appears that this error has fixed itself. Whilst I could reliably reproduce it when I reported it, it is no longer appearing. Unfortunately I don’t know what changed between than and now that could affect this. I did restart my machine recently (after installing updates) so that might have cleared something in the OS.

@ekampp
ekampp commented May 28, 2013

I have this problem appear intermittent on Heroku, when attempting to parse some markdown. The problem seems to be spread across all OS, browsers and countries, and there is no discernible pattern. Here is an example error:

MentosError: File "/app/vendor/bundle/ruby/2.0.0/gems/pygments.rb-0.5.0/lib/pygments/mentos.py", line 282, 
in start header = json.loads(line) File "/usr/local/lib/python2.7/json/__init__.py", line 326, in loads return 
_default_decoder.decode(s) File "/usr/local/lib/python2.7/json/decoder.py", line 369, in decode raise 
ValueError(errmsg("Extra data", s, end, len(s))) ValueError: Extra data: line 1 column 1 - line 1 column 129 
(char 1 - 129)
@radum
radum commented Jun 13, 2013

For Windows users, Python can run with multiple versions without any problem, so just install Python 2 and 3 and just change the env path. I know it is not really useful but could get you started at least.

@tschaub
tschaub commented Jun 25, 2013

@mattwildig - thanks for posting your fix here. I was getting the same with Ruby 1.8, Pygments 0.5.0, and Python 2.6.1. Commenting out the relevant block in mentos.py fixed the issue for me (and now Jekyll works again - see jekyll/jekyll#1181).

@milinda
milinda commented Jul 6, 2013

I got the same errror in Mountain Lion with Ruby 2.0.0p0 and Python 2.7.2. But it was not coming from the code block mentioned above. It was coming from ruby-2.0.0-p0/gems/pygments.rb-0.5.1/lib/pygments/popen.rb:366.

It says "in `rescue in get_header': Failed to get header. (MentosError)". Please let me know if any of you have any idea about this.

@milinda
milinda commented Jul 6, 2013

In my case popen.rb's get_header method's size[0..-2] failed saying

"
undefined method `[]' for nil:NilClass
"

It looks like size = @out.read(33) returns a null. This may be caused by a different error.

@milinda
milinda commented Jul 9, 2013

Finally I found the cause and a solution for my issue. It was due to corrupted python installation on Mac OS X Mountain Lion. pygmentize was throwing errors due to a permission issue.

@millisami

@milinda So wats the fix?

@vikstrous

I ran into this too. This guy's fix worked temporarily, but it would be great if there was a real fix ;).
http://www.nonsenseby.me/blog/2013/04/13/arch-linux/

@Ivoz
Ivoz commented Aug 30, 2013

You can also run the tool in a python virtualenv that uses python2, for a workaround.

@cdgz
cdgz commented Nov 22, 2013

Just to confirm that issue persists in 0.5.4 with ruby 1.8.7 and python 2.7.2 on SuSE Linux (3.0.13-0.27)

With python 2.6.7 Ok.

@phansch
phansch commented Dec 7, 2013

I had the same error but it seems for me the cause was something different? Running mentos.py gave me the following exception:

$ python mentos.py
Traceback (most recent call last):
  File "mentos.py", line 17, in <module>
    from pygments import lexers, formatters, styles, filters
  File "/home/philipp/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/gems/2.0.0/gems/pygments.rb-0.5.0/vendor/pygments-main/pygments/lexers/__init__.py", line 18, in <module>
    from pygments.plugin import find_plugin_lexers
  File "/home/philipp/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/gems/2.0.0/gems/pygments.rb-0.5.0/vendor/pygments-main/pygments/plugin.py", line 39, in <module>
    import pkg_resources
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2823, in <module>
    add_activation_listener(lambda dist: dist.activate())
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 710, in subscribe
    callback(dist)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2823, in <lambda>
    add_activation_listener(lambda dist: dist.activate())
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2255, in activate
    self.insert_on(path)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2362, in insert_on
    self.check_version_conflict()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2401, in check_version_conflict
    for modname in self._get_metadata('top_level.txt'):
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2249, in _get_metadata
    for line in self.get_metadata_lines(name):
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1219, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1211, in get_metadata
    return self._get(self._fn(self.egg_info,name))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1326, in _get
    stream = open(path, 'rb')
IOError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/oauthlib-0.6.0-py2.7.egg/EGG-INFO/top_level.txt'

Some time ago I installed a package via sudo easy_install. I don't use the python setuptools very often so maybe I did something wrong there.
For some reason mentos.py insisted on using the oauth package in /usr/local/lib/python2.7/dist-packages/... I removed that (sudo rm -rf /usr/local/lib/python2.7/dist-packages/oauthlib-0.6.0-py2.7.egg) and now jekyll is running again. Now, this is just a workaround because that probably breaks the python package, right?

@Ivoz
Ivoz commented Dec 7, 2013

I'm not sure without knowing why you have a python 2.7 in both /usr and /usr/local

@aleiphoenix

(The README used to mention, e.g., RubyPython.configure :python_exe => 'python2.7', but it seems 0.3 has moved to a different IPC system.)

Actually, pygments already support python 3 now, but it should install with python setup.py install or other tool, it will use 2to3 to turn some 2 specified syntax into 3-compatible. (such as turning except Error, err to except Error as err ). The original source code is just python 2 compatible. (see here)

mentos.py in pygments.rb just modifies sys.path in order to import pygments. I think maybe pygments.rb should hack with mentos.py, modify sys.path accroding to python executable version, for instance, add another python 3 compatible copy of pygments code

pygments-main/
pygments-main3/
base_dir = dirname(dirname(dirname(os.path.abspath(__file__))))
sys.path.append(base_dir + "/vendor")
if sys.version_info[0] > 2:
    sys.path.append(base_dir + "/vendor/pygments-main3")
else:
    sys.path.append(base_dir + "/vendor/pygments-main")
sys.path.append(base_dir + "/vendor/simplejson")
@flashlib

I fixed by re-install python 2.7.6. Refer to here:
jekyll/jekyll#1181 (comment)

@mattwildig

This has reared its head again for me, so I’ve had another look.

I’m adding my comment here since it continues from my previous one in this thread but arguably could warrant its own issue, and doesn’t apply to everyone seeing this error (e.g. this isn’t related to the Python2/Python3 problem, except that the error is the same). The exception is raised when there is an issue communicating with Python and could result from a number of underlying causes.

My problem is the same as before, an Illegal instruction caused by the line os.close(fd) in mentos.py. Digging a bit further I came up with this Python script that reproduces the problem on my machine:

import os

# this is from plugin.py
import pkg_resources

maxfd = 65536

# this is the loop from mentos.py
for fd in range(3, maxfd):
    try:
        os.close(fd)
    except:
        pass

The import pkg_resources is from plugin.py within Pygments, and changing that try...except block to just do pkg_resources = None (i.e. as if the package wasn’t found) fixes the problem when calling from Ruby.

With a little more digging I can actually get the same issue with this smaller snippet:

import os, time

import pkg_resources

# note no try...except needed, this is actually closing an fd
os.close(4)

time.sleep(10)

What seems to be happening is Pygments is importing pkg_resources, which is opening a file or doing something else to get a file descriptor. mentos.py is then closing that fd, assuming it has been inherited from Ruby. After it has been closed something happens in Pygments or pkg_resources that expects the fd to be open, and that causes the Illegal instruction. What that something is, what the file being opened is, why it always seems to be fd 4, whether it will always be fd 4 and why it seemed to be working for a while on my machine are all things I don’t know.

Is this something that can be fixed in pygments.rb? Looking at the code it seems it will only affect people with pkg_resources which might be why it hasn’t been a bigger problem. I should point out I’m still on Snow Leopard (I should get round to upgrading) – I don’t know if that would make a difference. Can anyone reproduce with my snippets on a later Mac version or different OS?

@Ivoz
Ivoz commented Apr 28, 2014

you could use lsof while the script is open (do a sleep first, before the close) to check what file is being open that's causing the issue

@mattwildig

Thanks @Ivoz. lsof produces this line which looks like the culprit:

python  41894 matt    4u  KQUEUE                                count=1, state=0x2
@kimpenhaus kimpenhaus referenced this issue in octopress/octopress Jan 21, 2015
Open

octopress docs fails on windows #77

@paulfariello

For the record, I had the same issue on OpenBSD with:

  • python 2.7.6
  • ruby 2.1.0
  • posix-spawn 0.3.9
  • pygments.rb 0.6.0

Running

python /usr/local/lib/ruby/gems/2.1/gems/pygments.rb-0.5.0/lib/pygments/mentos.py

was just fine (no output, waiting for some input).

The problem was with posix-spawn. Sorry but no more log about that.
Doing

gem install posix-spawn --version "=0.3.8"
gem uninstall posix-spawn --version ">0.3.8"

solved the issue.

@frenkel
frenkel commented Jan 25, 2015

Thanks @paulfariello I had the same problem on OpenBSD!

@vtamara vtamara referenced this issue in rtomayko/posix-spawn Jun 5, 2015
Closed

Does not compile on OpenBSD #24

@xupeng xupeng added a commit to xupeng/octopress-in-docker that referenced this issue Feb 26, 2016
@xupeng xupeng fix octopress "liquid error" 5eadbc3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment