Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MentosError header errors #45

Open
aprescott opened this issue Oct 18, 2012 · 33 comments
Open

MentosError header errors #45

aprescott opened this issue Oct 18, 2012 · 33 comments

Comments

@aprescott
Copy link

@aprescott aprescott commented Oct 18, 2012

1.9.3p286 :001 > require "pygments"
 => true 
1.9.3p286 :002 > Pygments.highlight("foo")
MentosError: Failed to get header.
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:351:in `rescue in get_header'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:332:in `get_header'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:229:in `block in mentos'
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:203:in `mentos'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:186:in `highlight'
    from (irb):2
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/bin/irb:16:in `<main>'

As far as I can tell, this is happening on a clean gem environment.

Curiously, installing python-pygments and calling Pygmentize.start("pygmentize") first gives, for one single subsequent call to highlight a slightly different message:

1.9.3p286 :003 > Pygments.start("pygmentize"); Pygments.highlight("foo")
MentosError: No header received back.
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:295:in `handle_header_and_return'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:232:in `block in mentos'
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:203:in `mentos'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:186:in `highlight'
    from (irb):3
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/bin/irb:16:in `<main>'

1.9.3p286 :004 > Pygments.highlight("foo")
MentosError: Failed to get header.
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:351:in `rescue in get_header'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:332:in `get_header'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:229:in `block in mentos'
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:203:in `mentos'
    from /home/adam/.rvm/gems/ruby-1.9.3-p286/gems/pygments.rb-0.3.2/lib/pygments/popen.rb:186:in `highlight'
    from (irb):4
    from /home/adam/.rvm/rubies/ruby-1.9.3-p286/bin/irb:16:in `<main>'

@tnm
Copy link
Collaborator

@tnm tnm commented Oct 20, 2012

Can you let me know what version of Python you are running on this machine?

@aprescott
Copy link
Author

@aprescott aprescott commented Oct 20, 2012

python --version is 3.3.0.

@DirtYiCE
Copy link

@DirtYiCE DirtYiCE commented Oct 20, 2012

I can confirm the same bug with Python 3.2.3, however it works fine with Python 2.7.3 (both on Gentoo Linux). Probably a Python 3 issue.

@aprescott
Copy link
Author

@aprescott aprescott commented Oct 20, 2012

What's the best way for me to specify a different executable to pygments.rb in the meantime? I have python2 on my PATH.

@aprescott
Copy link
Author

@aprescott aprescott commented Oct 20, 2012

(The README used to mention, e.g., RubyPython.configure :python_exe => 'python2.7', but it seems 0.3 has moved to a different IPC system.)

@tnm
Copy link
Collaborator

@tnm tnm commented Dec 7, 2012

We're officially only supporting 2.5, 2.6, and 2.7 for now. I'm planning support for Python 3, and patches are welcome for now.

@aprescott
Copy link
Author

@aprescott aprescott commented Dec 7, 2012

Good to know, thanks. My ~/bin/python + PATH workaround is working for now.

@matthiasbeyer
Copy link

@matthiasbeyer matthiasbeyer commented Dec 16, 2012

Having the same issue here! python2 and python2.7 are in my path! What to do?

@Ivoz
Copy link

@Ivoz Ivoz commented Jan 21, 2013

pygments should not assume that the system default python is going to be v2.x and not v3.x

It should be easy enough to find if a python2 exists on the path, and use that instead, no? Given that v2.x is now pretty much at EOL.

@obsoke
Copy link

@obsoke obsoke commented Jan 24, 2013

I've been getting this error as well in Jekyll. Using pygments 0.3.7. I have 'python2' aliased to python & in my path.

Has anyone managed to resolve this?

edit: Editing lib/pygments/mentos.py's first line from 'python' to 'python2' seemed to fix it for me.

@aprescott
Copy link
Author

@aprescott aprescott commented Jan 25, 2013

Has anyone managed to resolve this?

I just created ~/bin/python as a symlink to python2. ~/bin is in my PATH. I didn't need to edit anything in mentos.py.

@mattwildig
Copy link

@mattwildig mattwildig commented Mar 28, 2013

I’ve got the same error, but I’m using Python 2.7. As far as I know I don’t have Python 3 installed on my machine at all (Mac 10.6, Python installed via Homebrew):

1.9.3-p385 :001 > require 'pygments'
 => true 
1.9.3-p385 :002 > Pygments.highlight 'foo'
MentosError: Failed to get header.
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:357:in `rescue in get_header'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:338:in `get_header'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:235:in `block in mentos'
        from /Users/matt/.rvm/rubies/ruby-1.9.3-p385/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:209:in `mentos'
        from /Users/matt/.rvm/gems/ruby-1.9.3-p385/gems/pygments.rb-0.3.7/lib/pygments/popen.rb:192:in `highlight'
        from (irb):2
        from /Users/matt/.rvm/rubies/ruby-1.9.3-p385/bin/irb:16:in `<main>'

In order to try and track the issue down I tried running the mentos.py script independently. It failed with the error Illegal instruction, which seems to be produced by the line os.close(fd). Commenting out this line (actually the whole for block so the indentation is right) clears the error, and commenting the same block in the gem itself seems to fix it:

1.9.3-p385 :001 > require 'pygments'
 => true 
1.9.3-p385 :002 > Pygments.highlight 'foo'
 => "<div class=\"highlight\"><pre><span class=\"n\">foo</span>\n</pre></div>" 

Can anyone cast any light on this?

@tnm
Copy link
Collaborator

@tnm tnm commented Mar 28, 2013

@mattwildig Thanks for that. That can be fixed. I'll investigate.

@mattwildig
Copy link

@mattwildig mattwildig commented Apr 19, 2013

@tnm It appears that this error has fixed itself. Whilst I could reliably reproduce it when I reported it, it is no longer appearing. Unfortunately I don’t know what changed between than and now that could affect this. I did restart my machine recently (after installing updates) so that might have cleared something in the OS.

@ekampp
Copy link

@ekampp ekampp commented May 28, 2013

I have this problem appear intermittent on Heroku, when attempting to parse some markdown. The problem seems to be spread across all OS, browsers and countries, and there is no discernible pattern. Here is an example error:

MentosError: File "/app/vendor/bundle/ruby/2.0.0/gems/pygments.rb-0.5.0/lib/pygments/mentos.py", line 282, 
in start header = json.loads(line) File "/usr/local/lib/python2.7/json/__init__.py", line 326, in loads return 
_default_decoder.decode(s) File "/usr/local/lib/python2.7/json/decoder.py", line 369, in decode raise 
ValueError(errmsg("Extra data", s, end, len(s))) ValueError: Extra data: line 1 column 1 - line 1 column 129 
(char 1 - 129)
@radum
Copy link

@radum radum commented Jun 13, 2013

For Windows users, Python can run with multiple versions without any problem, so just install Python 2 and 3 and just change the env path. I know it is not really useful but could get you started at least.

@tschaub
Copy link

@tschaub tschaub commented Jun 25, 2013

@mattwildig - thanks for posting your fix here. I was getting the same with Ruby 1.8, Pygments 0.5.0, and Python 2.6.1. Commenting out the relevant block in mentos.py fixed the issue for me (and now Jekyll works again - see jekyll/jekyll#1181).

@milinda
Copy link

@milinda milinda commented Jul 6, 2013

I got the same errror in Mountain Lion with Ruby 2.0.0p0 and Python 2.7.2. But it was not coming from the code block mentioned above. It was coming from ruby-2.0.0-p0/gems/pygments.rb-0.5.1/lib/pygments/popen.rb:366.

It says "in `rescue in get_header': Failed to get header. (MentosError)". Please let me know if any of you have any idea about this.

@milinda
Copy link

@milinda milinda commented Jul 6, 2013

In my case popen.rb's get_header method's size[0..-2] failed saying

"
undefined method `[]' for nil:NilClass
"

It looks like size = @out.read(33) returns a null. This may be caused by a different error.

@milinda
Copy link

@milinda milinda commented Jul 9, 2013

Finally I found the cause and a solution for my issue. It was due to corrupted python installation on Mac OS X Mountain Lion. pygmentize was throwing errors due to a permission issue.

@millisami
Copy link

@millisami millisami commented Jul 14, 2013

@milinda So wats the fix?

@vikstrous
Copy link

@vikstrous vikstrous commented Aug 30, 2013

I ran into this too. This guy's fix worked temporarily, but it would be great if there was a real fix ;).
http://www.nonsenseby.me/blog/2013/04/13/arch-linux/

@Ivoz
Copy link

@Ivoz Ivoz commented Aug 30, 2013

You can also run the tool in a python virtualenv that uses python2, for a workaround.

@cdgz
Copy link

@cdgz cdgz commented Nov 22, 2013

Just to confirm that issue persists in 0.5.4 with ruby 1.8.7 and python 2.7.2 on SuSE Linux (3.0.13-0.27)

With python 2.6.7 Ok.

@phansch
Copy link

@phansch phansch commented Dec 7, 2013

I had the same error but it seems for me the cause was something different? Running mentos.py gave me the following exception:

$ python mentos.py
Traceback (most recent call last):
  File "mentos.py", line 17, in <module>
    from pygments import lexers, formatters, styles, filters
  File "/home/philipp/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/gems/2.0.0/gems/pygments.rb-0.5.0/vendor/pygments-main/pygments/lexers/__init__.py", line 18, in <module>
    from pygments.plugin import find_plugin_lexers
  File "/home/philipp/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/gems/2.0.0/gems/pygments.rb-0.5.0/vendor/pygments-main/pygments/plugin.py", line 39, in <module>
    import pkg_resources
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2823, in <module>
    add_activation_listener(lambda dist: dist.activate())
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 710, in subscribe
    callback(dist)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2823, in <lambda>
    add_activation_listener(lambda dist: dist.activate())
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2255, in activate
    self.insert_on(path)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2362, in insert_on
    self.check_version_conflict()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2401, in check_version_conflict
    for modname in self._get_metadata('top_level.txt'):
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2249, in _get_metadata
    for line in self.get_metadata_lines(name):
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1219, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1211, in get_metadata
    return self._get(self._fn(self.egg_info,name))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1326, in _get
    stream = open(path, 'rb')
IOError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/oauthlib-0.6.0-py2.7.egg/EGG-INFO/top_level.txt'

Some time ago I installed a package via sudo easy_install. I don't use the python setuptools very often so maybe I did something wrong there.
For some reason mentos.py insisted on using the oauth package in /usr/local/lib/python2.7/dist-packages/... I removed that (sudo rm -rf /usr/local/lib/python2.7/dist-packages/oauthlib-0.6.0-py2.7.egg) and now jekyll is running again. Now, this is just a workaround because that probably breaks the python package, right?

@Ivoz
Copy link

@Ivoz Ivoz commented Dec 7, 2013

I'm not sure without knowing why you have a python 2.7 in both /usr and /usr/local

@aleiphoenix
Copy link

@aleiphoenix aleiphoenix commented Dec 11, 2013

(The README used to mention, e.g., RubyPython.configure :python_exe => 'python2.7', but it seems 0.3 has moved to a different IPC system.)

Actually, pygments already support python 3 now, but it should install with python setup.py install or other tool, it will use 2to3 to turn some 2 specified syntax into 3-compatible. (such as turning except Error, err to except Error as err ). The original source code is just python 2 compatible. (see here)

mentos.py in pygments.rb just modifies sys.path in order to import pygments. I think maybe pygments.rb should hack with mentos.py, modify sys.path accroding to python executable version, for instance, add another python 3 compatible copy of pygments code

pygments-main/
pygments-main3/
base_dir = dirname(dirname(dirname(os.path.abspath(__file__))))
sys.path.append(base_dir + "/vendor")
if sys.version_info[0] > 2:
    sys.path.append(base_dir + "/vendor/pygments-main3")
else:
    sys.path.append(base_dir + "/vendor/pygments-main")
sys.path.append(base_dir + "/vendor/simplejson")
@flashlib
Copy link

@flashlib flashlib commented Feb 25, 2014

I fixed by re-install python 2.7.6. Refer to here:
jekyll/jekyll#1181 (comment)

@mattwildig
Copy link

@mattwildig mattwildig commented Apr 28, 2014

This has reared its head again for me, so I’ve had another look.

I’m adding my comment here since it continues from my previous one in this thread but arguably could warrant its own issue, and doesn’t apply to everyone seeing this error (e.g. this isn’t related to the Python2/Python3 problem, except that the error is the same). The exception is raised when there is an issue communicating with Python and could result from a number of underlying causes.

My problem is the same as before, an Illegal instruction caused by the line os.close(fd) in mentos.py. Digging a bit further I came up with this Python script that reproduces the problem on my machine:

import os

# this is from plugin.py
import pkg_resources

maxfd = 65536

# this is the loop from mentos.py
for fd in range(3, maxfd):
    try:
        os.close(fd)
    except:
        pass

The import pkg_resources is from plugin.py within Pygments, and changing that try...except block to just do pkg_resources = None (i.e. as if the package wasn’t found) fixes the problem when calling from Ruby.

With a little more digging I can actually get the same issue with this smaller snippet:

import os, time

import pkg_resources

# note no try...except needed, this is actually closing an fd
os.close(4)

time.sleep(10)

What seems to be happening is Pygments is importing pkg_resources, which is opening a file or doing something else to get a file descriptor. mentos.py is then closing that fd, assuming it has been inherited from Ruby. After it has been closed something happens in Pygments or pkg_resources that expects the fd to be open, and that causes the Illegal instruction. What that something is, what the file being opened is, why it always seems to be fd 4, whether it will always be fd 4 and why it seemed to be working for a while on my machine are all things I don’t know.

Is this something that can be fixed in pygments.rb? Looking at the code it seems it will only affect people with pkg_resources which might be why it hasn’t been a bigger problem. I should point out I’m still on Snow Leopard (I should get round to upgrading) – I don’t know if that would make a difference. Can anyone reproduce with my snippets on a later Mac version or different OS?

@Ivoz
Copy link

@Ivoz Ivoz commented Apr 28, 2014

you could use lsof while the script is open (do a sleep first, before the close) to check what file is being open that's causing the issue

@mattwildig
Copy link

@mattwildig mattwildig commented Apr 28, 2014

Thanks @Ivoz. lsof produces this line which looks like the culprit:

python  41894 matt    4u  KQUEUE                                count=1, state=0x2
@paulfariello
Copy link

@paulfariello paulfariello commented Jan 23, 2015

For the record, I had the same issue on OpenBSD with:

  • python 2.7.6
  • ruby 2.1.0
  • posix-spawn 0.3.9
  • pygments.rb 0.6.0

Running

python /usr/local/lib/ruby/gems/2.1/gems/pygments.rb-0.5.0/lib/pygments/mentos.py

was just fine (no output, waiting for some input).

The problem was with posix-spawn. Sorry but no more log about that.
Doing

gem install posix-spawn --version "=0.3.8"
gem uninstall posix-spawn --version ">0.3.8"

solved the issue.

@frenkel
Copy link

@frenkel frenkel commented Jan 25, 2015

Thanks @paulfariello I had the same problem on OpenBSD!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.