Python 3.3+ support

[sabertiger]

pygments.rb is currently using a development version of pygments 2.0.
Python3.3+ is already supported by the release version of pygments 2.0, which was released on Nov 9, 2014. Please pull latest released verison.

Current release from vendor/pygments-main/CHANGES

Version 2.0
-----------
(under development)

- Dropped Python 2.4 and 2.5 compatibility.  This is in favor of single-source
  compatibility between Python 2.6, 2.7 and 3.3+.

Current latest release version

Version 2.0.2
-------------
(released Jan 20, 2015)

.

[wigging]

Taken directly from http://pygments.org/faq/ for version 2.0.2:

So please update pygments.rb to version 2.0.2. Python 2 is now considered legacy code.

[aviaryan]

👍

[gengjiawen]

+1

[boutil]

Hi,
Debian is planning to remove python2, as it will very soon retire. Support for python3 is critical to allow the distribution of this project in Debian and its derivatives.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=943257

thank you!

[terceiro]

I have a PR that works for me in #192 . is there anyone here that has access to a windows machine that could help debug the ci failure there?

[ivanistheone]

@terceiro I don't have access to a windows machine, but I'm guessing this is the problem:
https://github.com/tmm1/pygments.rb/blob/master/.github/workflows/ci.yml#L21

Plz try changing that to 3.6 or 3.7 to see if it works. 🤞

UPDATE: Oh nevermind I see you've changed that: https://github.com/tmm1/pygments.rb/pull/192/files#diff-e9f950f17198d3d5e3122a44230a09b9R21
then maybe try also changing actions/setup-python@v1 to actions/setup-python@v2 --- if it doesn't fix it, at least it will refresh the log output (currently not there).

[slonopotamus]

The failure has nothing to do with GH-Actions, I also successfully reproduce it locally on Windows 10 + Ruby 2.7 + Python 3.8. I believe the problem is caused by newline character at the beginning of header received from mentos.py:

This might be caused by writing \r\n instead of just \n by _send_data in mentos.py, though I haven't debugged through it yet, so it is just a wild guess.

[slonopotamus]

UPDATE: Yes, my guess was correct. If I replace all "\n" with " " in _write_error and _send_data in mentos.py, it passes test initialization and half of tests.

So, the problem is: writing \n from Python side does not actually write \n but instead writes platform-dependent newline that becomes \r\n on Windows and breaks all byte counting.

I think you want to make sure that Python writes bytes to Ruby. Basically, all instances of sys.stdout.write need to be fixed.

[ivanistheone]

@slonopotamus Good catch. Indeed there was some mix of unicode and bytes writing in the code:
I updated this PR #194 (specifically this commit and windows tests pass now.

seems to be only some problem in ruby2.2 on mac
/Users/runner/work/pygments.rb/pygments.rb/lib/pygments/popen.rb:163:in `lexers!': undefined method `inject' for nil:NilClass (NoMethodError)