Merged in Javantea/pygments-main (pull request #501)

Fix Shell Injection in FontManager._get_nix_font_path

.hgignore

@@ -2,15 +2,18 @@ syntax: glob
 *.egg
 *.pyc
 *.pyo
+.*.sw[op]
 .idea/
 .ropeproject
+.project
 .tags
 .tox
 Pygments.egg-info/*
 TAGS
 build/*
 dist/*
 doc/_build
+TAGS
 tests/.coverage
 tests/cover
 tests/examplefiles/output

AUTHORS

@@ -7,12 +7,13 @@ Other contributors, listed alphabetically, are:
 
 * Sam Aaron -- Ioke lexer
 * Ali Afshar -- image formatter
-* Thomas Aglassinger -- Rexx lexer
+* Thomas Aglassinger -- Easytrieve, JCL and Rexx lexers
 * Kumar Appaiah -- Debian control lexer
 * Andreas Amann -- AppleScript lexer
 * Timothy Armstrong -- Dart lexer fixes
 * Jeffrey Arnold -- R/S, Rd, BUGS, Jags, and Stan lexers
 * Jeremy Ashkenas -- CoffeeScript lexer
+* José Joaquín Atria -- Praat lexer
 * Stefan Matthias Aust -- Smalltalk lexer
 * Lucas Bajolet -- Nit lexer
 * Ben Bangert -- Mako lexers
@@ -23,6 +24,8 @@ Other contributors, listed alphabetically, are:
 * Michael Bayer -- Myghty lexers
 * Thomas Beale -- Archetype lexers
 * John Benediktsson -- Factor lexer
+* Trevor Bergeron -- mIRC formatter
+* Vincent Bernat -- LessCSS lexer
 * Christopher Bertels -- Fancy lexer
 * Jarrett Billingsley -- MiniD lexer
 * Adam Blinkinsop -- Haskell, Redcode lexers
@@ -36,11 +39,14 @@ Other contributors, listed alphabetically, are:
 * Daniël W. Crompton - Pike lexer
 * Pete Curry -- bugfixes
 * Bryan Davis -- EBNF lexer
+* Bruno Deferrari -- Shen lexer
 * Owen Durni -- Haxe lexer
 * Alexander Dutton, Oxford University Computing Services -- SPARQL lexer
+* James Edwards -- Terraform lexer
 * Nick Efford -- Python 3 lexer
 * Sven Efftinge -- Xtend lexer
 * Artem Egorkine -- terminal256 formatter
+* Michael Ficarra -- CPSA lexer
 * James H. Fisher -- PostScript lexer
 * William S. Fulton -- SWIG lexer
 * Carlos Galdino -- Elixir and Elixir Console lexers
@@ -55,14 +61,18 @@ Other contributors, listed alphabetically, are:
 * Krzysiek Goj -- Scala lexer
 * Matt Good -- Genshi, Cheetah lexers
 * Michał Górny -- vim modeline support
+* Alex Gosse -- TrafficScript lexer
 * Patrick Gotthardt -- PHP namespaces support
 * Olivier Guibe -- Asymptote lexer
 * Jordi Gutiérrez Hermoso -- Octave lexer
+* Florian Hahn -- Boogie lexer
 * Martin Harriman -- SNOBOL lexer
 * Matthew Harrison -- SVG formatter
 * Steven Hazel -- Tcl lexer
+* Dan Michael Heggø -- Turtle lexer
 * Aslak Hellesøy -- Gherkin lexer
 * Greg Hendershott -- Racket lexer
+* Justin Hendrick -- ParaSail lexer
 * David Hess, Fish Software, Inc. -- Objective-J lexer
 * Varun Hiremath -- Debian control lexer
 * Rob Hoelz -- Perl 6 lexer
@@ -73,6 +83,7 @@ Other contributors, listed alphabetically, are:
 * Alastair Houghton -- Lexer inheritance facility
 * Tim Howard -- BlitzMax lexer
 * Ivan Inozemtsev -- Fantom lexer
+* Hiroaki Itoh -- Shell console rewrite
 * Brian R. Jackson -- Tea lexer
 * Christian Jann -- ShellSession lexer
 * Dennis Kaarsemaker -- sources.list lexer
@@ -93,10 +104,12 @@ Other contributors, listed alphabetically, are:
 * Gerd Kurzbach -- Modelica lexer
 * Jon Larimer, Google Inc. -- Smali lexer
 * Olov Lassus -- Dart lexer
+* Matt Layman -- TAP lexer
 * Sylvestre Ledru -- Scilab lexer
 * Mark Lee -- Vala lexer
 * Ben Mabey -- Gherkin lexer
 * Angus MacArthur -- QML lexer
+* Louis Mandel -- X10 lexer
 * Louis Marchand -- Eiffel lexer
 * Simone Margaritelli -- Hybris lexer
 * Kirk McDonald -- D lexer
@@ -114,6 +127,7 @@ Other contributors, listed alphabetically, are:
 * Jon Morton -- Rust lexer
 * Paulo Moura -- Logtalk lexer
 * Mher Movsisyan -- DTD lexer
+* Dejan Muhamedagic -- Crmsh lexer
 * Ana Nelson -- Ragel, ANTLR, R console lexers
 * Nam T. Nguyen -- Monokai style
 * Jesper Noehr -- HTML formatter "anchorlinenos"
@@ -153,6 +167,8 @@ Other contributors, listed alphabetically, are:
 * Jerome St-Louis -- eC lexer
 * James Strachan -- Kotlin lexer
 * Tom Stuart -- Treetop lexer
+* Colin Sullivan -- SuperCollider lexer
+* Edoardo Tenani -- Arduino lexer
 * Tiberius Teng -- default style overhaul
 * Jeremy Thurgood -- Erlang, Squid config lexers
 * Brian Tiffin -- OpenCOBOL lexer
@@ -174,5 +190,6 @@ Other contributors, listed alphabetically, are:
 * Enrique Zamudio -- Ceylon lexer
 * Alex Zimin -- Nemerle lexer
 * Rob Zimmerman -- Kal lexer
+* Vincent Zurczak -- Roboconf lexer
 
 Many thanks for all contributions!

CHANGES

@@ -18,18 +18,52 @@ Version 2.1
   * Modula-2 with multi-dialect support (#1090)
   * Fortran fixed format (PR#213)
   * Archetype Definition language (PR#483)
+  * Terraform (PR#432)
+  * Jcl, Easytrieve (PR#208)
+  * ParaSail (PR#381)
+  * Boogie (PR#420)
+  * Turtle (PR#425)
+  * Fish Shell (PR#422)
+  * Roboconf (PR#449)
+  * Test Anything Protocol (PR#428)
+  * Shen (PR#385)
+  * Component Pascal (PR#437)
+  * SuperCollider (PR#472)
+  * Shell consoles (Tcsh, PowerShell, MSDOS) (PR#479)
+  * Elm and J (PR#452)
+  * Crmsh (PR#440)
+  * Praat (PR#492)
+
 
 - Added styles:
 
   * Lovelace (PR#456)
   * Algol and Algol-nu (#1090)
 
+
+- Added formatters:
+
+  * IRC (PR#458)
+
+
 - Updated autopygmentize script (PR#445)
 
 - Fixed style inheritance for non-standard token types in HTML output.
 
 - Added support for async/await to Python 3 lexer.
 
+- Rewrote linenos option for TerminalFormatter (it's better, but slightly
+  different output than before).  (#1147)
+
+- Javascript lexer now supports most of ES6.  (#1100)
+
+- Cocoa builtins updated for iOS 8.1 (PR#433)
+
+- Combined BashSessionLexer and ShellSessionLexer, new version should support
+  the prompt styles of either.
+
+- Added option to pygmentize to show a full traceback on exceptions.
+
 
 Version 2.0.3
 -------------

README.rst

@@ -28,7 +28,7 @@ repository, tickets and pull requests can be viewed.
 Continuous testing runs on drone.io:
 
 .. image:: https://drone.io/bitbucket.org/birkenfeld/pygments-main/status.png
-   :target: https://drone.io/bitbucket.org/birkenfeld/pygments-main/
+   :target: https://drone.io/bitbucket.org/birkenfeld/pygments-main
 
 The authors
 -----------

doc/docs/lexerdevelopment.rst

@@ -145,7 +145,7 @@ Regex Flags
 
 You can either define regex flags locally in the regex (``r'(?x)foo bar'``) or
 globally by adding a `flags` attribute to your lexer class.  If no attribute is
-defined, it defaults to `re.MULTILINE`.  For more informations about regular
+defined, it defaults to `re.MULTILINE`.  For more information about regular
 expression flags see the page about `regular expressions`_ in the Python
 documentation.
 
@@ -345,15 +345,14 @@ There are a few more things you can do with states:
   `PythonLexer`'s string literal processing.
 
 - If you want your lexer to start lexing in a different state you can modify the
-  stack by overloading the `get_tokens_unprocessed()` method::
+  stack by overriding the `get_tokens_unprocessed()` method::
 
       from pygments.lexer import RegexLexer
 
       class ExampleLexer(RegexLexer):
           tokens = {...}
 
-          def get_tokens_unprocessed(self, text):
-              stack = ['root', 'otherstate']
+          def get_tokens_unprocessed(self, text, stack=('root', 'otherstate')):
               for item in RegexLexer.get_tokens_unprocessed(text, stack):
                   yield item
 

pygments/__init__.py

@@ -46,13 +46,13 @@ def lex(code, lexer):
     except TypeError as err:
         if isinstance(err.args[0], str) and \
            ('unbound method get_tokens' in err.args[0] or
-                'missing 1 required positional argument' in err.args[0]):
+            'missing 1 required positional argument' in err.args[0]):
             raise TypeError('lex() argument must be a lexer instance, '
                             'not a class')
         raise
 
 
-def format(tokens, formatter, outfile=None):
+def format(tokens, formatter, outfile=None):  # pylint: disable=redefined-builtin
     """
     Format a tokenlist ``tokens`` with the formatter ``formatter``.
 
@@ -70,7 +70,7 @@ def format(tokens, formatter, outfile=None):
     except TypeError as err:
         if isinstance(err.args[0], str) and \
            ('unbound method format' in err.args[0] or
-                'missing 1 required positional argument' in err.args[0]):
+            'missing 1 required positional argument' in err.args[0]):
             raise TypeError('format() argument must be a formatter instance, '
                             'not a class')
         raise

pygments/cmdline.py

@@ -19,11 +19,12 @@
 from pygments.util import ClassNotFound, OptionError, docstring_headline, \
     guess_decode, guess_decode_from_terminal, terminal_encoding
 from pygments.lexers import get_all_lexers, get_lexer_by_name, guess_lexer, \
-    get_lexer_for_filename, find_lexer_class_for_filename, TextLexer
+    get_lexer_for_filename, find_lexer_class_for_filename
+from pygments.lexers.special import TextLexer
 from pygments.formatters.latex import LatexEmbeddedLexer, LatexFormatter
 from pygments.formatters import get_all_formatters, get_formatter_by_name, \
-    get_formatter_for_filename, find_formatter_class, \
-    TerminalFormatter  # pylint:disable-msg=E0611
+    get_formatter_for_filename, find_formatter_class
+from pygments.formatters.terminal import TerminalFormatter
 from pygments.filters import get_all_filters, find_filter_class
 from pygments.styles import get_all_styles, get_style_by_name
 
@@ -247,7 +248,7 @@ def main_inner(popts, args, usage):
             print(usage, file=sys.stderr)
             return 2
 
-        what, name = args
+        what, name = args  # pylint: disable=unbalanced-tuple-unpacking
         if what not in ('lexer', 'formatter', 'filter'):
             print(usage, file=sys.stderr)
             return 2
@@ -269,7 +270,7 @@ def main_inner(popts, args, usage):
     opts.pop('-P', None)
 
     # encodings
-    inencoding  = parsed_opts.get('inencoding', parsed_opts.get('encoding'))
+    inencoding = parsed_opts.get('inencoding', parsed_opts.get('encoding'))
     outencoding = parsed_opts.get('outencoding', parsed_opts.get('encoding'))
 
     # handle ``pygmentize -N``

pygments/filter.py

@@ -69,6 +69,6 @@ def __init__(self, **options):
         Filter.__init__(self, **options)
 
     def filter(self, lexer, stream):
-        # pylint: disable-msg=E1102
+        # pylint: disable=not-callable
         for ttype, value in self.function(lexer, stream, self.options):
             yield ttype, value

pygments/formatters/_mapping.py

@@ -20,6 +20,7 @@
     'BmpImageFormatter': ('pygments.formatters.img', 'img_bmp', ('bmp', 'bitmap'), ('*.bmp',), 'Create a bitmap image from source code. This uses the Python Imaging Library to generate a pixmap from the source code.'),
     'GifImageFormatter': ('pygments.formatters.img', 'img_gif', ('gif',), ('*.gif',), 'Create a GIF image from source code. This uses the Python Imaging Library to generate a pixmap from the source code.'),
     'HtmlFormatter': ('pygments.formatters.html', 'HTML', ('html',), ('*.html', '*.htm'), "Format tokens as HTML 4 ``<span>`` tags within a ``<pre>`` tag, wrapped in a ``<div>`` tag. The ``<div>``'s CSS class can be set by the `cssclass` option."),
+    'IRCFormatter': ('pygments.formatters.irc', 'IRC', ('irc', 'IRC'), (), 'Format tokens with IRC color sequences'),
     'ImageFormatter': ('pygments.formatters.img', 'img', ('img', 'IMG', 'png'), ('*.png',), 'Create a PNG image from source code. This uses the Python Imaging Library to generate a pixmap from the source code.'),
     'JpgImageFormatter': ('pygments.formatters.img', 'img_jpg', ('jpg', 'jpeg'), ('*.jpg',), 'Create a JPEG image from source code. This uses the Python Imaging Library to generate a pixmap from the source code.'),
     'LatexFormatter': ('pygments.formatters.latex', 'LaTeX', ('latex', 'tex'), ('*.tex',), 'Format tokens as LaTeX code. This needs the `fancyvrb` and `color` standard packages.'),

pygments/formatters/html.py

@@ -140,7 +140,7 @@ class HtmlFormatter(Formatter):
 
     When `tagsfile` is set to the path of a ctags index file, it is used to
     generate hyperlinks from names to their definition.  You must enable
-    `anchorlines` and run ctags with the `-n` option for this to work.  The
+    `lineanchors` and run ctags with the `-n` option for this to work.  The
     `python-ctags` module from PyPI must be installed to use this feature;
     otherwise a `RuntimeError` will be raised.