New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

simplejson.scanner.JSONDecodeError: Unpaired high surrogate #62

Closed
d0ugal opened this Issue Feb 23, 2013 · 19 comments

Comments

Projects
None yet
4 participants
@d0ugal
Contributor

d0ugal commented Feb 23, 2013

Hi,

Sorry for taking a while to report this. It took me some time to find the JSON (in about 2.5TB of data) and then I realised I should probably trim it down to the relevant section.

I am unable to parse the following JSON with simplejson 3.1.0 under Linux. This JSON is taken from a result set from ElasticSearch and obviously trimmed down to be as small as possible.

{"a": "\ud8e9"}

I done some testing on a fresh vagrant Ubuntu Precise 64bit virtual machine but previously we have been seeing the error with lucid on EC2.

The error get is;

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/__init__.py", line 398, in load
    use_decimal=use_decimal, **kw)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/__init__.py", line 454, in loads
    return _default_decoder.decode(s)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/decoder.py", line 393, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Unpaired high surrogate: line 1 column 8 (char 7)

If I disable the speedups with simplejson._toggle_speedups(False) I get a (possibly) slightly more helpful error.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/__init__.py", line 398, in load
    use_decimal=use_decimal, **kw)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/__init__.py", line 454, in loads
    return _default_decoder.decode(s)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/decoder.py", line 393, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/scanner.py", line 119, in scan_once
    return _scan_once(string, idx)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/scanner.py", line 90, in _scan_once
    _scan_once, object_hook, object_pairs_hook, memo)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/decoder.py", line 198, in JSONObject
    value, end = scan_once(s, end)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/scanner.py", line 87, in _scan_once
    return parse_string(string, idx + 1, encoding, strict)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/decoder.py", line 118, in py_scanstring
    raise JSONDecodeError(msg, s, end)
simplejson.scanner.JSONDecodeError: Unpaired high surrogate: line 1 column 8 (char 7)

However, I'm not totally sure where the blame lies here. I've done a bit of research and this is what I've found.

  • I mentioned before I only have this problem under Linux. Locally on my mac it works fine. I've not been able to test Windows. I was pointed to this issue by @bigkevmcd: http://bugs.python.org/issue11489 which seems relevant but not conclusive.
  • I have tested this file with a few other languages for comparison - works fine in Java and Ruby. JavaScript seems to mostly work, although I noticed some display issues in the Chrome console.
  • I have discovered some other JSON documents cause the same error however they do work when I turn off the speedups. I've not been logging these as I'm turning off the speedups and trying again and then only logging those that still fail. I'll do another run (probably not until Monday now) and add logging of these so I can provide them.
  • In my tweet I was confused why it worked, failed and then worked after a pip re-install. This was with a different document and because simplejson was originally installed on the server with a package. Then when I pip installed it failed to install speedups - thus I "disabled" them without realising.

So, yeah, that's what I've found. I'm not sure how helpful it is. It does seem that the JSON that work without c speedups (but not with) only are causing a bug but it also seems to me that unpaired surrogate issues should either always work or always not work...

@etrepum

This comment has been minimized.

Show comment
Hide comment
@etrepum

etrepum Feb 23, 2013

Member

So the reason this works on Mac but not on Linux is because you're using Python 2.x which has two different configure-time options for unicode: wide (UCS4) and narrow (UCS2). Linux has pretty much always done UCS4 and everywhere else has done UCS2 (so that Py_UNICODE is the same type as wchar_t, mostly). This is historically a huge pain in the ass and it's fixed in Python 3, which has a far more correct unicode implementation. The behavior difference is because on a UCS4 build simplejson must combine surrogates, but on a UCS2 build it just blindly stuffs the UCS2 code points in there. It would be more correct if simplejson were to validate this even on UCS2 builds but it simply doesn't because it's easier not to bother (Python 2.x doesn't go through much effort to validate unicode in general).

I'll consider adding a flag to loosen up handling of unpaired surrogates, what behavior would you want? On some builds of Python, it's not possible to represent them as-is... so sane behavior would be something like replacing them with a placeholder or removing them entirely.

Member

etrepum commented Feb 23, 2013

So the reason this works on Mac but not on Linux is because you're using Python 2.x which has two different configure-time options for unicode: wide (UCS4) and narrow (UCS2). Linux has pretty much always done UCS4 and everywhere else has done UCS2 (so that Py_UNICODE is the same type as wchar_t, mostly). This is historically a huge pain in the ass and it's fixed in Python 3, which has a far more correct unicode implementation. The behavior difference is because on a UCS4 build simplejson must combine surrogates, but on a UCS2 build it just blindly stuffs the UCS2 code points in there. It would be more correct if simplejson were to validate this even on UCS2 builds but it simply doesn't because it's easier not to bother (Python 2.x doesn't go through much effort to validate unicode in general).

I'll consider adding a flag to loosen up handling of unpaired surrogates, what behavior would you want? On some builds of Python, it's not possible to represent them as-is... so sane behavior would be something like replacing them with a placeholder or removing them entirely.

@d0ugal

This comment has been minimized.

Show comment
Hide comment
@d0ugal

d0ugal Feb 23, 2013

Contributor

You obviously have a much stronger understanding of this than I do - so its hard for me to suggest what the best behaviour is, I'll give this some thought. On one hand being able to replace bad characters would be a great for us and certainly solve our use case. Incidentally, presumably I can remove them myself before parsing - perhaps that's should be the recommended way and maybe this is simply a documentation bug and a a note could be added for people that get stuck in this situation.

For the cases when I had different results with and without the speedups. Which side would you consider the bug? Is it a bug for not failing or a bug for failing? I'm assuming the inconsistency is a bug. I'll can get you all the samples of these if it helps on Monday (I'm avoiding my work machine until then).

Contributor

d0ugal commented Feb 23, 2013

You obviously have a much stronger understanding of this than I do - so its hard for me to suggest what the best behaviour is, I'll give this some thought. On one hand being able to replace bad characters would be a great for us and certainly solve our use case. Incidentally, presumably I can remove them myself before parsing - perhaps that's should be the recommended way and maybe this is simply a documentation bug and a a note could be added for people that get stuck in this situation.

For the cases when I had different results with and without the speedups. Which side would you consider the bug? Is it a bug for not failing or a bug for failing? I'm assuming the inconsistency is a bug. I'll can get you all the samples of these if it helps on Monday (I'm avoiding my work machine until then).

@etrepum

This comment has been minimized.

Show comment
Hide comment
@etrepum

etrepum Feb 24, 2013

Member

I'm a bit skeptical that there's difference with and without speedups, I
haven't tried reproducing this yet. As much evidence as you can provide
would be appreciated.

On Saturday, February 23, 2013, Dougal Matthews wrote:

You obviously have a much stronger understanding of this than I do - so
its hard for me to suggest what the best behaviour is, I'll give this some
thought. On one hand being able to replace bad characters would be a great
for us and certainly solve our use case. Incidentally, presumably I can
remove them myself before parsing - perhaps that's should be the
recommended way and maybe this is simply a documentation bug and a a note
could be added for people that get stuck in this situation.

For the cases when I had different results with and without the speedups.
Which side would you consider the bug? Is it a bug for not failing or a bug
for failing? I'm assuming the inconsistency is a bug. I'll can get you all
the samples of these if it helps on Monday (I'm avoiding my work machine
until then).


Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-13997223.

Member

etrepum commented Feb 24, 2013

I'm a bit skeptical that there's difference with and without speedups, I
haven't tried reproducing this yet. As much evidence as you can provide
would be appreciated.

On Saturday, February 23, 2013, Dougal Matthews wrote:

You obviously have a much stronger understanding of this than I do - so
its hard for me to suggest what the best behaviour is, I'll give this some
thought. On one hand being able to replace bad characters would be a great
for us and certainly solve our use case. Incidentally, presumably I can
remove them myself before parsing - perhaps that's should be the
recommended way and maybe this is simply a documentation bug and a a note
could be added for people that get stuck in this situation.

For the cases when I had different results with and without the speedups.
Which side would you consider the bug? Is it a bug for not failing or a bug
for failing? I'm assuming the inconsistency is a bug. I'll can get you all
the samples of these if it helps on Monday (I'm avoiding my work machine
until then).


Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-13997223.

@d0ugal

This comment has been minimized.

Show comment
Hide comment
@d0ugal

d0ugal Feb 25, 2013

Contributor

It looks like my findings may be out of date. I have tested this with simplejson 3.1.0 - I can't find any that fail with speedups and work without. I was only able to reproduce this behaviour with simplejson 2.3.2 (which our server still runs.)

Any idea what happens to the unpaired surrogates on this version? Here is my quick test. It shows 3.1.0 with speedups, without speedups and then 2.3.2 with and without (when it works on the final version).

(I'm going to try and trim down to the problem part of the file shortly.)

vagrant@precise64:/vagrant$ sudo pip install simplejson -U
Downloading/unpacking simplejson from http://pypi.python.org/packages/source/s/simplejson/simplejson-3.1.0.tar.gz#md5=71df0076d4a35d29bfea530cb8226c26
  Downloading simplejson-3.1.0.tar.gz (64kB): 64kB downloaded
  Running setup.py egg_info for package simplejson

Installing collected packages: simplejson
  Found existing installation: simplejson 3.0.0
    Uninstalling simplejson:
      Successfully uninstalled simplejson
  Running setup.py install for simplejson
    building 'simplejson._speedups' extension
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c simplejson/_speedups.c -o build/temp.linux-x86_64-2.7/simplejson/_speedups.o
    gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/simplejson/_speedups.o -o build/lib.linux-x86_64-2.7/simplejson/_speedups.so

Successfully installed simplejson
Cleaning up...
vagrant@precise64:/vagrant$ pip freeze | grep simplejson
Warning: cannot find svn location for distribute==0.6.24dev-r0
simplejson==3.1.0
vagrant@precise64:/vagrant$ python -c 'import simplejson as json; print json.load(open("failing2.json")).keys()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 398, in load
    use_decimal=use_decimal, **kw)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 454, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 393, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Unpaired high surrogate: line 1 column 330665 (char 330664)
vagrant@precise64:/vagrant$ python -c 'import simplejson as json; json._toggle_speedups(False); print json.load(open("failing2.json")).keys()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 398, in load
    use_decimal=use_decimal, **kw)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 454, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 393, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
  File "/usr/local/lib/python2.7/dist-packages/simplejson/scanner.py", line 119, in scan_once
    return _scan_once(string, idx)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/scanner.py", line 90, in _scan_once
    _scan_once, object_hook, object_pairs_hook, memo)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 198, in JSONObject
    value, end = scan_once(s, end)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/scanner.py", line 90, in _scan_once
    _scan_once, object_hook, object_pairs_hook, memo)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 198, in JSONObject
    value, end = scan_once(s, end)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/scanner.py", line 92, in _scan_once
    return parse_array((string, idx + 1), _scan_once)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 254, in JSONArray
    value, end = scan_once(s, end)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/scanner.py", line 90, in _scan_once
    _scan_once, object_hook, object_pairs_hook, memo)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 198, in JSONObject
    value, end = scan_once(s, end)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/scanner.py", line 90, in _scan_once
    _scan_once, object_hook, object_pairs_hook, memo)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 198, in JSONObject
    value, end = scan_once(s, end)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/scanner.py", line 87, in _scan_once
    return parse_string(string, idx + 1, encoding, strict)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 127, in py_scanstring
    raise JSONDecodeError(msg, s, end)
simplejson.scanner.JSONDecodeError: Unpaired high surrogate: line 1 column 330659 (char 330658)
vagrant@precise64:/vagrant$ sudo pip install simplejson==2.3.2
Downloading/unpacking simplejson==2.3.2
  Downloading simplejson-2.3.2.tar.gz (50kB): 50kB downloaded
  Running setup.py egg_info for package simplejson

Installing collected packages: simplejson
  Found existing installation: simplejson 3.1.0
    Uninstalling simplejson:
      Successfully uninstalled simplejson
  Running setup.py install for simplejson
    building 'simplejson._speedups' extension
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c simplejson/_speedups.c -o build/temp.linux-x86_64-2.7/simplejson/_speedups.o
    gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/simplejson/_speedups.o -o build/lib.linux-x86_64-2.7/simplejson/_speedups.so

Successfully installed simplejson
Cleaning up...
vagrant@precise64:/vagrant$ pip freeze | grep simplejson
Warning: cannot find svn location for distribute==0.6.24dev-r0
simplejson==2.3.2
vagrant@precise64:/vagrant$ python -c 'import simplejson as json; print json.load(open("failing2.json")).keys()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 357, in load
    use_decimal=use_decimal, **kw)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 413, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 402, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 418, in raw_decode
    obj, end = self.scan_once(s, idx)
simplejson.decoder.JSONDecodeError: Unpaired high surrogate: line 1 column 330664 (char 330664)
vagrant@precise64:/vagrant$ python -c 'import simplejson as json; json._toggle_speedups(False); print json.load(open("failing2.json")).keys()'
[u'hits', u'_shards', u'took', u'_scroll_id', u'timed_out']
Contributor

d0ugal commented Feb 25, 2013

It looks like my findings may be out of date. I have tested this with simplejson 3.1.0 - I can't find any that fail with speedups and work without. I was only able to reproduce this behaviour with simplejson 2.3.2 (which our server still runs.)

Any idea what happens to the unpaired surrogates on this version? Here is my quick test. It shows 3.1.0 with speedups, without speedups and then 2.3.2 with and without (when it works on the final version).

(I'm going to try and trim down to the problem part of the file shortly.)

vagrant@precise64:/vagrant$ sudo pip install simplejson -U
Downloading/unpacking simplejson from http://pypi.python.org/packages/source/s/simplejson/simplejson-3.1.0.tar.gz#md5=71df0076d4a35d29bfea530cb8226c26
  Downloading simplejson-3.1.0.tar.gz (64kB): 64kB downloaded
  Running setup.py egg_info for package simplejson

Installing collected packages: simplejson
  Found existing installation: simplejson 3.0.0
    Uninstalling simplejson:
      Successfully uninstalled simplejson
  Running setup.py install for simplejson
    building 'simplejson._speedups' extension
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c simplejson/_speedups.c -o build/temp.linux-x86_64-2.7/simplejson/_speedups.o
    gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/simplejson/_speedups.o -o build/lib.linux-x86_64-2.7/simplejson/_speedups.so

Successfully installed simplejson
Cleaning up...
vagrant@precise64:/vagrant$ pip freeze | grep simplejson
Warning: cannot find svn location for distribute==0.6.24dev-r0
simplejson==3.1.0
vagrant@precise64:/vagrant$ python -c 'import simplejson as json; print json.load(open("failing2.json")).keys()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 398, in load
    use_decimal=use_decimal, **kw)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 454, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 393, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Unpaired high surrogate: line 1 column 330665 (char 330664)
vagrant@precise64:/vagrant$ python -c 'import simplejson as json; json._toggle_speedups(False); print json.load(open("failing2.json")).keys()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 398, in load
    use_decimal=use_decimal, **kw)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 454, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 393, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
  File "/usr/local/lib/python2.7/dist-packages/simplejson/scanner.py", line 119, in scan_once
    return _scan_once(string, idx)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/scanner.py", line 90, in _scan_once
    _scan_once, object_hook, object_pairs_hook, memo)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 198, in JSONObject
    value, end = scan_once(s, end)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/scanner.py", line 90, in _scan_once
    _scan_once, object_hook, object_pairs_hook, memo)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 198, in JSONObject
    value, end = scan_once(s, end)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/scanner.py", line 92, in _scan_once
    return parse_array((string, idx + 1), _scan_once)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 254, in JSONArray
    value, end = scan_once(s, end)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/scanner.py", line 90, in _scan_once
    _scan_once, object_hook, object_pairs_hook, memo)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 198, in JSONObject
    value, end = scan_once(s, end)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/scanner.py", line 90, in _scan_once
    _scan_once, object_hook, object_pairs_hook, memo)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 198, in JSONObject
    value, end = scan_once(s, end)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/scanner.py", line 87, in _scan_once
    return parse_string(string, idx + 1, encoding, strict)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 127, in py_scanstring
    raise JSONDecodeError(msg, s, end)
simplejson.scanner.JSONDecodeError: Unpaired high surrogate: line 1 column 330659 (char 330658)
vagrant@precise64:/vagrant$ sudo pip install simplejson==2.3.2
Downloading/unpacking simplejson==2.3.2
  Downloading simplejson-2.3.2.tar.gz (50kB): 50kB downloaded
  Running setup.py egg_info for package simplejson

Installing collected packages: simplejson
  Found existing installation: simplejson 3.1.0
    Uninstalling simplejson:
      Successfully uninstalled simplejson
  Running setup.py install for simplejson
    building 'simplejson._speedups' extension
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c simplejson/_speedups.c -o build/temp.linux-x86_64-2.7/simplejson/_speedups.o
    gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/simplejson/_speedups.o -o build/lib.linux-x86_64-2.7/simplejson/_speedups.so

Successfully installed simplejson
Cleaning up...
vagrant@precise64:/vagrant$ pip freeze | grep simplejson
Warning: cannot find svn location for distribute==0.6.24dev-r0
simplejson==2.3.2
vagrant@precise64:/vagrant$ python -c 'import simplejson as json; print json.load(open("failing2.json")).keys()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 357, in load
    use_decimal=use_decimal, **kw)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 413, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 402, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 418, in raw_decode
    obj, end = self.scan_once(s, idx)
simplejson.decoder.JSONDecodeError: Unpaired high surrogate: line 1 column 330664 (char 330664)
vagrant@precise64:/vagrant$ python -c 'import simplejson as json; json._toggle_speedups(False); print json.load(open("failing2.json")).keys()'
[u'hits', u'_shards', u'took', u'_scroll_id', u'timed_out']
@d0ugal

This comment has been minimized.

Show comment
Hide comment
@d0ugal

d0ugal Feb 25, 2013

Contributor

Example of an inconsistent result, again from an old version 2.3.2 - so it may just be that this has been fixed.

The JSON in question is;

{"a": "\uda90\uac14"}

Test results;

vagrant@precise64:/vagrant$ python -c 'import simplejson as json; print json.__version__; print json.loads("{\"a\": \"\uda90\uac14\"}").keys()';
2.3.2
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 413, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 402, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 418, in raw_decode
    obj, end = self.scan_once(s, idx)
simplejson.decoder.JSONDecodeError: Unpaired high surrogate: line 1 column 14 (char 14)

vagrant@precise64:/vagrant$ python -c 'import simplejson as json; json._toggle_speedups(False); print json.loads("{\"a\": \"\uda90\uac14\"}").keys()';
[u'a']
Contributor

d0ugal commented Feb 25, 2013

Example of an inconsistent result, again from an old version 2.3.2 - so it may just be that this has been fixed.

The JSON in question is;

{"a": "\uda90\uac14"}

Test results;

vagrant@precise64:/vagrant$ python -c 'import simplejson as json; print json.__version__; print json.loads("{\"a\": \"\uda90\uac14\"}").keys()';
2.3.2
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 413, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 402, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 418, in raw_decode
    obj, end = self.scan_once(s, idx)
simplejson.decoder.JSONDecodeError: Unpaired high surrogate: line 1 column 14 (char 14)

vagrant@precise64:/vagrant$ python -c 'import simplejson as json; json._toggle_speedups(False); print json.loads("{\"a\": \"\uda90\uac14\"}").keys()';
[u'a']
@etrepum

This comment has been minimized.

Show comment
Hide comment
@etrepum

etrepum Feb 25, 2013

Member

If you go far enough in the past, anything is possible. I'm only really
interested in current behavior, there are no maintenance releases for old
versions.

On Monday, February 25, 2013, Dougal Matthews wrote:

Example of an inconsistent result, again from an old version 2.3.2 - so it
may just be that this has been fixed.

The JSON in question is;

{"a": "\uda90\uac14"}

Test results;

vagrant@precise64:/vagrant$ python -c 'import simplejson as json; print json.version; print json.loads("{"a": "\uda90\uac14"}").keys()';
2.3.2
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/simplejson/init.py", line 413, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 402, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 418, in raw_decode
obj, end = self.scan_once(s, idx)
simplejson.decoder.JSONDecodeError: Unpaired high surrogate: line 1 column 14 (char 14)

vagrant@precise64:/vagrant$ python -c 'import simplejson as json; json._toggle_speedups(False); print json.loads("{"a": "\uda90\uac14"}").keys()';
[u'a']


Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-14049234.

Member

etrepum commented Feb 25, 2013

If you go far enough in the past, anything is possible. I'm only really
interested in current behavior, there are no maintenance releases for old
versions.

On Monday, February 25, 2013, Dougal Matthews wrote:

Example of an inconsistent result, again from an old version 2.3.2 - so it
may just be that this has been fixed.

The JSON in question is;

{"a": "\uda90\uac14"}

Test results;

vagrant@precise64:/vagrant$ python -c 'import simplejson as json; print json.version; print json.loads("{"a": "\uda90\uac14"}").keys()';
2.3.2
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/simplejson/init.py", line 413, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 402, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 418, in raw_decode
obj, end = self.scan_once(s, idx)
simplejson.decoder.JSONDecodeError: Unpaired high surrogate: line 1 column 14 (char 14)

vagrant@precise64:/vagrant$ python -c 'import simplejson as json; json._toggle_speedups(False); print json.loads("{"a": "\uda90\uac14"}").keys()';
[u'a']


Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-14049234.

@d0ugal

This comment has been minimized.

Show comment
Hide comment
@d0ugal

d0ugal Feb 25, 2013

Contributor

Indeed - I wasn't suggesting that it should be fixed or changed etc. but simply trying to understand what was happening in this case. i.e. how was it able to work previously? What was the behaviour then? Sorry if I'm failing to be clear.

Contributor

d0ugal commented Feb 25, 2013

Indeed - I wasn't suggesting that it should be fixed or changed etc. but simply trying to understand what was happening in this case. i.e. how was it able to work previously? What was the behaviour then? Sorry if I'm failing to be clear.

@etrepum

This comment has been minimized.

Show comment
Hide comment
@etrepum

etrepum Feb 25, 2013

Member

If there was different behavior, it was a bug. The behavior before was
probably just garbage in, garbage out. I'm hesitant to make that a
supported behavior because it will just cause problems somewhere else
(especially in Python 3).

On Monday, February 25, 2013, Dougal Matthews wrote:

Indeed - I wasn't suggesting that it should be fixed or changed etc. but
simply trying to understand what was happening in this case. i.e. how was
it able to work previously? What was the behaviour then? Sorry if I'm
failing to be clear.


Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-14057221.

Member

etrepum commented Feb 25, 2013

If there was different behavior, it was a bug. The behavior before was
probably just garbage in, garbage out. I'm hesitant to make that a
supported behavior because it will just cause problems somewhere else
(especially in Python 3).

On Monday, February 25, 2013, Dougal Matthews wrote:

Indeed - I wasn't suggesting that it should be fixed or changed etc. but
simply trying to understand what was happening in this case. i.e. how was
it able to work previously? What was the behaviour then? Sorry if I'm
failing to be clear.


Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-14057221.

@d0ugal

This comment has been minimized.

Show comment
Hide comment
@d0ugal

d0ugal Feb 26, 2013

Contributor

Ok, thanks. I was just trying to understand how it "worked" before. I'm fairly sure this shouldn't be changed now. If anything, I think a documentation change would be best. Are you able to explain how to remove/replace the surrogates?

Contributor

d0ugal commented Feb 26, 2013

Ok, thanks. I was just trying to understand how it "worked" before. I'm fairly sure this shouldn't be changed now. If anything, I think a documentation change would be best. Are you able to explain how to remove/replace the surrogates?

@etrepum

This comment has been minimized.

Show comment
Hide comment
@etrepum

etrepum Feb 26, 2013

Member

Since these unpaired surrogates occur as escape sequences it's not
straightforward to remove them without making changes to simplejson. The
only simple way I can think of is to catch the exception and run it again
with an edited document. Not ideal. I'd rather modify simplejson than try
and document something like that.

On Tuesday, February 26, 2013, Dougal Matthews wrote:

Ok, thanks. I was just trying to understand how it "worked" before. I'm
fairly sure this shouldn't be changed now. If anything, I think a
documentation change would be best. Are you able to explain how to
remove/replace the surrogates?


Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-14103380
.

Member

etrepum commented Feb 26, 2013

Since these unpaired surrogates occur as escape sequences it's not
straightforward to remove them without making changes to simplejson. The
only simple way I can think of is to catch the exception and run it again
with an edited document. Not ideal. I'd rather modify simplejson than try
and document something like that.

On Tuesday, February 26, 2013, Dougal Matthews wrote:

Ok, thanks. I was just trying to understand how it "worked" before. I'm
fairly sure this shouldn't be changed now. If anything, I think a
documentation change would be best. Are you able to explain how to
remove/replace the surrogates?


Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-14103380
.

@etrepum

This comment has been minimized.

Show comment
Hide comment
@etrepum

etrepum May 1, 2013

Member

I haven't forgotten about this, just haven't had time to do anything about it yet. It's on my list for 3.3.0.

Member

etrepum commented May 1, 2013

I haven't forgotten about this, just haven't had time to do anything about it yet. It's on my list for 3.3.0.

@d0ugal

This comment has been minimized.

Show comment
Hide comment
@d0ugal

d0ugal May 1, 2013

Contributor

Awesome - thanks. Let me know if I can help at all. I worked around our issue with horrific regexes to find IDs and then find the documents and repair them. 😄

Contributor

d0ugal commented May 1, 2013

Awesome - thanks. Let me know if I can help at all. I worked around our issue with horrific regexes to find IDs and then find the documents and repair them. 😄

@ltrabuco

This comment has been minimized.

Show comment
Hide comment
@ltrabuco

ltrabuco May 2, 2013

@d0ugal Would you mind sharing how you repaired the documents?

ltrabuco commented May 2, 2013

@d0ugal Would you mind sharing how you repaired the documents?

@d0ugal

This comment has been minimized.

Show comment
Hide comment
@d0ugal

d0ugal May 2, 2013

Contributor

It was fairly horrific and I don't imagine it's wise as a general approach. However, with the number of documents we have it works for now. Basically, use a really old simplejson that is more lax about invalid Unicode and then the few that still failed have been updated and the invalid characters removed.

Contributor

d0ugal commented May 2, 2013

It was fairly horrific and I don't imagine it's wise as a general approach. However, with the number of documents we have it works for now. Basically, use a really old simplejson that is more lax about invalid Unicode and then the few that still failed have been updated and the invalid characters removed.

@ltrabuco

This comment has been minimized.

Show comment
Hide comment
@ltrabuco

ltrabuco commented May 2, 2013

@d0ugal Thanks!

@serhiy-storchaka

This comment has been minimized.

Show comment
Hide comment
@serhiy-storchaka

serhiy-storchaka May 4, 2013

Contributor

I think it will be good to have an "error" parameter in decoder/encoder. error="strict" (by default) corresponds to current behavior on wide builds, while error="surrogatepass" corresponds to behavior on narrow builds and non-strict parsers on other languages. Other error handlers are possible but less important.

I have opened an issue on CPython bugtracker [1] and I'm going to prepare patch for standard json module and then port it to simplejson.

[1] http://bugs.python.org/issue17906

Contributor

serhiy-storchaka commented May 4, 2013

I think it will be good to have an "error" parameter in decoder/encoder. error="strict" (by default) corresponds to current behavior on wide builds, while error="surrogatepass" corresponds to behavior on narrow builds and non-strict parsers on other languages. Other error handlers are possible but less important.

I have opened an issue on CPython bugtracker [1] and I'm going to prepare patch for standard json module and then port it to simplejson.

[1] http://bugs.python.org/issue17906

@etrepum

This comment has been minimized.

Show comment
Hide comment
@etrepum

etrepum May 4, 2013

Member

Are you sure "error" is the best name for this? There are a number of other
options to control other kinds of errors. I know there's a parallel to
str/unicode but I think a more explicit name might be more appropriate
here.

On Saturday, May 4, 2013, Serhiy Storchaka wrote:

I think it will be good to have an "error" parameter in decoder/encoder.
error="strict" (by default) corresponds to current behavior on wide builds,
while error="surrogatepass" corresponds to behavior on narrow builds and
non-strict parsers on other languages. Other error handlers are possible
but less important.

I have opened an issue on CPython bugtracker [1] and I'm going to prepare
patch for standard json module and then port it to simplejson.

[1] http://bugs.python.org/issue17906


Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-17434306
.

Member

etrepum commented May 4, 2013

Are you sure "error" is the best name for this? There are a number of other
options to control other kinds of errors. I know there's a parallel to
str/unicode but I think a more explicit name might be more appropriate
here.

On Saturday, May 4, 2013, Serhiy Storchaka wrote:

I think it will be good to have an "error" parameter in decoder/encoder.
error="strict" (by default) corresponds to current behavior on wide builds,
while error="surrogatepass" corresponds to behavior on narrow builds and
non-strict parsers on other languages. Other error handlers are possible
but less important.

I have opened an issue on CPython bugtracker [1] and I'm going to prepare
patch for standard json module and then port it to simplejson.

[1] http://bugs.python.org/issue17906


Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-17434306
.

@d0ugal

This comment has been minimized.

Show comment
Hide comment
@d0ugal

d0ugal May 7, 2013

Contributor

I don't think "error" is the best name either, for the builtin unicode() the "errors" arg works because at that point you are in the unicode problem space so its clear. Something like unicode_errors while rather verbose would be clearer - I can't think of a better alternative at the moment.

Contributor

d0ugal commented May 7, 2013

I don't think "error" is the best name either, for the builtin unicode() the "errors" arg works because at that point you are in the unicode problem space so its clear. Something like unicode_errors while rather verbose would be clearer - I can't think of a better alternative at the moment.

@serhiy-storchaka

This comment has been minimized.

Show comment
Hide comment
@serhiy-storchaka

serhiy-storchaka May 7, 2013

Contributor

After investigating the problem deeper, I see that new parameter is not needed. RFC 4627 does not make exceptions for the range 0xD800-0xDFFF, and the decoder must accept lone surrogates, both escaped and unescaped. See a patch at http://bugs.python.org/issue17906.

Contributor

serhiy-storchaka commented May 7, 2013

After investigating the problem deeper, I see that new parameter is not needed. RFC 4627 does not make exceptions for the range 0xD800-0xDFFF, and the decoder must accept lone surrogates, both escaped and unescaped. See a patch at http://bugs.python.org/issue17906.

@etrepum etrepum closed this May 8, 2013

jsonn pushed a commit to jsonn/pkgsrc that referenced this issue Jan 3, 2014

rodent
Update to latest release, 3.3.1. Add:
PYTHON_VERSIONS_ACCEPTED=       33 27 26
PYTHON_VERSIONS_INCLUDE_3X=     yes

Not clear from documentation if this package works with python32.

From CHANGES.txt:

Version 3.3.1 released 2013-10-05

* JSONDecodeError exceptions can now be pickled
  simplejson/simplejson#78

Version 3.3.0 released 2013-05-07

* Unpaired surrogates once again pass through the decoder, to match older
  behavior and the RFC-4627 spec.
  simplejson/simplejson#62

Version 3.2.0 released 2013-05-01

* New ignore_nan kwarg in encoder that serializes out
  of range floats (Infinity, -Infinity, NaN) as null for ECMA-262
  compliance.
  simplejson/simplejson#63
* New for_json kwarg in encoder to make it possible to for
  subclasses of dict and list to be specialized.
  simplejson/simplejson#69

Version 3.1.3 released 2013-04-06

* Updated documentation to discourage subclassing whenever possible.
   default, object_hook, and object_pairs_hook provide almost all of
   the functionality of subclassing.

Version 3.1.2 released 2013-03-20

* Updated documentation to reflect separators behavior when indent is
  not None
  simplejson/simplejson#59
* Test suite should be compatible with debug builds of Python 2.x and 3.x
  simplejson/simplejson#65

Version 3.1.1 released 2013-02-21

* setup.py now has another workaround for Windows machines without
  MSVC installed
  http://bugs.python.org/issue7511

Version 3.1.0 released 2013-02-21

* Updated JSON conformance test suite
  http://bugs.python.org/issue16559
* simplejson.tool tests and bugfix for Python 3.x
  http://bugs.python.org/issue16549
* Improve error messages for certain kinds of truncated input
  http://bugs.python.org/issue16009
* Moved JSONDecodeError to json.scanner (still available for import
  from json.decoder)
* Changed scanner to use JSONDecodeError directly rather than
  StopIteration to improve error messages

Version 3.0.9 released 2013-02-21

* Fix an off-by-one error in the colno property of JSONDecodeError
  (when lineno == 1)
  http://bugs.python.org/issue17225

Version 3.0.8 released 2013-02-19

* Fix a Python 2.x compiler warning for narrow unicode builds
  simplejson/simplejson#56

Version 3.0.7 released 2013-01-11

* NOTE: this release only changes the license.
* simplejson is now dual-licensed software, MIT or AFL v2.1. It is
  also made explicit that this code is also licensed to the PSF under
  a Contributor Agreement.

Version 3.0.6 released 2013-01-11

* Fix for major Python 2.x ensure_ascii=False encoding regression
  introduced in simplejson 3.0.0. If you use this setting, please
  upgrade immediately.
  simplejson/simplejson#50

Version 3.0.5 released 2013-01-03

* NOTE: this release only changes the tests, it is
  not essential to upgrade
* Tests now run with deprecation warnings printed
* Fixed Python 3 syntax error in simplejson.tool
  simplejson/simplejson#49
* Fixed Python 3.3 deprecation warnings in test suite
  simplejson/simplejson#48

Version 3.0.4 released 2013-01-02

* MSVC compatibility for Python 3.3
  simplejson/simplejson#47

Version 3.0.3 released 2013-01-01

* Fixes for bugs introduced in 3.0.2
* Fixes for Python 2.5 compatibility
* MSVC compatibility for Python 2.x
  simplejson/simplejson#46

Version 3.0.2 released 2013-01-01

* THIS VERSION HAS BEEN REMOVED
* Missed a changeset to _speedups.c in the 3.0.1 branch cut

Version 3.0.1 released 2013-01-01

* THIS VERSION HAS BEEN REMOVED
* Add accumulator optimization to encoder, equivalent to the usage of
  `_Py_Accu` in the Python 3.3 json library. Only relevant if encoding
  very large JSON documents.

Version 3.0.0 released 2012-12-30

* Python 3.3 is now supported, thanks to Vinay Sajip
  simplejson/simplejson#8
* `sort_keys`/`item_sort_key` now sort on the stringified verison of the
  key, rather than the original object. This ensures that the sort
  only compares string types and makes the behavior consistent between
  Python 2.x and Python 3.x.
* Like other number types, Decimal instances used as keys are now
  coerced to strings when use_decimal is True.

jsonn pushed a commit to jsonn/pkgsrc that referenced this issue Jan 20, 2014

rodent
Update to latest release, 3.3.1. Add:
PYTHON_VERSIONS_ACCEPTED=       33 27 26
PYTHON_VERSIONS_INCLUDE_3X=     yes

Not clear from documentation if this package works with python32.

From CHANGES.txt:

Version 3.3.1 released 2013-10-05

* JSONDecodeError exceptions can now be pickled
  simplejson/simplejson#78

Version 3.3.0 released 2013-05-07

* Unpaired surrogates once again pass through the decoder, to match older
  behavior and the RFC-4627 spec.
  simplejson/simplejson#62

Version 3.2.0 released 2013-05-01

* New ignore_nan kwarg in encoder that serializes out
  of range floats (Infinity, -Infinity, NaN) as null for ECMA-262
  compliance.
  simplejson/simplejson#63
* New for_json kwarg in encoder to make it possible to for
  subclasses of dict and list to be specialized.
  simplejson/simplejson#69

Version 3.1.3 released 2013-04-06

* Updated documentation to discourage subclassing whenever possible.
   default, object_hook, and object_pairs_hook provide almost all of
   the functionality of subclassing.

Version 3.1.2 released 2013-03-20

* Updated documentation to reflect separators behavior when indent is
  not None
  simplejson/simplejson#59
* Test suite should be compatible with debug builds of Python 2.x and 3.x
  simplejson/simplejson#65

Version 3.1.1 released 2013-02-21

* setup.py now has another workaround for Windows machines without
  MSVC installed
  http://bugs.python.org/issue7511

Version 3.1.0 released 2013-02-21

* Updated JSON conformance test suite
  http://bugs.python.org/issue16559
* simplejson.tool tests and bugfix for Python 3.x
  http://bugs.python.org/issue16549
* Improve error messages for certain kinds of truncated input
  http://bugs.python.org/issue16009
* Moved JSONDecodeError to json.scanner (still available for import
  from json.decoder)
* Changed scanner to use JSONDecodeError directly rather than
  StopIteration to improve error messages

Version 3.0.9 released 2013-02-21

* Fix an off-by-one error in the colno property of JSONDecodeError
  (when lineno == 1)
  http://bugs.python.org/issue17225

Version 3.0.8 released 2013-02-19

* Fix a Python 2.x compiler warning for narrow unicode builds
  simplejson/simplejson#56

Version 3.0.7 released 2013-01-11

* NOTE: this release only changes the license.
* simplejson is now dual-licensed software, MIT or AFL v2.1. It is
  also made explicit that this code is also licensed to the PSF under
  a Contributor Agreement.

Version 3.0.6 released 2013-01-11

* Fix for major Python 2.x ensure_ascii=False encoding regression
  introduced in simplejson 3.0.0. If you use this setting, please
  upgrade immediately.
  simplejson/simplejson#50

Version 3.0.5 released 2013-01-03

* NOTE: this release only changes the tests, it is
  not essential to upgrade
* Tests now run with deprecation warnings printed
* Fixed Python 3 syntax error in simplejson.tool
  simplejson/simplejson#49
* Fixed Python 3.3 deprecation warnings in test suite
  simplejson/simplejson#48

Version 3.0.4 released 2013-01-02

* MSVC compatibility for Python 3.3
  simplejson/simplejson#47

Version 3.0.3 released 2013-01-01

* Fixes for bugs introduced in 3.0.2
* Fixes for Python 2.5 compatibility
* MSVC compatibility for Python 2.x
  simplejson/simplejson#46

Version 3.0.2 released 2013-01-01

* THIS VERSION HAS BEEN REMOVED
* Missed a changeset to _speedups.c in the 3.0.1 branch cut

Version 3.0.1 released 2013-01-01

* THIS VERSION HAS BEEN REMOVED
* Add accumulator optimization to encoder, equivalent to the usage of
  `_Py_Accu` in the Python 3.3 json library. Only relevant if encoding
  very large JSON documents.

Version 3.0.0 released 2012-12-30

* Python 3.3 is now supported, thanks to Vinay Sajip
  simplejson/simplejson#8
* `sort_keys`/`item_sort_key` now sort on the stringified verison of the
  key, rather than the original object. This ensures that the sort
  only compares string types and makes the behavior consistent between
  Python 2.x and Python 3.x.
* Like other number types, Decimal instances used as keys are now
  coerced to strings when use_decimal is True.

jsonn pushed a commit to jsonn/pkgsrc that referenced this issue Mar 12, 2014

rodent
Update to latest release, 3.3.1. Add:
PYTHON_VERSIONS_ACCEPTED=       33 27 26
PYTHON_VERSIONS_INCLUDE_3X=     yes

Not clear from documentation if this package works with python32.

From CHANGES.txt:

Version 3.3.1 released 2013-10-05

* JSONDecodeError exceptions can now be pickled
  simplejson/simplejson#78

Version 3.3.0 released 2013-05-07

* Unpaired surrogates once again pass through the decoder, to match older
  behavior and the RFC-4627 spec.
  simplejson/simplejson#62

Version 3.2.0 released 2013-05-01

* New ignore_nan kwarg in encoder that serializes out
  of range floats (Infinity, -Infinity, NaN) as null for ECMA-262
  compliance.
  simplejson/simplejson#63
* New for_json kwarg in encoder to make it possible to for
  subclasses of dict and list to be specialized.
  simplejson/simplejson#69

Version 3.1.3 released 2013-04-06

* Updated documentation to discourage subclassing whenever possible.
   default, object_hook, and object_pairs_hook provide almost all of
   the functionality of subclassing.

Version 3.1.2 released 2013-03-20

* Updated documentation to reflect separators behavior when indent is
  not None
  simplejson/simplejson#59
* Test suite should be compatible with debug builds of Python 2.x and 3.x
  simplejson/simplejson#65

Version 3.1.1 released 2013-02-21

* setup.py now has another workaround for Windows machines without
  MSVC installed
  http://bugs.python.org/issue7511

Version 3.1.0 released 2013-02-21

* Updated JSON conformance test suite
  http://bugs.python.org/issue16559
* simplejson.tool tests and bugfix for Python 3.x
  http://bugs.python.org/issue16549
* Improve error messages for certain kinds of truncated input
  http://bugs.python.org/issue16009
* Moved JSONDecodeError to json.scanner (still available for import
  from json.decoder)
* Changed scanner to use JSONDecodeError directly rather than
  StopIteration to improve error messages

Version 3.0.9 released 2013-02-21

* Fix an off-by-one error in the colno property of JSONDecodeError
  (when lineno == 1)
  http://bugs.python.org/issue17225

Version 3.0.8 released 2013-02-19

* Fix a Python 2.x compiler warning for narrow unicode builds
  simplejson/simplejson#56

Version 3.0.7 released 2013-01-11

* NOTE: this release only changes the license.
* simplejson is now dual-licensed software, MIT or AFL v2.1. It is
  also made explicit that this code is also licensed to the PSF under
  a Contributor Agreement.

Version 3.0.6 released 2013-01-11

* Fix for major Python 2.x ensure_ascii=False encoding regression
  introduced in simplejson 3.0.0. If you use this setting, please
  upgrade immediately.
  simplejson/simplejson#50

Version 3.0.5 released 2013-01-03

* NOTE: this release only changes the tests, it is
  not essential to upgrade
* Tests now run with deprecation warnings printed
* Fixed Python 3 syntax error in simplejson.tool
  simplejson/simplejson#49
* Fixed Python 3.3 deprecation warnings in test suite
  simplejson/simplejson#48

Version 3.0.4 released 2013-01-02

* MSVC compatibility for Python 3.3
  simplejson/simplejson#47

Version 3.0.3 released 2013-01-01

* Fixes for bugs introduced in 3.0.2
* Fixes for Python 2.5 compatibility
* MSVC compatibility for Python 2.x
  simplejson/simplejson#46

Version 3.0.2 released 2013-01-01

* THIS VERSION HAS BEEN REMOVED
* Missed a changeset to _speedups.c in the 3.0.1 branch cut

Version 3.0.1 released 2013-01-01

* THIS VERSION HAS BEEN REMOVED
* Add accumulator optimization to encoder, equivalent to the usage of
  `_Py_Accu` in the Python 3.3 json library. Only relevant if encoding
  very large JSON documents.

Version 3.0.0 released 2012-12-30

* Python 3.3 is now supported, thanks to Vinay Sajip
  simplejson/simplejson#8
* `sort_keys`/`item_sort_key` now sort on the stringified verison of the
  key, rather than the original object. This ensures that the sort
  only compares string types and makes the behavior consistent between
  Python 2.x and Python 3.x.
* Like other number types, Decimal instances used as keys are now
  coerced to strings when use_decimal is True.

jsonn pushed a commit to jsonn/pkgsrc that referenced this issue Oct 11, 2014

rodent
Update to latest release, 3.3.1. Add:
PYTHON_VERSIONS_ACCEPTED=       33 27 26
PYTHON_VERSIONS_INCLUDE_3X=     yes

Not clear from documentation if this package works with python32.

From CHANGES.txt:

Version 3.3.1 released 2013-10-05

* JSONDecodeError exceptions can now be pickled
  simplejson/simplejson#78

Version 3.3.0 released 2013-05-07

* Unpaired surrogates once again pass through the decoder, to match older
  behavior and the RFC-4627 spec.
  simplejson/simplejson#62

Version 3.2.0 released 2013-05-01

* New ignore_nan kwarg in encoder that serializes out
  of range floats (Infinity, -Infinity, NaN) as null for ECMA-262
  compliance.
  simplejson/simplejson#63
* New for_json kwarg in encoder to make it possible to for
  subclasses of dict and list to be specialized.
  simplejson/simplejson#69

Version 3.1.3 released 2013-04-06

* Updated documentation to discourage subclassing whenever possible.
   default, object_hook, and object_pairs_hook provide almost all of
   the functionality of subclassing.

Version 3.1.2 released 2013-03-20

* Updated documentation to reflect separators behavior when indent is
  not None
  simplejson/simplejson#59
* Test suite should be compatible with debug builds of Python 2.x and 3.x
  simplejson/simplejson#65

Version 3.1.1 released 2013-02-21

* setup.py now has another workaround for Windows machines without
  MSVC installed
  http://bugs.python.org/issue7511

Version 3.1.0 released 2013-02-21

* Updated JSON conformance test suite
  http://bugs.python.org/issue16559
* simplejson.tool tests and bugfix for Python 3.x
  http://bugs.python.org/issue16549
* Improve error messages for certain kinds of truncated input
  http://bugs.python.org/issue16009
* Moved JSONDecodeError to json.scanner (still available for import
  from json.decoder)
* Changed scanner to use JSONDecodeError directly rather than
  StopIteration to improve error messages

Version 3.0.9 released 2013-02-21

* Fix an off-by-one error in the colno property of JSONDecodeError
  (when lineno == 1)
  http://bugs.python.org/issue17225

Version 3.0.8 released 2013-02-19

* Fix a Python 2.x compiler warning for narrow unicode builds
  simplejson/simplejson#56

Version 3.0.7 released 2013-01-11

* NOTE: this release only changes the license.
* simplejson is now dual-licensed software, MIT or AFL v2.1. It is
  also made explicit that this code is also licensed to the PSF under
  a Contributor Agreement.

Version 3.0.6 released 2013-01-11

* Fix for major Python 2.x ensure_ascii=False encoding regression
  introduced in simplejson 3.0.0. If you use this setting, please
  upgrade immediately.
  simplejson/simplejson#50

Version 3.0.5 released 2013-01-03

* NOTE: this release only changes the tests, it is
  not essential to upgrade
* Tests now run with deprecation warnings printed
* Fixed Python 3 syntax error in simplejson.tool
  simplejson/simplejson#49
* Fixed Python 3.3 deprecation warnings in test suite
  simplejson/simplejson#48

Version 3.0.4 released 2013-01-02

* MSVC compatibility for Python 3.3
  simplejson/simplejson#47

Version 3.0.3 released 2013-01-01

* Fixes for bugs introduced in 3.0.2
* Fixes for Python 2.5 compatibility
* MSVC compatibility for Python 2.x
  simplejson/simplejson#46

Version 3.0.2 released 2013-01-01

* THIS VERSION HAS BEEN REMOVED
* Missed a changeset to _speedups.c in the 3.0.1 branch cut

Version 3.0.1 released 2013-01-01

* THIS VERSION HAS BEEN REMOVED
* Add accumulator optimization to encoder, equivalent to the usage of
  `_Py_Accu` in the Python 3.3 json library. Only relevant if encoding
  very large JSON documents.

Version 3.0.0 released 2012-12-30

* Python 3.3 is now supported, thanks to Vinay Sajip
  simplejson/simplejson#8
* `sort_keys`/`item_sort_key` now sort on the stringified verison of the
  key, rather than the original object. This ensures that the sort
  only compares string types and makes the behavior consistent between
  Python 2.x and Python 3.x.
* Like other number types, Decimal instances used as keys are now
  coerced to strings when use_decimal is True.

notxarb referenced this issue in nanobox-io/nanobox-pkgsrc-lite May 8, 2015

Initial copy of pkgsrc project
Squashed commit of the following:

commit 130c8eb2df4b63034353a232af8e5fa4526693a5
Merge: 5888b51 a980f97
Author: Jonathan Perkin <jperkin@joyent.com>
Date:   Sun Apr 26 23:00:03 2015 +0100

    Merge branch 'joyent/feature/miscfix/2015Q1' into joyent/release/2015Q1

commit a980f97415f48af9407551d6e515a09104079a8d
Merge: 1cbc272 fdbe317
Author: Jonathan Perkin <jperkin@joyent.com>
Date:   Sun Apr 26 22:55:29 2015 +0100

    Merge branch 'pkgsrc_2015Q1' into joyent/feature/miscfix/2015Q1

commit fdbe317df493f113aa0a8bdcf1ec1b6b65937bf4
Author: tron <tron>
Date:   Sun Apr 26 08:47:38 2015 +0000

    Pullup tickets #4684, #4685 and #4686.

commit faae9e8bdb9fe7ab45c47c87b904784650e473d4
Author: tron <tron>
Date:   Sun Apr 26 08:47:12 2015 +0000

    Pullup ticket #4686 - requested by taca
    lang/ruby22-base: security update

    Revisions pulled up:
    - lang/ruby/rubyversion.mk                                      1.141
    - lang/ruby22-base/distinfo                                     1.2

    ---
       Module Name:	pkgsrc
       Committed By:	taca
       Date:		Sun Apr 19 16:25:10 UTC 2015

       Modified Files:
       	pkgsrc/lang/ruby: rubyversion.mk
       	pkgsrc/lang/ruby22-base: distinfo

       Log Message:
       Update ruby22-base and ruby22 package to 2.2.2.

       >From release announce:

       We are pleased to announce the release of Ruby 2.2.2. This is a TEENY version
       release of the stable 2.2 series.

       This release includes the security fix for a OpenSSL extension's hostname
       verification vulnerability.

           CVE-2015-1855: Ruby OpenSSL Hostname Verification

       There are also some bugfixes. See ChangeLog for details.

commit 6e88dba1add92447fc97f35ec9518d1a58ce2484
Author: tron <tron>
Date:   Sun Apr 26 08:36:34 2015 +0000

    Pullup ticket #4685 - requested by taca
    lang/ruby21-base: security update

    Revisions pulled up:
    - lang/ruby/rubyversion.mk                                      1.140
    - lang/ruby21-base/PLIST                                        1.6
    - lang/ruby21-base/distinfo                                     1.14-1.15
    - lang/ruby21-base/patches/patch-configure                      1.3
    - lang/ruby21-base/patches/patch-lib_rdoc_text.rb               deleted
    - lang/ruby21/Makefile                                          1.3

    ---
       Module Name:	pkgsrc
       Committed By:	jperkin
       Date:		Fri Apr  3 09:02:24 UTC 2015

       Modified Files:
       	pkgsrc/lang/ruby21-base: distinfo
       	pkgsrc/lang/ruby21-base/patches: patch-configure

       Log Message:
       Disable CPU detection on Darwin, the result for 32-bit (i486) is incompatible
       with pkgsrc MACHINE_ARCH (i386).  Fixes 32-bit build, no change for 64-bit.

    ---
       Module Name:	pkgsrc
       Committed By:	taca
       Date:		Sun Apr 19 16:19:00 UTC 2015

       Modified Files:
       	pkgsrc/lang/ruby: rubyversion.mk
       	pkgsrc/lang/ruby21-base: PLIST distinfo
       Removed Files:
       	pkgsrc/lang/ruby21-base/patches: patch-lib_rdoc_text.rb

       Log Message:
       Update ruby21-base and ruby21 packages to 2.1.6.

       >From release announce:

       Ruby 2.1.6 has been released.

       This release includes a security fix for OpenSSL extension. Please view the
       topic below for more details.

           CVE-2015-1855: Ruby OpenSSL Hostname Verification

       And, many bug fixes are also included. See tickets and ChangeLog for details.

    ---
       Module Name:	pkgsrc
       Committed By:	taca
       Date:		Sun Apr 19 16:20:42 UTC 2015

       Modified Files:
       	pkgsrc/lang/ruby21: Makefile

       Log Message:
       Reset PKGREVISION.

commit 072ebbdcf4981a4582ea66498337b1e25e302b8c
Author: tron <tron>
Date:   Sun Apr 26 08:26:15 2015 +0000

    Pullup ticket #4684 - requested by taca
    lang/ruby200-base: security update

    Revisions pulled up:
    - lang/ruby/rubyversion.mk                                      1.139
    - lang/ruby200-base/distinfo                                    1.23 via patch

    ---
       Module Name:	pkgsrc
       Committed By:	taca
       Date:		Sun Apr 19 16:12:23 UTC 2015

       Modified Files:
       	pkgsrc/lang/ruby: rubyversion.mk
       	pkgsrc/lang/ruby200-base: distinfo

       Log Message:
       Update ruby200 package to 2.0.0p645 (Ruby 2.0.0-p645).

       >From release announce:

       We are pleased to announce the release of Ruby 2.0.0-p645.

       This release includes a security fix for OpenSSL extension. Please view the
       topic below for more details.

           CVE-2015-1855: Ruby OpenSSL Hostname Verification

       Ruby 2.0.0 is now under the state of the security maintenance phase, until
       Feb. 24th, 2016. After the date, maintenance of Ruby 2.0.0 will be ended. We
       recommend you start planning migration to newer versions of Ruby, such as 2.1
       or 2.2.

       This release includes the security fix mentioned above along with small
       changes required for test environment (that shouldn't affect normal users).

       See ChangeLog for full details.

commit 58cd83802d5111831a785666eefc272c1c43051f
Author: bsiegert <bsiegert>
Date:   Sat Apr 25 20:08:09 2015 +0000

    Pullup ticket 4670

commit 259293bfa47febf9a2b4d4d71c94c3c467f0397f
Author: bsiegert <bsiegert>
Date:   Sat Apr 25 20:08:04 2015 +0000

    Pullup ticket #4670 - requested by joerg
    devel/rsltc: build fix

    Revisions pulled up:
    - devel/rsltc/Makefile                                          1.16

    ---
       Module Name:	pkgsrc
       Committed By:	joerg
       Date:		Sat Apr 18 20:40:46 UTC 2015

       Modified Files:
       	pkgsrc/devel/rsltc: Makefile

       Log Message:
       Let clang ignore the mixed up return use of the K&R code.

commit 5342fbe8148fa3cc6051f8087e07c96b646a573c
Author: bsiegert <bsiegert>
Date:   Sat Apr 25 19:28:44 2015 +0000

    Pullup ticket #4682 - requested by hiramatsu
    inputmethod/librime: build fix

    Revisions pulled up:
    - inputmethod/librime/distinfo                                  1.3
    - inputmethod/librime/patches/patch-src_dict_table.cc           1.2

    ---
       Module Name:	pkgsrc
       Committed By:	hiramatsu
       Date:		Tue Apr 21 13:59:31 UTC 2015

       Modified Files:
       	pkgsrc/inputmethod/librime: distinfo
       	pkgsrc/inputmethod/librime/patches: patch-src_dict_table.cc

       Log Message:
       Fix build break with gcc.

commit 924e69ca4577415c75e6b584de574ca82b51dcc1
Author: hiramatsu <hiramatsu>
Date:   Sat Apr 25 04:06:02 2015 +0000

    pullup #4683

commit ca4f2559fbd0e9f02fa251517d3b9d92ff68d961
Author: hiramatsu <hiramatsu>
Date:   Sat Apr 25 04:04:43 2015 +0000

    Pullup ticket #4683 - requested by joerg
    editors/p5-Wx-Scintilla: link fix patch

    Revisions pulled up:
    - editors/p5-Wx-Scintilla/distinfo                              1.3
    - editors/p5-Wx-Scintilla/patches/patch-aa                      1.3

    ---
       Module Name: pkgsrc
       Committed By:        joerg
       Date:                Fri Apr 17 15:22:44 UTC 2015

       Modified Files:
            pkgsrc/editors/p5-Wx-Scintilla: distinfo
            pkgsrc/editors/p5-Wx-Scintilla/patches: patch-aa

       Log Message:
       Don't depend on ${PREFIX}/lib to be added implicitly by the wrappers.
       Don't use empty arguments to -o.

commit 26b7efffbf80de9703df2d47f9fcd08a5374ec2c
Author: hiramatsu <hiramatsu>
Date:   Fri Apr 24 16:18:39 2015 +0000

    pullup #4654.

commit dc5bb12c74c6a65858d49e497554493bd3f63564
Author: hiramatsu <hiramatsu>
Date:   Fri Apr 24 16:07:10 2015 +0000

    Pullup ticket #4654 - requested by dsainty
    security/zoneminder: security update

    Revisions pulled up:
    - security/zoneminder/Makefile                                  1.18
    - security/zoneminder/distinfo                                  1.7
    - security/zoneminder/MESSAGE                                   1.3
    - security/zoneminder/PLIST                                     1.4
    - security/zoneminder/patches/patch-Makefile_am                 1.3
    - security/zoneminder/patches/patch-configure_ac                1.4
    - security/zoneminder/patches/patch-scripts_ZoneMinder_lib_ZoneMinder_General_pm 1.2
    - security/zoneminder/patches/patch-scripts_zm_in               1.3
    - security/zoneminder/patches/patch-src_Makefile_am             1.3
    - security/zoneminder/patches/patch-src_zm__thread.h            1.4
    - security/zoneminder/patches/patch-src_zm__timer.h             1.3
    - security/zoneminder/patches/patch-src_zm__utils.h             1.2
    - security/zoneminder/patches/patch-src_zm_ffmpeg_camera_cpp    1.1
    - security/zoneminder/patches/patch-src_zm_remote_camera_h      1.4
    - security/zoneminder/patches/patch-src_zm_signal_cpp           1.1
    - security/zoneminder/patches/patch-src_zm_signal_h             deleted
    - security/zoneminder/patches/patch-src_zmf_cpp                 deleted

    ---
       Module Name:    pkgsrc
       Committed By:   dsainty
       Date:           Sun Apr  5 08:51:08 UTC 2015

       Modified Files:
               pkgsrc/security/zoneminder: MESSAGE Makefile PLIST distinfo
               pkgsrc/security/zoneminder/patches: patch-Makefile_am
                   patch-configure_ac
                   patch-scripts_ZoneMinder_lib_ZoneMinder_General_pm
                   patch-scripts_zm_in patch-src_Makefile_am patch-src_zm__thread.h
                   patch-src_zm__timer.h patch-src_zm__utils.h
                   patch-src_zm_remote_camera_h
       Removed Files:
               pkgsrc/security/zoneminder/patches: patch-src_zm_signal_h
                   patch-src_zmf_cpp

       Log Message:
       Update ZoneMinder from 1.25.0 to 1.28.1.

       Numerous changes, documented at:
        https://github.com/ZoneMinder/ZoneMinder/releases

       Addresses two security advisories:

         https://github.com/ZoneMinder/ZoneMinder/releases/tag/v1.28.0
         http://secunia.com/advisories/62918/

       Pkgsrc changes:

       patch-src_zm_signal_h is no longer necessary because zm_signal.h uses
       HAVE_EXECINFO_H.

       patch-src_zmf_cpp appears to be applied upstream.

       patch-configure_ac no longer needs to set PATH_BUILD to
       PREFIX/share/zoneminder, so that zmupdate.pl can locate the database build
       scripts as installed files.  Upstream has now implemented this via the
       ZM_PATH_DATA entry in zm.conf, and adds a ZM_PATH_DATA/db subdirectory.

       src/Makefile.am no longer setuid's zmfix, as zmfix was removed from
       ZoneMinder 1.26.6.

       The code now uses clock_gettime(), which on some systems (like Linux), calls
       for -lrt.  Since the build system isn't aware of this, but Pkgsrc is, just set
       PTHREAD_AUTO_VARS=yes.

       The PHP code now uses PDO for DB access, but it looks like there are some
       straggling dependencies on the raw MySQL driver, so both are pulled in.

    ---
       Module Name:    pkgsrc
       Committed By:   dsainty
       Date:           Tue Apr  7 12:32:59 UTC 2015

       Modified Files:
               pkgsrc/security/zoneminder: Makefile

       Log Message:
       NetBSD's tar (6.1) does an exit(1) on the source archive, complaining:

        Invalid header, starting valid header search.

       As a workaround: EXTRACT_USING=gtar

    ---
       Module Name:    pkgsrc
       Committed By:   dsainty
       Date:           Tue Apr  7 12:54:36 UTC 2015

       Modified Files:
               pkgsrc/security/zoneminder: distinfo
               pkgsrc/security/zoneminder/patches: patch-configure_ac

       Log Message:
       Recognise lower-case "netbsd" as BSD.

       NB: This doesn't fix the threads portability issues yet for NetBSD, but gets
       through configuration.

    ---
       Module Name:    pkgsrc
       Committed By:   dsainty
       Date:           Fri Apr 10 02:53:20 UTC 2015

       Modified Files:
               pkgsrc/security/zoneminder: Makefile

       Log Message:
       New ZoneMinder supports the newer ffmpeg APIs, so switch the ffmpeg
       dependency from ffmpeg010 to ffmpeg2.

    ---
       Module Name:    pkgsrc
       Committed By:   dsainty
       Date:           Fri Apr 10 02:58:49 UTC 2015

       Modified Files:
               pkgsrc/security/zoneminder: Makefile distinfo
               pkgsrc/security/zoneminder/patches: patch-src_zm__thread.h
                   patch-src_zm__timer.h
       Added Files:
               pkgsrc/security/zoneminder/patches: patch-src_zm_ffmpeg_camera_cpp
                   patch-src_zm_signal_cpp

       Log Message:
       Patch up some Linux-specific assumptions in the code.

       Fixes build under NetBSD.

       Bump PKGREVISION for switch to ffmpeg2, and some portability changes that
       aren't expected to affect functionality.

    ---
       Module Name:    pkgsrc
       Committed By:   dsainty
       Date:           Wed Apr 22 15:13:44 UTC 2015

       Modified Files:
               pkgsrc/security/zoneminder: Makefile

       Log Message:
       Fix paths in two missed installed scripts.

       The intention of zmsystemctl.pl is to use bin/pkexec to allow the apache user
       to start and stop the ZoneMinder services on operating systems using systemd
       and newer versions of Polkit than Pkgsrc currently has.

       If the base OS doesn't use systemd (E.g. anything not Linux), this file
       shouldn't be used anyway.

       In Pkgsrc we ignore the potentially absent pkexec interpreter in this file.
       If the base OS uses systemd, it probably also has pkexec in its base
       installation.

       Bump PKGREVISION.

commit 5888b51a0b3b68add698d985f00528ba171625b4
Merge: 0d3b50c 1cbc272
Author: Jonathan Perkin <jperkin@joyent.com>
Date:   Thu Apr 23 12:34:40 2015 +0100

    Merge branch 'joyent/feature/miscfix/2015Q1' into joyent/release/2015Q1

commit 1cbc272373cd255cd6a62f723a2d6e3efd36e50e
Merge: ccca9bf cc4bcc6
Author: Jonathan Perkin <jperkin@joyent.com>
Date:   Thu Apr 23 12:33:54 2015 +0100

    Merge branch 'pkgsrc_2015Q1' into joyent/feature/miscfix/2015Q1

commit 0d3b50ce89ba6306650de22989cd367056fea258
Merge: a5f8d21 ccca9bf
Author: Jonathan Perkin <jperkin@joyent.com>
Date:   Thu Apr 23 10:40:08 2015 +0100

    Merge branch 'joyent/feature/miscfix/2015Q1' into joyent/release/2015Q1

commit ccca9bf3edab64da44b0dc2b92675a00155d6fed
Author: Jonathan Perkin <jperkin@joyent.com>
Date:   Thu Apr 23 10:38:13 2015 +0100

    www/curl: backport 7.42.0 update.

commit 9e10d96a16dc43313e438319d39b61658df06820
Author: Jonathan Perkin <jperkin@joyent.com>
Date:   Thu Apr 23 10:36:53 2015 +0100

    Backport changes to disable epoll/inotify.

commit cc4bcc659cd1f7d2b5f05c11d767b03ed87f07f4
Author: tron <tron>
Date:   Wed Apr 22 22:55:10 2015 +0000

    Pullup tickets #4680 and #4681.

commit da8721d64da4906e7d9cec770d35c0227d1e8c79
Author: tron <tron>
Date:   Wed Apr 22 22:54:34 2015 +0000

    Pullup ticket #4681 - requested by joerg
    graphics/p5-PerlMagick: bug fix patch

    Revisions pulled up:
    - graphics/p5-PerlMagick/Makefile                               1.110

    ---
       Module Name:	pkgsrc
       Committed By:	joerg
       Date:		Tue Apr 21 17:21:18 UTC 2015

       Modified Files:
       	pkgsrc/graphics/p5-PerlMagick: Makefile

       Log Message:
       Don't use the shipped Magick.pm, it has the wrong bootstrap version.

commit c9a8d86c67b8dbd0b626c588d5210245c521a90f
Author: tron <tron>
Date:   Wed Apr 22 22:43:54 2015 +0000

    Pullup ticket #4680 - requested by taca
    lang/php56: security update

    Revisions pulled up:
    - lang/php/phpversion.mk                                        1.96
    - lang/php56/distinfo                                           1.9

    ---
       Module Name:	pkgsrc
       Committed By:	taca
       Date:		Fri Apr 17 16:42:43 UTC 2015

       Modified Files:
       	pkgsrc/lang/php: phpversion.mk
       	pkgsrc/lang/php56: distinfo

       Log Message:
       Update php56 to 5.6.8.

       16 Apr 2015, PHP 5.6.8

       - Core:
         . Fixed bug #66609 (php crashes with __get() and ++ operator in some cases).
           (Dmitry, Laruence)
         . Fixed bug #68021 (get_browser() browser_name_regex returns non-utf-8
           characters). (Tjerk)
         . Fixed bug #68917 (parse_url fails on some partial urls). (Wei Dai)
         . Fixed bug #69134 (Per Directory Values overrides PHP_INI_SYSTEM
           configuration options). (Anatol Belski)
         . Additional fix for bug #69152 (Type confusion vulnerability in
           exception::getTraceAsString). (Stas)
         . Fixed bug #69210 (serialize function return corrupted data when sleep has
           non-string values). (Juan Basso)
         . Fixed bug #69212 (Leaking VIA_HANDLER func when exception thrown in
           __call/... arg passing). (Nikita)
         . Fixed bug #69221 (Segmentation fault when using a generator in combination
           with an Iterator). (Nikita)
         . Fixed bug #69337 (php_stream_url_wrap_http_ex() type-confusion
           vulnerability). (Stas)
         . Fixed bug #69353 (Missing null byte checks for paths in various PHP
           extensions). (Stas)

       - Apache2handler:
         . Fixed bug #69218 (potential remote code execution with apache 2.4
           apache2handler). (Gerrit Venema)

       - cURL:
         . Implemented FR#69278 (HTTP2 support). (Masaki Kagaya)
         . Fixed bug #68739 (Missing break / control flow). (Laruence)
         . Fixed bug #69316 (Use-after-free in php_curl related to
           CURLOPT_FILE/_INFILE/_WRITEHEADER). (Laruence)

       - Date:
         . Fixed bug #69336 (Issues with "last day of <monthname>"). (Derick Rethans)

       - Enchant:
         . Fixed bug #65406 (Enchant broker plugins are in the wrong place in windows
           builds). (Anatol)

       - Ereg:
         . Fixed bug #68740 (NULL Pointer Dereference). (Laruence)

       - Fileinfo:
         . Fixed bug #68819 (Fileinfo on specific file causes spurious OOM and/or
           segfault). (Anatol Belski)

       - Filter:
         . Fixed bug #69202: (FILTER_FLAG_STRIP_BACKTICK ignored unless other
           flags are used). (Jeff Welch)
         . Fixed bug #69203 (FILTER_FLAG_STRIP_HIGH doesn't strip ASCII 127). (Jeff
           Welch)

       - OPCache:
         . Fixed bug #69297 (function_exists strange behavior with OPCache on
           disabled function). (Laruence)
         . Fixed bug #69281 (opcache_is_script_cached no longer works). (danack)
         . Fixed bug #68677 (Use After Free). (CVE-2015-1351) (Laruence)

       - OpenSSL
         . Fixed bugs #68853, #65137 (Buffered crypto stream data breaks IO polling
           in stream_select() contexts) (Chris Wright)
         . Fixed bug #69197 (openssl_pkcs7_sign handles default value incorrectly)
           (Daniel Lowrey)
         . Fixed bug #69215 (Crypto servers should send client CA list)
           (Daniel Lowrey)
         . Add a check for RAND_egd to allow compiling against LibreSSL (Leigh)

       - Phar:
         . Fixed bug #64343 (PharData::extractTo fails for tarball created by BSD tar).
           (Mike)
         . Fixed bug #64931 (phar_add_file is too restrictive on filename). (Mike)
         . Fixed bug #65467 (Call to undefined method cli_arg_typ_string). (Mike)
         . Fixed bug #67761 (Phar::mapPhar fails for Phars inside a path containing
           ".tar"). (Mike)
         . Fixed bug #69324 (Buffer Over-read in unserialize when parsing Phar). (Stas)
         . Fixed bug #69441 (Buffer Overflow when parsing tar/zip/phar in
           phar_set_inode). (Stas)

       - Postgres:
         . Fixed bug #68741 (Null pointer dereference). (CVE-2015-1352) (Laruence)

       - SPL:
         . Fixed bug #69227 (Use after free in zval_scan caused by
            spl_object_storage_get_gc). (adam dot scarr at 99designs dot com)

       - SOAP:
         . Fixed bug #69293 (NEW segfault when using SoapClient::__setSoapHeader
            (bisected, regression)). (Laruence)

       - Sqlite3:
         . Fixed bug #68760 (SQLITE segfaults if custom collator throws an exception).
            (Dan Ackroyd)
         . Fixed bug #69287 (Upgrade bundled libsqlite to 3.8.8.3). (Anatol)
         . Fixed bug #66550 (SQLite prepared statement use-after-free). (Sean Heelan)

commit fcea77f015634d2ceccf87aa8cb9d4d6f37c4fb8
Author: tron <tron>
Date:   Tue Apr 21 21:55:20 2015 +0000

    Pullup tickets #4676, #4677, #4678 and #4679.

commit 925ab7098bb6cc3ce910092b754e612ad6aad606
Author: tron <tron>
Date:   Tue Apr 21 21:53:52 2015 +0000

    Pullup ticket #4679 - requested by taca
    lang/php55: security update

    Revisions pulled up:
    - lang/php/phpversion.mk                                        1.95
    - lang/php55/distinfo                                           1.39

    ---
       Module Name:	pkgsrc
       Committed By:	taca
       Date:		Fri Apr 17 16:40:58 UTC 2015

       Modified Files:
       	pkgsrc/lang/php: phpversion.mk
       	pkgsrc/lang/php55: distinfo

       Log Message:
       Update php55 to 5.5.24.

       16 Apr 2015, PHP 5.5.24

       - Apache2handler:
         . Fixed bug #69218 (potential remote code execution with apache 2.4
           apache2handler). (Gerrit Venema)

       - Core:
         . Fixed bug #66609 (php crashes with __get() and ++ operator in some cases).
           (Dmitry, Laruence)
         . Fixed bug #67626 (User exceptions not properly handled in streams).
           (Julian)
         . Fixed bug #68021 (get_browser() browser_name_regex returns non-utf-8
           characters). (Tjerk)
         . Fixed bug #68917 (parse_url fails on some partial urls). (Wei Dai)
         . Fixed bug #69134 (Per Directory Values overrides PHP_INI_SYSTEM
           configuration options). (Anatol Belski)
         . Additional fix for bug #69152 (Type confusion vulnerability in
           exception::getTraceAsString). (Stas)
         . Fixed bug #69212 (Leaking VIA_HANDLER func when exception thrown in
           __call/... arg passing). (Nikita)
         . Fixed bug #69221 (Segmentation fault when using a generator in combination
           with an Iterator). (Nikita)
         . Fixed bug #69337 (php_stream_url_wrap_http_ex() type-confusion
           vulnerability). (Stas)
         . Fixed bug #69353 (Missing null byte checks for paths in various PHP
           extensions). (Stas)

       - Curl:
         . Implemented FR#69278 (HTTP2 support). (Masaki Kagaya)
         . Fixed bug #69316 (Use-after-free in php_curl related to
           CURLOPT_FILE/_INFILE/_WRITEHEADER). (Laruence)

       - Date:
         . Export date_get_immutable_ce so that it can be used by extensions. (Derick
           Rethans)
         . Fixed bug #69336 (Issues with "last day of <monthname>"). (Derick Rethans)

       - Enchant:
         . Fixed bug #65406 (Enchant broker plugins are in the wrong place in windows
           builds). (Anatol)

       - Fileinfo:
         . Fixed bug #68819 (Fileinfo on specific file causes spurious OOM and/or
           segfault). (Anatol Belski)

       - Filter:
         . Fixed bug #69202 (FILTER_FLAG_STRIP_BACKTICK ignored unless other
           flags are used). (Jeff Welch)
         . Fixed bug #69203 (FILTER_FLAG_STRIP_HIGH doesn't strip ASCII 127). (Jeff
           Welch)

       - Mbstring:
         . Fixed bug #68846 (False detection of CJK Unified Ideographs Extension E).
           (Masaki Kagaya)

       - OPCache
          . Fixed bug #68677 (Use After Free). (CVE-2015-1351) (Laruence)
          . Fixed bug #69281 (opcache_is_script_cached no longer works). (danack)

       - OpenSSL:
         . Fixed bug #67403 (Add signatureType to openssl_x509_parse).
         . Add a check for RAND_egd to allow compiling against LibreSSL (Leigh)

       - Phar:
         . Fixed bug #64343 (PharData::extractTo fails for tarball created by BSD tar).
           (Mike)
         . Fixed bug #64931 (phar_add_file is too restrictive on filename). (Mike)
         . Fixed bug #65467 (Call to undefined method cli_arg_typ_string). (Mike)
         . Fixed bug #67761 (Phar::mapPhar fails for Phars inside a path containing
           ".tar"). (Mike)
         . Fixed bug #69324 (Buffer Over-read in unserialize when parsing Phar). (Stas)
         . Fixed bug #69441 (Buffer Overflow when parsing tar/zip/phar in
           phar_set_inode). (Stas)

       - Postgres:
         . Fixed bug #68741 (Null pointer dereference). (CVE-2015-1352) (Laruence)

       - SPL:
         . Fixed bug #69227 (Use after free in zval_scan caused by
            spl_object_storage_get_gc). (adam dot scarr at 99designs dot com)

       - SOAP:
         . Fixed bug #69293 (NEW segfault when using SoapClient::__setSoapHeader
            (bisected, regression)). (thomas at shadowweb dot org, Laruence)

       - SQLITE:
         . Fixed bug #68760 (SQLITE segfaults if custom collator throws an exception).
            (Dan Ackroyd)
         . Fixed bug #69287 (Upgrade bundled sqlite to 3.8.8.3). (Anatol)

commit 06cfdb83dfa6add7a1890739e737320f7ca3044b
Author: tron <tron>
Date:   Tue Apr 21 21:50:09 2015 +0000

    Pullup ticket #4677 - requested by taca
    lang/php54: security update

    Revisions pulled up:
    - lang/php/phpversion.mk                                        1.94
    - lang/php54/distinfo                                           1.56

    ---
       Module Name:	pkgsrc
       Committed By:	taca
       Date:		Fri Apr 17 16:39:56 UTC 2015

       Modified Files:
       	pkgsrc/lang/php: phpversion.mk
       	pkgsrc/lang/php54: distinfo

       Log Message:
       Update php54 to 5.4.40.

       16 Apr 2015 PHP 5.4.40

       - Apache2handler:
         . Fixed bug #69218 (potential remote code execution with apache 2.4
           apache2handler). (Gerrit Venema)

       - Core:
         . Additional fix for bug #69152 (Type confusion vulnerability in
           exception::getTraceAsString). (Stas)
         . Fixed bug #69337 (php_stream_url_wrap_http_ex() type-confusion
           vulnerability). (Stas)
         . Fixed bug #69353 (Missing null byte checks for paths in various PHP
           extensions). (Stas)

       - cURL:
         . Fixed bug #69316 (Use-after-free in php_curl related to
           CURLOPT_FILE/_INFILE/_WRITEHEADER). (Laruence)

       - Ereg:
         . Fixed bug #68740 (NULL Pointer Dereference). (Laruence)

       - Fileinfo:
         . Fixed bug #68819 (Fileinfo on specific file causes spurious OOM and/or
           segfault). (Anatol Belski)

       - GD:
         . Fixed bug #68601 (buffer read overflow in gd_gif_in.c). (Remi)

       - Phar:
         . Fixed bug #68901 (use after free). (bugreports at internot dot info)
         . Fixed bug #69324 (Buffer Over-read in unserialize when parsing Phar). (Stas)
         . Fixed bug #69441 (Buffer Overflow when parsing tar/zip/phar in
           phar_set_inode). (Stas)

       - Postgres:
         . Fixed bug #68741 (Null pointer deference) (CVE-2015-1352). (Xinchen Hui)

       - SOAP:
         . Fixed bug #69152 (Type Confusion Infoleak Vulnerability in unserialize()
           with SoapFault). (Dmitry)

       - Sqlite3:
         . Fixed bug #66550 (SQLite prepared statement use-after-free). (Sean Heelan)

commit 259235aeb1391a7d418c7f05567d8fd849ecbf64
Author: tron <tron>
Date:   Tue Apr 21 21:44:22 2015 +0000

    Pullup ticket #4678 - requested by taca
    net/ntp4: security update

    Revisions pulled up:
    - net/ntp4/Makefile                                             1.85
    - net/ntp4/PLIST                                                1.18
    - net/ntp4/distinfo                                             1.21

    ---
       Module Name:	pkgsrc
       Committed By:	taca
       Date:		Wed Apr  8 03:31:34 UTC 2015

       Modified Files:
       	pkgsrc/net/ntp4: Makefile PLIST distinfo

       Log Message:
       Update ntp4 package to 4.2.8p2.

       NTP 4.2.8p2 (Harlan Stenn <stenn@ntp.org>, 2015/04/xx)

       Focus: Security and Bug fixes, enhancements.

       Severity: MEDIUM

       In addition to bug fixes and enhancements, this release fixes the
       following medium-severity vulnerabilities involving private key
       authentication:

       * [Sec 2779] ntpd accepts unauthenticated packets with symmetric key crypto.

           References: Sec 2779 / CVE-2015-1798 / VU#374268
           Affects: All NTP4 releases starting with ntp-4.2.5p99 up to but not
       	including ntp-4.2.8p2 where the installation uses symmetric keys
       	to authenticate remote associations.
           CVSS: (AV:A/AC:M/Au:N/C:P/I:P/A:P) Base Score: 5.4
           Date Resolved: Stable (4.2.8p2) 07 Apr 2015
           Summary: When ntpd is configured to use a symmetric key to authenticate
       	a remote NTP server/peer, it checks if the NTP message
       	authentication code (MAC) in received packets is valid, but not if
       	there actually is any MAC included. Packets without a MAC are
       	accepted as if they had a valid MAC. This allows a MITM attacker to
       	send false packets that are accepted by the client/peer without
       	having to know the symmetric key. The attacker needs to know the
       	transmit timestamp of the client to match it in the forged reply
       	and the false reply needs to reach the client before the genuine
       	reply from the server. The attacker doesn't necessarily need to be
       	relaying the packets between the client and the server.

       	Authentication using autokey doesn't have this problem as there is
       	a check that requires the key ID to be larger than NTP_MAXKEY,
       	which fails for packets without a MAC.
           Mitigation:
               Upgrade to 4.2.8p2, or later, from the NTP Project Download Page
       	or the NTP Public Services Project Download Page
               Configure ntpd with enough time sources and monitor it properly.
           Credit: This issue was discovered by Miroslav Lichvar, of Red Hat.

       * [Sec 2781] Authentication doesn't protect symmetric associations against
         DoS attacks.

           References: Sec 2781 / CVE-2015-1799 / VU#374268
           Affects: All NTP releases starting with at least xntp3.3wy up to but
       	not including ntp-4.2.8p2 where the installation uses symmetric
       	key authentication.
           CVSS: (AV:A/AC:M/Au:N/C:P/I:P/A:P) Base Score: 5.4
           Note: the CVSS base Score for this issue could be 4.3 or lower, and
       	it could be higher than 5.4.
           Date Resolved: Stable (4.2.8p2) 07 Apr 2015
           Summary: An attacker knowing that NTP hosts A and B are peering with
       	each other (symmetric association) can send a packet to host A
       	with source address of B which will set the NTP state variables
       	on A to the values sent by the attacker. Host A will then send
       	on its next poll to B a packet with originate timestamp that
       	doesn't match the transmit timestamp of B and the packet will
       	be dropped. If the attacker does this periodically for both
       	hosts, they won't be able to synchronize to each other. This is
       	a known denial-of-service attack, described at
       	https://www.eecis.udel.edu/~mills/onwire.html .

       	According to the document the NTP authentication is supposed to
       	protect symmetric associations against this attack, but that
       	doesn't seem to be the case. The state variables are updated even
       	when authentication fails and the peers are sending packets with
       	originate timestamps that don't match the transmit timestamps on
       	the receiving side.

       	This seems to be a very old problem, dating back to at least
       	xntp3.3wy. It's also in the NTPv3 (RFC 1305) and NTPv4 (RFC 5905)
       	specifications, so other NTP implementations with support for
       	symmetric associations and authentication may be vulnerable too.
       	An update to the NTP RFC to correct this error is in-process.
           Mitigation:
               Upgrade to 4.2.8p2, or later, from the NTP Project Download Page
       	or the NTP Public Services Project Download Page
               Note that for users of autokey, this specific style of MITM attack
       	is simply a long-known potential problem.
               Configure ntpd with appropriate time sources and monitor ntpd.
       	Alert your staff if problems are detected.
           Credit: This issue was discovered by Miroslav Lichvar, of Red Hat.

       * New script: update-leap
       The update-leap script will verify and if necessary, update the
       leap-second definition file.
       It requires the following commands in order to work:

       	wget logger tr sed shasum

       Some may choose to run this from cron.  It needs more portability testing.

commit 94c1b5104684448d27261c42d658ac54910af9ac
Author: tron <tron>
Date:   Tue Apr 21 21:39:41 2015 +0000

    Pullup ticket #4676 - requested by tron
    net/tigervnc: build fix

    Revisions pulled up:
    - net/tigervnc/Makefile                                         1.4

    ---
       Module Name:	pkgsrc
       Committed By:	joerg
       Date:		Sat Apr 18 20:44:47 UTC 2015

       Modified Files:
       	pkgsrc/net/tigervnc: Makefile

       Log Message:
       Needs explicit libdrm dependency.

commit 9ac98dab09e6d06bb842b68ed5f6011621f1b13f
Author: tron <tron>
Date:   Tue Apr 21 19:07:46 2015 +0000

    Pullup tickets #4669, #4674 and #4675.

commit f4dd44867374ea54f7a364d6262a851839ca6aec
Author: tron <tron>
Date:   Tue Apr 21 19:07:14 2015 +0000

    Pullup ticket #4675 - requested by joerg
    games/ioquake3: build fix

    Revisions pulled up:
    - games/ioquake3/Makefile                                       1.4
    - games/ioquake3/PLIST                                          1.3
    - games/ioquake3/distinfo                                       1.5
    - games/ioquake3/patches/patch-aa                               1.3

    ---
       Module Name:	pkgsrc
       Committed By:	joerg
       Date:		Sat Apr 18 20:44:08 UTC 2015

       Modified Files:
       	pkgsrc/games/ioquake3: Makefile PLIST distinfo
       	pkgsrc/games/ioquake3/patches: patch-aa

       Log Message:
       Allow building on NetBSD/arm.

commit d3ef2ddfc6c5d191f3a29bb4af90aab71356dffa
Author: tron <tron>
Date:   Tue Apr 21 19:00:54 2015 +0000

    Pullup ticket #4674 - requested by joerg
    games/orbital_eunuchs_sniper: packaging fix

    Revisions pulled up:
    - games/orbital_eunuchs_sniper/Makefile                         1.23

    ---
       Module Name:	pkgsrc
       Committed By:	joerg
       Date:		Sat Apr 18 20:43:19 UTC 2015

       Modified Files:
       	pkgsrc/games/orbital_eunuchs_sniper: Makefile

       Log Message:
       Use uname -m output for NetBSD/ARM in the PLIST as the package does.

commit c3e33c1c17b50b1641b75315f564d4f914c8d49e
Author: tron <tron>
Date:   Tue Apr 21 18:55:45 2015 +0000

    Pullup ticket #4669 - requested by manu
    databases/php-ldap: build fix

    Revisions pulled up:
    - databases/php-ldap/files/ldap-ctrl-exop.patch                 deleted
    - databases/php-ldap/files/ldap-ctrl-exop54.patch               1.1-1.2
    - databases/php-ldap/files/ldap-ctrl-exop55.patch               1.1-1.2
    - databases/php-ldap/files/ldap-ctrl-exop56.patch               1.1-1.2
    - databases/php-ldap/options.mk                                 1.4

    ---
       Module Name:    pkgsrc
       Committed By:   manu
       Date:           Mon Apr 13 08:04:17 UTC 2015

       Modified Files:
               pkgsrc/databases/php-ldap: options.mk
       Added Files:
               pkgsrc/databases/php-ldap/files: ldap-ctrl-exop54.patch
                   ldap-ctrl-exop55.patch ldap-ctrl-exop56.patch
       Removed Files:
               pkgsrc/databases/php-ldap/files: ldap-ctrl-exop.patch

       Log Message:
       Update the LDAP EXOP support patch for PHP 5.4, 5.5 and 5.6

    ---
       Module Name:    pkgsrc
       Committed By:   manu
       Date:           Mon Apr 20 03:24:36 UTC 2015

       Modified Files:
               pkgsrc/databases/php-ldap/files: ldap-ctrl-exop54.patch
                   ldap-ctrl-exop55.patch ldap-ctrl-exop56.patch

       Log Message:
       Fix build with SASL disabled

       Do not force-define HAVE_LDAP_SASL, since the PHP configure script
       actually takes care of that one. This lets us build with the exop
       option enabled while sasl is disabled.

commit a5f8d21945363b03bec896fb6af2714201ab919e
Merge: 7d962d8 faf0e76
Author: Jonathan Perkin <jperkin@joyent.com>
Date:   Tue Apr 21 14:40:27 2015 +0100

    Merge branch 'joyent/feature/miscfix/2015Q1' into joyent/release/2015Q1

commit faf0e763cdbe20b49f8215cb58530dff6e44d97f
Merge: 86554d2 a295034
Author: Jonathan Perkin <jperkin@joyent.com>
Date:   Tue Apr 21 14:40:13 2015 +0100

    Merge branch 'joyent/feature/miscfix/2015Q1' of github.com:joyent/pkgsrc into joyent/feature/miscfix/2015Q1

commit 7d962d8f53c204790d4d7120d6e6dbfe107a7af6
Author: Jonathan Perkin <jperkin@joyent.com>
Date:   Tue Apr 21 14:39:38 2015 +0100

    Update 'joyent' and 'wip' submodules.

commit b163d85dc024e7f042e22cfb6f1fc3cc5279114b
Merge: 952134b 86554d2
Author: Jonathan Perkin <jperkin@joyent.com>
Date:   Tue Apr 21 14:39:13 2015 +0100

    Merge branch 'joyent/feature/miscfix/2015Q1' into joyent/release/2015Q1

commit 86554d2c5a721a99fd2e7c4b5862b35b69918090
Merge: 0b4c611 6996bfa
Author: Jonathan Perkin <jperkin@joyent.com>
Date:   Tue Apr 21 14:38:23 2015 +0100

    Merge branch 'pkgsrc_2015Q1' into joyent/feature/miscfix/2015Q1

commit a295034d9249a35784c34cc387513f5de3c73570
Author: Filip Hajny <filip@joyent.com>
Date:   Mon Apr 20 17:09:03 2015 +0000

    Update sysutils/cfengine3 to 3.6.5. Add lmdb support option.

    Upstream changelog follows.

    3.6.5

    Features:
    - Introduced "systemd" hard class.
    - Added paths to dtrace, zfs and zpool on FreeBSD in masterfiles.

    Bug fixes:
    - Fixed build error on certain RHEL5 and SLES10 setups.
    - Fixed a bug which caused dangling symlinks not to be removed.
    - Fixed data_readstringarrayidx function not preserving the order of the
      array it's producing.
    - Fixed a bug which sometimes caused CFEngine to kill the wrong daemon
      if both the host and a container inside the host were running
      CFEngine.
    - Made sure the rm_rf_depth bundle also deletes the base directory.
    - Fixed monitord reporting wrongly on open ports.
    - Skip adding the class when its name is longer than 1024 characters.
      Fixed core dump when the name is too large.
    - Fixed detection of stopped process on Solaris.
    - Fixed infinite loop plus a couple more minor
      bugs in edit_xml promises.

    3.6.4

    Features:
    - Introduced users promises support on HP-UX platform.
    - Introduced process promises support on HP-UX platform.

    Bug fixes:
    - Fixed bug on FreeBSD which sometimes led to the wrong process being killed
    - Fixed package version comparison sometimes failing with rpm package manager
    - Fixed a bug in users promises which would sometimes set the wrong
      password hash if the user would also be unlocked at the same time.
    - Fixed a bug on AIX which would occationally kill the wrong process.
    - Improved error message for functions that require an absolute path.
    - Fixed some spelling errors in examples.
    - Fixed error in out-of-tree builds when building cf-upgrade.
    - Fixed a bug which would make cf-agent exit with an error if it was
      built with a custom log directory, and that directory did not exist.
    - Fixed ordering of evaluating promises when depends_on is used.
    - Skip non-empty directories silently when recursively deleting.
    - Fix memory exhaustion with list larger than 4994 items.
    - Fix cf-execd segfault on IP address detection
    - Fix hard class detection of RHEL6 ComputeNode

    3.6.3

    New features:
    - support for HP-UX 11.23 and later
    - experimental support for Red Hat Enterprise Linux 7

    Bug fixes:
    - fix getindices on multi-dimensional arrays
    - fix mustache template method to run in dryrun mode
    - set mailto and mailfrom settings for execd in def.cf
    - fix conflation of multi-index entries in arrays
    - fix promise locking when transferring using update.cf
    - update JSON parser to return an error on truncation
    - fix sys.hardware_addresses not expanded
    - fix opening database txn /var/cfengine/cf_lastseen.lmdb:
      MDB_READERS_FULL when running cf-keys --show-hosts
    - fix segfault (Null pointer dereference) when select_end in
      delete_lines never matches
    - fix max_file_size => "0" not disabling or allowing any size
     - fix ifvarclass, with iteration over list, failing when deleting
      files with time condition
    - fix classes defined with "or" constraint are never set if any value
      doesn't evaluate to a scalar
    - update "mailfrom" default in default policy
    - fix logrotate ambiguity of filename
    - fix parsing JSON files
    - reduce write count activity to /var partition
    - fix files delete attribute incorrectly triggering promise_kept
    - update services bundle output related to chkconfig when run in
      inform mode.
    - fix Solaris serverd tests
    - fix broken bechaviour of merging arrays with readstringarray
    - fix ifelapsed bug with bundle nesting
    - fix handling cf_null in bundlesequence
    - fix maparray reading whole input array when using subarray
     - fix directories being randomly changed to files
    - update defaults promise type to work with classes
    - systemd integration in services promises
    - fix touch attribute ignoring action = warn_only
    - fix 4KB string limit in functions readfile, string_downcase,
      string_head, string_reverse, string_length, string_tail,
      string_upcase

    3.6.2

    Bug fixes:
    - don't regenerate software_packages.csv every time
    - improve verbose message for package_list_command
    - fix missing log output on AIX
    - assorted fixes to dirname() esp on Windows
    - fix package manager detection
    - fix build issues on FreeBSD
    - allow copying of dead symbolic links
    - preserve order in readstringarrayidx
    - fix passing of unexpanded variable references to arrays
     - use entries for new {admin,deny}_{ips,hostnames} constraints in
      the relevant legacy lists
    - cope with ps's numeric fields overflowing to the right
    - interpret failing function calls in ifvarclass as class not set
     - remove unexpanded lists when extending lists
    - infer start-time of a process from elapsed when needed
     - fix input range definition for laterthan() function
    - don't add trailing delimiter when join()'ing lists ending with a
      null-value
    - 9999999999 (ten 9s) or higher has been historically used as an upper
      bound in CFEngine code and policy but because of overflow on 32-bit
      platforms it caused problems with big numbers. Fixed in two ways:
      first change all existing policy uses to 999999999 (nine 9s instead
      of eleven 9s), second fix the C code to not wrap-around in case of
      overflow, but use the LONG_MAX value
    - cf-serverd and other daemons no longer reload their configuration
      every minute if CFEngine is built with an inputs directory outside
      of the work directory (not the default).

    3.6.1

    New features:
    - Introduced Solaris and AIX support into the 3.6 series, with many associated
      build and bug fixes.

    Changes:
    - Short-circuit evaluation of classes promises if class is already set
     - fix to assume all non-specified return codes are failed in commands
      promises
    - cf-serverd logs reconfiguration message to NOTICE (was INFO) so that it's
      always logged in syslog

    Bug fixes:
    - File monitoring has been completely rewritten (changes attribute in files
      promise), which eliminates many bugs, particularly regarding files that
      are deleted. Upgrading will keep all monitoring data, but downgrading again
      will reinitialize the DB, so all files will be reported as if they were new.
     - $(this.promiser) expands in files promises for 'transformer',
      'edit_template', 'copy_from.source', 'file_select.exec_program', 'classes'
      and 'action' bodies
    - 'body changes' notifies about disappeared files in file monitoring
    - Set not-kept classes when files or commands promise should be repaired,
      but is warn-only
    - Fixed CFEngine template producing a zero-sized file
    - Add 0-9 A-Z _ to allowed context of module protocol
    - Extend ps command column width on Solaris and filter on zone
      rather than adding it to the ps output.
    - Fixed strftime() function on Solaris when called with certain specifiers.
    - Fixed users promise bug regarding password hashes in a NIS/NSS setup.
    - Fixed $(sys.uptime), $(sys.systime) and $(sys.sysday) in AIX.
      #5148, #5206)
    - Fixed processes_select complaining about "Unacceptable model uncertainty
      examining processes"
    - ps command for linux has been changed to cope with big rss values
    - Address ps -axo shift on FreeBSD 10 and later
    - methods and services promises respect action_policy => "warn"
     - LMDB should no longer deadlock if an agent is killed on the hub while
      holding the DB lock. Note that the change only affects binary packages
      shipped by CFEngine, since the upstream LMDB project has not yet integrated
      the change.

    3.6.0

    Changes:
    - Changes to logging output
        - add process name and pid in syslog message (GitHub #789)
        - cf-serverd logging levels are now more standardised:
              - INFO logs only failures
              - VERBOSE logs successful requests as well
              - DEBUG logs actual protocol traffic.
        - cf-serverd now logs the relevant client IP address on
          each message.
        - Logging contexts to local database (cf_classes.tcdb) has been deprecated.
        - 'usebundle' promisees are logged for all the bundle promises
        - output from 'reports' promises has nothing prefixed except 'R: '
        - a log line with stack path is generated when the promise type evaluated
          changes
    - LMDB (symas.com/mdb) is the default database for local data storage : use
      version 0.9.9 or later. cf-agent --self-diagnostics (-x) is only
      implemented for TCDB, not for LMDB
    - port argument in readtcp() and selectservers() may be a
      service name (e.g. "http", "pop3").
    - Enable source file in agent copy_from promises to be a relative path.
        - file "changes" reporting now reports with log level "notice", instead
          of "error".
    - process_results default to AND'ing of set attributes if not specified
     - interface is now canonified in sys.hardware_mac[interface] to align with
      sys.ipv4[interface]
    - cf-promises no longer errors on missing bodies when run without
      --full-check (-c)
    - Linux flavor "SUSE" now correctly spelled with all uppercase in variables
      and class names. The "suse" lowercase version is also
      provided for convenience.
    - $(this.promise_filename) and $(..._dirname) variables are now absolute
      paths.
    - including the same file multiple times in 'body control inputs' is
      not an error
    - portnumber in body copy_from now supports service names like "cfengine",
      "pop3" etc, check /etc/services for more.
    - The failsafe.cf policy, run on bootstrap and in some other
      unusual cases, has been extracted from C code into libpromises/failsafe.cf
    - masterfiles
        - cf_promises_validated is now in JSON format
        - timestamp key is timestamp (sec since unix epoch) of last time validated
        - the masterfiles now come from https://github.com/cfengine/masterfiles
          and are not in the core repository
    - cf-serverd calls cf-agent with -Dcfruncommand when executing cf-runagent
      requests
    Mark as removed: promise_notkept_log_include, promise_notkept_log_exclude,
    promise_repaired_log_include, promise_repaired_log_exclude, classes_include,
    classes_exclude, variables_include, variables_exclude attributes from
    report_data_select body (syntax is valid but not functional). They have been
    replaced by the following attributes: promise_handle_include,
    promise_handle_exclude, metatags_include, metatags_exclude.

    New features:
    - New promise type "users" for managing local user accounts.
    - TLS authentication and fully encrypted network protocol.
      Additions specific to the new type of connections:
        - New attribute "allowlegacyconnects" in body server control,
          which enables serving policy via non-latest cfengine protocol,
          to the given list of hosts. If the option is absent, it
          defaults to allow all hosts. To refuse non-TLS connections,
          specify an empty list.
        - New attribute "protocol_version" in body copy_from, and body
          common control, which defines the preferred protocol for
          outgoing connections.. Allowed values at the moment: "0" or
          "undefined", "classic" or "1", "latest" or "2". By leaving the
          copy_from option as undefined the common control option is
          used, and if both are undefined then classic protocol is used
          by default.
        - The new networking protocol uses TLS for authentication,
          after which all dialog is encrypted within the established
          TLS session.  cf-serverd is still able to speak the legacy
          protocol with old agents.
        - The 'skipverify' option in 'body server control' is
          deprecated and only left for compatibility; it does
          nothing
        - cf-serverd does not hang up the connection if some request
          fails, so that the client can add more requests.
        - For the connections using the new protocol, all of the
          paths in bundle server access_rules now differentiate
          between a directory and a file using the trailing
          slash. If the path exists then this is auto-detected and
          trailing slash appended automatically. You have to append
          a trailing slash manually to an inexistent or symbolic
          path (e.g. "/path/to/$(connection.ip)/") to force
          recursive access.
    - New in 'access' promises for 'bundle server access_rules'
        - Attributes "admit_ips", "admit_hostnames", "admit_keys",
          "deny_ips", "deny_hostnames", "deny_keys"
        - "admit_keys" and "deny_keys" add the new functionality
          of controlling access according to host identity,
          regardless of the connecting IP.
        - For these new attributes, regular expressions
          are not allowed, only CIDR notation for "admit/deny_ips", exact
          "SHA=..." strings for "admit/deny_keys", and exact hostnames
          (e.g. "cfengine.com") or subdomains (starting with dot,
          e.g. ".cfengine.com") for "admit/deny"_hostnames. Same rules
          apply to 'deny_*' attributes.
        - These new constraints and the paths in access_rules, can contain
          special variables "$(connection.ip)", "$(connection.hostname)",
          "$(connection.key)", which are expanded dynamically for every
          received connection.
        - For connections using the new protocol, "admit" and "deny"
          constraints in bundle server access_rules are being phased
          out, preferred attributes are now "admit_ips", "deny_ips",
          "admit_hostnames", "deny_hostnames", "admit_keys",
          "deny_keys".
        - New "shortcut" attribute in bundle server access_rules used to
          dynamically expand non-absolute request paths.
    - masterfiles
            - standard library split: lib/3.5 (compatibility) and lib/3.6
              (mainline)
            - many standard library bundles and bodies, especially packages-
              and file-related, were revised and fixed
            - supports both Community and Enterprise
            - new 'inventory/' structure to provide OS, dmidecode, LSB, etc.
              system inventory (configured mainly in def.cf)
            - cf_promises_release_id contains the policy release ID which is
              the GIT HEAD SHA if available or hash of tree
            - a bunch'o'bundles to make starting with CFEngine easier:
            - file-related: file_mustache, file_mustache_jsonstring, file_tidy,
              dir_sync, file_copy, file_link, file_hardlink, file_empty, file_make
            - packages-related: package_absent, package_present, package_latest,
              package_specific_present, package_specific_absent,
              package_specific_latest, package_specific
            - XML-related: xml_insert_tree_nopath, xml_insert_tree, xml_set_value,
              xml_set_attribute
            - VCS-related: git_init, git_add, git_checkout,
              git_checkout_new_branch, git_clean, git_stash, git_stash_and_clean,
              git_commit, git
            - process-related: process_kill
            - other: cmerge, url_ping, logrotate, prunedir
    - New command line options for agent binaries
        - New options to cf-promises
            - '--show-classes' and '--show-vars'
            - '--eval-functions' controls whether cf-promises should evaluate
              functions
        - Colorized output for agent binaries with command line option '--color'
          (auto-enabled if you set CFENGINE_COLOR=1)
    - New language features
        - New variable type 'data' for handling of structured data (ie JSON),
          including supporting functions:
            - 'data_readstringarray' - read a delimited file into a data map
            - 'data_readstringarrayidx' - read a delimited file into a data array
            - 'datastate' - create a data variable with currently set classes
              and variables
            - 'datatype' - determine the type of the top element of a container
            - 'format' - %S can be used to serialize 'data' containers into
              a string
            - 'mergedata' - merge two data containers, slists/ilists/rlists,
              or "classic" arrays into a data container
            - 'parsejson' - create a data container from a JSON string
            - 'readjson' - create a data container from a file that contains JSON
            - 'storejson' - serialize a data container into a string
            - Most functions operating on lists can also operate on data containers
            - pass a data container to a bundle with the @(container) notation
            - the module protocol accepts JSON for data containers with
              the '%' sigil
        - Tagging of classes and variables allows annotating of language
          construct with meta data; supporting functionality:
            - The module protocol in 'commands' promises has been extended
              to allow setting of tags of created variables and classes,
              and the context of created variables
            - 'getclassmetatags' - returns list of meta tags for a class
            - 'getvariablemetatags' - returns list of meta tags for a variable
        - 'body file control' has an 'inputs' attribute to include library files
          and other dependencies
        - bundlesequences can be built with bundlesmatching() based on bundle
          name and tags
    - New attributes in existing promise types and bodies
        - New option 'preserve_all_lines' for insert_type in insert_lines promises
        - Caching of expensive system functions to avoid multiple executions of
          execresult() etc, can be controlled via cache_system_functions
          attribute in body common control
        - New option 'mailsubject' in body executor control allows defining
          the subject in emails sent by CFEngine
        - Support for Mustache templates in 'files' promises; use
          'template_method' and 'template_data' attributes.  Without
          'template_data' specified, uses datastate().
    - New and improved functions
        - 'bundlesmatching' - returns list of defined bundles matching
          a regex and tags
        - 'canonifyuniquely' - converts a string into a unique, legal class name
        - 'classesmatching' - returns list of set classes matching a regex
          and tags
        - 'eval' - evaluates mathematical expressions; knows SI k, m, g
          quantifiers, e.g. "100k"
        - 'findfiles' - list files matching a search pattern; use "**" for
          recursive searches
        - 'makerule' - evaluates whether a target file needs to be rebuilt
          from sources
        - 'max', 'min' - returns maximum and minimum of the numbers in
          a container or list (sorted by a 'sort' method)
        - 'mean' - returns the mean of the numbers in a container or list
        - 'nth' - learned to look up by key in a data container holding a map
        - 'packagesmatching' - returns a filtered list of installed packages.
        - 'readfile' - learned to read system files of unknown size like those
          in /proc
        - 'sort' - can sort lexicographically, numerically (int or real), by IP,
          or by MAC
        - 'string_downcase', 'string_upcase' - returns the lower-/upper-case
          version of a string
        - 'string_head', 'string_tail' - returns the beginning/end of a string
        - 'string_length' - returns the length of a string
        - 'string_reverse' - reverses a string
        - 'string_split' - improved implementation, deprecates 'splitstring'
        - 'variablesmatching' - returns a list of variables matching a regex
          and tags
        - 'variance' - returns the variance of numbers in a list or container
    - New hard classes
        - Introduced alias 'policy_server' for context 'am_policy_hub'
          (the latter will be deprecated)
        - all the time-based classes have GMT equivalents
    - New variables
        - 'sys.bindir' - the location of the CFEngine binaries
        - 'sys.failsafe_policy_path' - the location of the failsafe policy file
        - 'sys.inputdir' - the directory where CFEngine searches for policy files
        - 'sys.key_digest' - the digest of the host's cryptographic key
        - 'sys.libdir', 'sys.local_libdir' - the location of the CFEngine libraries
        - 'sys.logdir' - the directory where the CFEngine log files are saved
        - 'sys.masterdir' - the location of masterfiles on the policy server
        - 'sys.piddir' - the directory where the daemon pid files are saved
        - 'sys.sysday' - the number of days since the beginning of the UNIX epoch
        - 'sys.systime' - the number of seconds since the beginning of the UNIX epoch
        - 'sys.update_policy_path' - the name of the update policy file
        - 'sys.uptime' - the number of minutes the host has been online
        - 'this.promise_dirname' - the name of the file in which the current promise
          is defined
        - 'this.promiser_uid' - the ID of the user running cf-agent
        - 'this.promiser_gid' - the group ID of the user running cf-agent
        - 'this.promiser_ppid' - the ID of the  parent process running cf-agent

    Deprecations:
    - 'splitstring' - deprecated by 'string_split'
    - 'track_value'
    - 'skipverify'

    Bug fixes: for a complete list of fixed bugs, see https://cfengine.com/dev
    - various fixes in evaluation and variable resolution
    - Improve performance of list iteration
    - Removed limitation of input length to internal buffer sizes
        - directories ending with "/" are not ignored
        - lsdir() always return a list now, never a scalar
    - 'abortclasses' fixed to work in common bundles and other cases
    - namespaced 'edit_line' bundles now work
    - lists are interpolated in correct order
    - cf-serverd reloads policies properly when they change
    - lots of leaks (memory and file descriptor) fixed

    3.5.3

    Changes:
    - Improved security checks of symlink ownership. A symlink created by a user
      pointing to resources owned by a different user will no longer be followed.
    - Changed the way package versions are compared in package promises.
      In previous versions the comparison was inconsistent. This has been fixed,
      but may also lead to behavior changes in certain cases.

    Bug fixes:
    - fix cf-monitord crash due to incorrect array initialization
    - fix cf-serverd stat()'ing the file tree every second
    - correctly populate sys.hardware_addresses variable
    - add support for Debian's GNU/kfreebsd to build system
    - fix possible stack corruption in guest_environments promises
    - work-around hostname trunctation in HP-UX's uname
    - fix body copy purging of empty directories
    - make discovery and loading of avahi libraries more robust
    - compile and packaging fixes for HP-UX, AIX and Solaris
    - fix fatal error in lsdir() when directory doesn't exist
    - fix epoch calculation for stime inrange calculation

    3.5.2

    Bug fixes:
    - fix delayed abortclasses checking
    - fix maplist arguments bug
    - fix segfaults in cf-pomises
    - fix build on Solaris 10/SmartOS
    - sanitize characters from /etc/issue in sys.flavor for Debian
    - Fix segfault when dealing with files or data > 4K
    - Don't truncate keys to 126 characters in getindices
    - files created via log_* actions now have mode 600
    - fix wrong log message when a promise is ignored due to 'ifvarclass'
      not matching
    - fix lifetime of persistent classes
    - fix segfault when process_select body had no process_result attribute
      Default to AND'ed expression of all specified attributes
    - include system message in output when acl promises fail
    - fix invocation of standard_services bundle and corresponding promise
      compliance

    3.5.1

    Changes:
    - file changes are logged with log level Notice, not Error
    - the CFEngine Standard Library in masterfiles/libraries is now split into
      promise-type specific policy files, and lives in a version-specific
      directory. This should have no impact on current code, but allows more
      granular include of needed stdlib elements

    Bug fixes:
    - fix recursive copying of files
    - respect classes in templates
    - fix timestamps on Windows
    - fix non-root cf-agent flooding syslog
    - fix email flood from cf-execd due to timestamps in agent output
    - Preserve security context when editing or copying local files
    - fix path for sys.crontab on redhat systems
    - prevent incorrect "insert_lines promise uses the same select_line_matching
      anchor" warning
    - Fix regression of setting VIPADDRESS to 127.0.0.1
    - Fix "changes" promise not receiving status when file is missing
    - Fix symlinks being destroyed when editing them
    - Fix missing "promise kept" status for the last line in a file

    3.5.0

    New features:
    - classes promises now take an optional scope constraint.
    - new built-in functions: every, none, some, nth, sublist, uniq, filter
      - every
      - none
      - some
      - nth
      - sublist
      - uniq
      - filter
      - classesmatching
      - strftime
      - filestat
      - ifelse
      - maparray
      - format
    - cf-promises flag --parse-tree is replaced by --policy-output-format=,
      requiring the
      user to specify the output format (none, cf, json)
    - cf-promises allows partial check of policy (without body common control)
      without integrity check; --full-check enforces integrity check
    - agent binaries support JSON input format (.json file as generated by
      cf-promises)
    - cf-key: new options --trust-key/-t and --print-digest/-p
    - Class "failsafe_fallback" is defined in failsafe.cf when main policy
      contains errors and failsafe is run because of this
    - add scope attribute for body classes
    - Better diagnostics of parsing errors
    - Error messages from parser now show the context of error
    - new cf-agent option: --self-diagnostics
    - new output format, and --legacy-output
    - warnings for cf-promises.
    - Enable zeroconf-discovery of policy hubs for automatic bootstrapping
      if Avahi is present
    - Support for sys.cpus on more platforms than Linux & HPUX

    Changes:
    - parser no longer allows ',' after promiser or promisee. must be either
      ';' or lval
    - Make parser output in GCC compatible format the only supported format
      (remove --gcc-brief-format flag)
    - Silence license warnings in Enterprise Free25 installations
    - action_policy => "warn" causes not_kept classes to be set on promise
      needing repair.
    - command line option version (-V) now prints a shorter parsable version
      without graphic
    - implicit execution of server and common bundles taking arguments is
      skipped in cf-serverd.
    - WARNING: option --policy-server removed, require option to --bootstrap
      instead
    - process promises don't log if processes are out of rang…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment