Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

msgpack(-python), document versions and issues #3753

Closed
ThomasWaldmann opened this Issue Apr 3, 2018 · 15 comments

Comments

Projects
None yet
3 participants
@ThomasWaldmann
Copy link
Member

commented Apr 3, 2018

The 0.5+ releases of msgpack(-python) were a bit of a PITA:

  • rename of the package and bad side effects of that (the fix from msgpack project was to make them also available under the old name)
  • TypeErrors due to api changes in 0.5.3 (fixed in later versions)
  • introducing that FutureWarning, which made borg output somehow unpretty / annoying (the fix from msgpack was to remove the FutureWarning again at some version)
  • lack of py34 testing introducing some uncertainty for borg, which still supports py34 (no fix)

Likely there are some working 0.5+ versions if one has py35+ or does own testing on py34, but one needs to check if there are FutureWarnings in borg output.

So the msgpack-python<0.5 is basically a "we know that the given versions work, but when using newer versions you need to do own tests".

There are currently no plans to drop py34 compatibility in borg within the 1.1.x stable release series, so using any 0.5+ version is problematic due to the lack of testing within the msgpack project.

Also, it is not planned to change the name from msgpack-python to msgpack in our requirements within the 1.1.x stable release series, because the required versions are available under the old name and we do not want to break any distribution which uses the old name still.

If borg gets packaged for distribution based on py35 or py36, the lack of py34 testing is no issue, so newer msgpack-python versions can be used, but should be tested first. You can also change the package name to msgpack when making a distribution package, but you need to test if the result actually works.

@ThomasWaldmann ThomasWaldmann changed the title msgpack-python / msgpack, document versions and issues msgpack(-python), document versions and issues Apr 3, 2018

@LocutusOfBorg

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2018

   * Add debian/python{3,}-msgpack.pydist to help transition to the new upstream
     egg name.

In Debian we are providing the old and new namespace with the same source package, so we don't care anymore about the renaming.

felixonmars-bot pushed a commit to felixonmars/archlinux-community that referenced this issue Apr 4, 2018

bpiotrowski
1.1.5-2: remove msgpack version constraint
Latest release "blacklists" msgpack 0.5.0 due to the project not supporting
py34. We are on 3.6 already and test suite passes so sounds pointless.

For reference borgbackup/borg#3753


git-svn-id: file:///srv/repos/svn-community/svn@314208 9fca08f4-af9d-4005-b8df-a31f2cc04f65
@bpereto

This comment has been minimized.

Copy link

commented Apr 10, 2018

In recent fedora Version (rawhide, f28) there is no python3-msgpack lower than 0.5.0+.

So, basically I must use 0.5.0+ but don't know if its working?

@ThomasWaldmann

This comment has been minimized.

Copy link
Member Author

commented Apr 10, 2018

@bpereto I'll do some tests using the vagrant-based borgbackup tests:

  • platforms: linux (debian jessie 64bit), freebsd 64bit, darwin 64bit
  • pythons: 3.4.0, 3.5.0, 3.6.0
  • msgpacks: msgpack-python and msgpack package (via pip)

At the end, I'll have a look at the changelog also:

https://github.com/msgpack/msgpack-python/blob/master/ChangeLog.rst

@ThomasWaldmann

This comment has been minimized.

Copy link
Member Author

commented Apr 10, 2018

0.4.6...0.4.8: heavily tested with borg, works ok.

a msgpack pure python data corruption was fixed in 0.4.6, so don't use anything less than 0.4.6.

0.4.8 last version with official support for python 3.4.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member Author

commented Apr 10, 2018

pip install msgpack-python==0.5.0

broken: installs msgpack-python==0.5.0 AND msgpack==0.5.6 !?


pip install msgpack==0.5.0

broken: FutureWarning: use bin_type option is not specified (in msgpack), 20 borg tests broken.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member Author

commented Apr 10, 2018

pip install msgpack-python==0.5.1

tox tests linux64: OK
tox tests freebsd64: OK
tox tests darwin64: OK


pip install msgpack==0.5.1

tox tests linux64: OK

@ThomasWaldmann

This comment has been minimized.

Copy link
Member Author

commented Apr 10, 2018

pip install msgpack-python==0.5.2

broken: "TypeError: expected bytes, str found" (in msgpack.Packer / packb)

@ThomasWaldmann

This comment has been minimized.

Copy link
Member Author

commented Apr 10, 2018

pip install msgpack-python==0.5.3

broken: no package available on pypi.


pip install msgpack==0.5.3

tox tests linux64: OK
tox tests freebsd64: OK
tox tests darwin64: OK

@ThomasWaldmann

This comment has been minimized.

Copy link
Member Author

commented Apr 10, 2018

pip install msgpack-python==0.5.4

tox tests linux64: OK


pip install msgpack==0.5.4

tox tests linux64: OK
tox tests freebsd64: OK
tox tests darwin64: OK

This version (at least, unclear whether previous versions are also affected) has a memory leak in the pure python fallback implementation, see #3573 and msgpack/msgpack-python#283.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member Author

commented Apr 10, 2018

pip install msgpack-python==0.5.5

tox tests linux64: OK
tox tests freebsd64: OK
tox tests darwin64: OK


pip install msgpack==0.5.5

tox tests linux64: OK

@ThomasWaldmann

This comment has been minimized.

Copy link
Member Author

commented Apr 10, 2018

pip install msgpack-python==0.5.6

tox tests linux64: OK
tox tests freebsd64: OK
tox tests darwin64: OK


pip install msgpack==0.5.6

tox tests linux64: OK

@ThomasWaldmann

This comment has been minimized.

Copy link
Member Author

commented Apr 11, 2018

OK, looking at the test results from above AND the changelog at https://github.com/msgpack/msgpack-python/blob/master/ChangeLog.rst , I come to this conclusion:

From the 0.5.x series, 0.5.6 looks good, but everything else looks too problematic / broken.

TODO: update requirements: 0.4.6, 0.4.7, 0.4.8, 0.5.6.

@ThomasWaldmann ThomasWaldmann added this to the 1.1.6 milestone Apr 11, 2018

@bpereto

This comment has been minimized.

Copy link

commented Apr 11, 2018

bad news, 0.5.1 is already in epel. no other version available.

https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-3b0dd1f4e0

ThomasWaldmann added a commit to ThomasWaldmann/borg that referenced this issue Apr 12, 2018

@ThomasWaldmann ThomasWaldmann self-assigned this Apr 12, 2018

ThomasWaldmann added a commit to ThomasWaldmann/borg that referenced this issue Apr 13, 2018

ThomasWaldmann added a commit that referenced this issue Apr 13, 2018

uqs pushed a commit to freebsd/freebsd-ports that referenced this issue Apr 24, 2018

sunpoet
Fix msgpack dependency
- Update RUN_DEPENDS
- Bump PORTREVISION for package change

Bump msgpack requirement to 0.5.6 which is the only acceptable 0.5.x version [1].

PR:		227717
Reported by:	<iron.udjin@gmail.com>
Submitted by:	sunpoet (myself)
Approved by:	Jose Garcia Juanino <jjuanino@gmail.com>
Reference:	borgbackup/borg#3753 [1]


git-svn-id: svn+ssh://svn.freebsd.org/ports/head@468235 35697150-7ecd-e111-bb59-0022644237b5

uqs pushed a commit to freebsd/freebsd-ports that referenced this issue Apr 24, 2018

Fix msgpack dependency
- Update RUN_DEPENDS
- Bump PORTREVISION for package change

Bump msgpack requirement to 0.5.6 which is the only acceptable 0.5.x version [1].

PR:		227717
Reported by:	<iron.udjin@gmail.com>
Submitted by:	sunpoet (myself)
Approved by:	Jose Garcia Juanino <jjuanino@gmail.com>
Reference:	borgbackup/borg#3753 [1]

Jehops pushed a commit to Jehops/freebsd-ports that referenced this issue Apr 24, 2018

Fix msgpack dependency
- Update RUN_DEPENDS
- Bump PORTREVISION for package change

Bump msgpack requirement to 0.5.6 which is the only acceptable 0.5.x version [1].

PR:		227717
Reported by:	<iron.udjin@gmail.com>
Submitted by:	sunpoet (myself)
Approved by:	Jose Garcia Juanino <jjuanino@gmail.com>
Reference:	borgbackup/borg#3753 [1]


git-svn-id: svn+ssh://svn.freebsd.org/ports/head@468235 35697150-7ecd-e111-bb59-0022644237b5

mat813 pushed a commit to mat813/freebsd-ports that referenced this issue Apr 25, 2018

sunpoet
Fix msgpack dependency
- Update RUN_DEPENDS
- Bump PORTREVISION for package change

Bump msgpack requirement to 0.5.6 which is the only acceptable 0.5.x version [1].

PR:		227717
Reported by:	<iron.udjin@gmail.com>
Submitted by:	sunpoet (myself)
Approved by:	Jose Garcia Juanino <jjuanino@gmail.com>
Reference:	borgbackup/borg#3753 [1]


git-svn-id: https://svn.freebsd.org/ports/head@468235 35697150-7ecd-e111-bb59-0022644237b5

swills pushed a commit to swills/freebsd-ports that referenced this issue Apr 25, 2018

Fix msgpack dependency
- Update RUN_DEPENDS
- Bump PORTREVISION for package change

Bump msgpack requirement to 0.5.6 which is the only acceptable 0.5.x version [1].

PR:		227717
Reported by:	<iron.udjin@gmail.com>
Submitted by:	sunpoet (myself)
Approved by:	Jose Garcia Juanino <jjuanino@gmail.com>
Reference:	borgbackup/borg#3753 [1]


git-svn-id: svn+ssh://svn.freebsd.org/ports/head@468235 35697150-7ecd-e111-bb59-0022644237b5
@ThomasWaldmann

This comment has been minimized.

Copy link
Member Author

commented Apr 28, 2018

Fixed by #3767.

ThomasWaldmann added a commit to ThomasWaldmann/borg that referenced this issue May 18, 2018

ThomasWaldmann added a commit that referenced this issue May 18, 2018

Merge pull request #3842 from ThomasWaldmann/msgpack-requirement-master
update msgpack requirement, fixes #3753 (master)

refacto added a commit to refacto/aports that referenced this issue May 28, 2018

@ThomasWaldmann

This comment has been minimized.

Copy link
Member Author

commented Jan 14, 2019

Just for completeness:

borg 1.1.x does not work with msgpack >= 0.6.0, there are multiple issues (search this issue tracker, if interested).

master branch has quite some changes to support recent msgpack versions (and drops support for old msgpack versions).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.