msgpack 1.0.0 changes default string encoding / breaks fluentbit communication

[ptramsey]

https://pypi.org/project/msgpack/1.0.0/ -- see the section, "Major breaking changes in 1.0":

use_bin_type=True by default. bytes are encoded in bin type in msgpack. If you are still sing Python 2, you must use unicode for all string types. You can use use_bin_type=False to encode into old msgpack format.

if use_bin_type=False isn't passed to FluentSender in msgpack_kwargs, fluentbit will see that the first element of each received message is not of type MSGPACK_OBJECT_STR, and refuse to enqueue the message. If you have debug output turned on in fluent-bit, you should see it output a message like,

[2020/02/20 21:19:55] [debug] [in_fw] parser: invalid tag format, skip.

I have verified that this happens in fluent-bit 1.3.5, and while I haven't tried it using 1.3.8, the relevant code in that release appears unchanged.

[ptramsey]

I'm not sure what the correct path forward is --- perhaps use_bin_type=False should be included by default in msgpack_kwargs. Alternatively, the dependency could be pinned to python-msgpack <= 0.6.2, since 1.0.0 has degraded performance for python 2.

[arcivanov]

I'm going to pin it under 1.0.0 for now as a short term solution. Python 2 being dead is not a terrible concern anymore.

Thanks for the report!

[keuko]

I'm going to pin it under 1.0.0 for now as a short term solution. Python 2 being dead is not a terrible concern anymore.

Thanks for the report!

Hello,

Firstly I would like to say thank you for python-fluent-logger !
Now, please could I ask if there is a plan to support msgpack > 1.0.0 ? As we have plans in Kolla Openstack to integrate fluent-logger into central_logging of openstack. And newest lower-constraints in openstack is msgpack 1.0.2 ..

https://review.opendev.org/c/openstack/kolla-ansible/+/755775
https://review.opendev.org/c/openstack/kolla/+/759855

I am also packaging python-fluent-logger for debian distro, so can I expect new version ?
This issue was closed almost one year ago and from your comment pin msgpack to < 1.0.0 was only short solution..

Thank you very much !
Michal Arbet ( kevko )

[keuko]

There is also pull request for unpin msgpack and some patch here ->
#171

Could you please verify and merge ?

[arcivanov]

In #173 I'm removing Python 2 compatibility and setting msgpack>1.0 dependency. This way everyone on Python 2 still can use 0.9.x.