Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add numbin as another serialization library #88

Open
kemingy opened this issue Mar 15, 2023 · 2 comments
Open

Add numbin as another serialization library #88

kemingy opened this issue Mar 15, 2023 · 2 comments

Comments

@kemingy
Copy link

kemingy commented Mar 15, 2023

Hi, I'm the author of numbin (An efficient binary serialization format for numpy data.) I just found this repo and I wonder if you will accept numbin as another serialization library.

Check the benchmark here.

BTW, I don't know there is a library called msgpack-numpy before I develop numbin. After explore the source code, I think numbin would provide better performance and flexibility.

@adrianeboyd
Copy link
Contributor

Thanks for the pointer! To a large extent srsly is developed for the formats we're using for Explosion packages/projects and my initial feeling is that we're unlikely to replace msgpack-numpy with another option unless there are large advantages because it could be complicated to handle a lot of the existing saved data.

We do occasionally run into the size limits for items saved with msgpack, but it looks like this would be the same for numbin?

@kemingy
Copy link
Author

kemingy commented Mar 18, 2023

We do occasionally run into the size limits for items saved with msgpack, but it looks like this would be the same for numbin?

That depends. Numbin doesn't require msgpack if your data is just NumPy array. It's only required when you have the mixed data type.

I'm not sure about the msgpack size limitation. Will take a look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants