Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use srsly for JSON serialization #19

Merged
merged 10 commits into from
Apr 21, 2023
Merged

Use srsly for JSON serialization #19

merged 10 commits into from
Apr 21, 2023

Conversation

prrao87
Copy link
Owner

@prrao87 prrao87 commented Apr 21, 2023

Updates

  • srsly is a fast and lightweight JSON serialization library from Explosion.

    • It eliminates a lot of boilerplate for util functions that read/write compressed JSONL files (in gzip format)
    • Using this library each bulk indexing script is very simple, doesn't add much overhead to the pip install time, and reduces the number of lines of code quite significantly
    • The code base for Elasticsearch, Meilisearch and Neo4j have all been updated to use srsly to read gzipped JSONL
    • For future DBs, the same approach will be used to also keep things clean and readable
  • For Meilisearch, the settings specification is moved over to a settings.json to keep things clean and easy to find all in one place

@prrao87 prrao87 added the enhancement New feature or request label Apr 21, 2023
@prrao87 prrao87 changed the title Use Srsly for JSON serialization Use srsly for JSON serialization Apr 21, 2023
@prrao87 prrao87 merged commit def1585 into main Apr 21, 2023
1 check passed
@prrao87 prrao87 deleted the srsly branch April 21, 2023 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants