Skip to content

Conversation

@vbrodsky
Copy link
Contributor

@vbrodsky vbrodsky commented May 12, 2023

Story: https://labelbox.atlassian.net/browse/AL-5686

Even though this is a large PR, 90% of files is just a drop-in replacement of the 3d party ndjson library with our own. The rest 10% are important updates to the parser library to address issues from integration tests and removal of the 3d party ndjson library from our setup and requirements.txt

List of Affected SDK API's

  • prediction upload
    • LabelImport#create_from_objects
    • MEAPredictionImport#create_from_objects
  • export v2
    • Batch#export_data_rows
    • Dataset#export_data_rows (can just test this and not Batch)
    • ModelRun#export_labels
    • BulkImportRequest create_from_url, create_from_objects, create_from_local_file, also in errors, statuses etc
  • other
    • Project#export_queued_data_rows
    • Task#failed_data_rows, errors, result etc

@vbrodsky vbrodsky force-pushed the VB/replace-ndjson branch 5 times, most recently from e821dfe to 7dabe51 Compare May 12, 2023 20:32
Val Brodsky added 9 commits May 12, 2023 13:33
Replace ndjson.loads with parser.loads
…d Python json decoder

To be able to handle loading byte, bytearray etc. all other types as support by the json library itself
Also add dump() to the parser, used in tests only
@vbrodsky vbrodsky force-pushed the VB/replace-ndjson branch from 7dabe51 to c366c6b Compare May 12, 2023 20:35
@vbrodsky vbrodsky requested a review from kkim-labelbox May 12, 2023 20:39
Copy link
Contributor

@whistler whistler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like an elegant solution!

Copy link
Contributor

@yamini-labelbox yamini-labelbox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Super clean! Ship it after MLSE approves.

@vbrodsky vbrodsky merged commit 7552be6 into develop May 18, 2023
@vbrodsky vbrodsky deleted the VB/replace-ndjson branch May 18, 2023 00:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants