[IO-1445][external] Changes to LocalDataset() & get_annotations() to account for local releases pulled with folders

[JBWilkie]

… pulled with folders

Problem

When local releases are pulled with folders, construction of LocalDataset objects & the get_annotations() function fail due to a mismatch between annotation file names and item names. A more in-depth breakdown is available in the IO-1445 Linear ticket

Solution

Instead of comparing the file names of the JSON with items, parse each JSON file to guarantee the correct local path for each item is checked against. This has the side effect of introducing significantly higher runtimes when parsing large JSON files. Therefore, make LocalDataset & get_annotations() use a streaming library to prevent this.

Changelog

• Changes to LocalDataset construction & get_annotations() function to account for local releases pulled with folders, including releases with identically named files.
• Addition of the json_stream library to darwin-py to improve runtimes when reading certain fields from JSON files

[linear]

IO-1445 BUG: Classification model failed A potential customer (trial) contacted us saying that their classification model failed

BUG submission from: Patryk Gronostajski
Summary (describe an issue):
Classification model failed

A potential customer (trial) contacted us saying that their classification model failed. I tried running one in their team and it failed as well. They want to use the "Formal" and "Informal" tags, and all items in the dataset.
Share Loom/Screenshots with Console/Network opened:
N/A
Darwin affected version
V2
Environment (production/staging; browser and OS version)
2.0 prod
Impact
Single (only 1 customer is impacted)
Priority
🔴 SWAT Urgent - an urgent problem that blocks users
Steps to reproduce

1. Attempt to train a model using the "Formal" and "Informal" tags, and all items in the dataset.
   Expected Behaviour
   The model trains successfully.
   Team & Dataset Link
   Dataset: https://darwin.v7labs.com/datasets/658275/dataset-management
   Team: https://darwin.v7labs.com/super_admin/teams/5630
   Intercom ticket
   https://app.intercom.com/a/inbox/pb9z3asr/inbox/admin/6578327/conversation/235699?view=List
   Customer Name
   Beder Acosta Borges
   Renewal timeline

ARR

Tier

Assigned CSM

[JBWilkie]

Failing tests, need to investigate

[Nathanjp91]

LGTM. Revisions seem fine with the QA and testing done.