Investigate Collection loading error

[yuanzhou]

Dominic at PSC created a Collection on the ingest portal, but got an error when trying to visit the collection at https://ingest.hubmapconsortium.org/collection/14f93f4c97996eefd5779f6a29d3f13c

Upon further check, entity-api returned 500 at https://entity.api.hubmapconsortium.org/entities/14f93f4c97996eefd5779f6a29d3f13c, ruling out the UI handling issue.

[yuanzhou]

Several datasets in this collection have a very large ingest_metadata field, which is causing the backend processing issue.

╒══════════════════════════════════╤════════════════════╕
│dataset.uuid                      │ingest_metadata_size│
╞══════════════════════════════════╪════════════════════╡
│"d3f248fdcf73714f355d08c26d0ad3ce"│4355177             │
├──────────────────────────────────┼────────────────────┤
│"37b059eacd469d2d77bd351c03343afd"│4216219             │
├──────────────────────────────────┼────────────────────┤
│"9dd9e791fafb87652b36a53e8d930ba4"│4121243             │
├──────────────────────────────────┼────────────────────┤
│"b366be9d87169630e3c009bb6a1542b5"│4094716             │
├──────────────────────────────────┼────────────────────┤
│"7ccde454e6c81b08d2d6a582e8c16bfb"│3556863             │
├──────────────────────────────────┼────────────────────┤
│"38fb0fc081d58a1e7f1821b646122ac9"│3531690             │
├──────────────────────────────────┼────────────────────┤
│"fd8638d080baa2bf40a93381d5e6aa38"│681                 │
├──────────────────────────────────┼────────────────────┤
│"3d50d67060511effab1613275d811bc7"│681                 │

...

│"5d5b8c80091966f53bda50a1053838bf"│588                 │
└──────────────────────────────────┴────────────────────┘

[yuanzhou]

Backstory

Over a year or two years ago we changed to only return the necessary/minimal attributes of Collection.datasets without running any trigger methods, mainly for performance need. Later an explicit allowlist was used to specify the fields agreed with Harvard portal-ui team. Afterwards the Harvard team figured out how to retrieve each individual dataset using a separate query on the portal-ui's collection page. So we stopped using the allowlist and switched to a Neo4j query returning all the attributes directly available in Neo4j, still no trigger methods being used.

With Dataset.ingest_metadata growing constantly, especially the nested files field, our current handling is posing a performance issue, that's where this card came from.

Short-term solution combing with ingest_metadata reorg

Exclude files, metadata, and ingest_metadata from the Neo4j result for both Collection and Upload associated datasets:

entity-api/src/schema/schema_neo4j_queries.py

Line 1184 in c9714a3

def get_collection_datasets(neo4j_driver, uuid):

entity-api/src/schema/schema_neo4j_queries.py

Line 1390 in c9714a3

def get_upload_datasets(neo4j_driver, uuid, property_key = None):

Long-term solution based on real need and milestone priority

Only return a list of dataset uuids for a given Collection/Upload. This will require corresponding changes made in ingest-ui's Collection/Epicollection/Upload pages to fetch the dataset details using separate Elasticsearch query on page loading. Examples:

• Collection HBM585.QPDV.454 with 211 datasets: https://ingest.hubmapconsortium.org/collection/3ae4ddfc175d768af5526a010bfe95aa
• Upload HBM398.VCBS.396 with 186 datasets: https://ingest.hubmapconsortium.org/upload/0a30f2eddaa1eb2f4ba2b1f7bcb17524

[shirey]

@yuanzhou For the Long-term solution how feasible would it be to still include all attributes except files, metadata and ingest_metadata in the search/Elasticsearch indices?

[yuanzhou]

@shirey the long-term solution will not affect any documents indexed into Elasticsearch. Currently in search-api when indexing Collections and Uploads, it iterates over the Collection.datasets and Upload.datasets with only parsing the dataset.uuid and making individual calls to build the document. In the future if entity-api only returns Collection.dataset_uuids and Upload.datasets_uuids (the name of the field is pending), it won't cause any issues to search-api.

With the above being said, we also don't have to exclude any of the fields from the indices.

[yuanzhou]

For the record, reindexing this Collection also failed.

[2024-12-03 23:06:04] ERROR in es_writer:95: Failed to write doc of uuid: 14f93f4c97996eefd5779f6a29d3f13c to index: fill20241203_fresh_index_hm_prod_consortium_entities
[2024-12-03 23:06:04] ERROR in es_writer:96: Error Message: {"error":{"root_cause":[{"type":"mapper_exception","reason":"timed out while waiting for a dynamic mapping update"}],"type":"mapper_exception","reason":"timed out while waiting for a dynamic mapping update"},"status":500}
[2024-12-03 23:23:34] ERROR in es_writer:95: Failed to write doc of uuid: 14f93f4c97996eefd5779f6a29d3f13c to index: fill20241203_fresh_index_hm_prod_consortium_portal

[yuanzhou]

Removing the offending fields from Neo4j result via PR #769