Stash large response payload to S3 for `/descendants/<id>`

[yuanzhou]

The https://entity.api.hubmapconsortium.org/descendants/HBM925.HSGR.556 request to get back all descendants of Donor HBM925.HSGR.556 retuned 500 with

{
    "message": "Internal server error"
}

Further logging indicates the response payload is 11.7 MB:

Mon Oct 21 20:56:21 2024 - uwsgi_response_write_body_do(): Connection reset by peer [core/writer.c line 341] during GET /descendants/HBM925.HSGR.556 (172.31.23.74)
OSError: write error
[pid: 259|app: 0|req: 2284600/3109683] 172.31.23.74 () {52 vars in 993 bytes} [Mon Oct 21 20:56:16 2024] GET /descendants/HBM925.HSGR.556 => generated 11743268 bytes in 5791 msecs (HTTP/1.1 200) 2 headers in 77 bytes (4 switches on core 10)

Follow the existing implementation, use hubmap_commons.S3_worker to stash the results (when the response payload is >=10MB) in AWS S3 bucket and return a URL and 303 status code.

Note: /descendants/HBM925.HSGR.556?property=uuid returned 200 with 24798 uuids, and the response payload is 848 KB.

[yuanzhou]

A related issue #778 was observed for Dataset https://entity.api.hubmapconsortium.org/descendants/4ae5a725fa9400daf76bc24c0dbe1e1d but there's no error message in the log. And essentially this is different from the Donor descendants problem.

From what I can tell, it's caused by the large number of files in the descendants. Similar to what we encountered for the Collection/Upload with associated datasets containing a very big list of entries in the files field. A decision needs to be made if we'll just want to exclude those offending fields from the /ancestors, /parents, /descendants, and /children calls rather than returning a S3 link.

[kburke]

#821 for stated problem.
Dataset 4ae5a725fa9400daf76bc24c0dbe1e1d not in Dev for evaluation.

[yuanzhou]

@kburke For testing purposes, I found these entities on DEV that have a lot of descendants. Can you verify which one returns 500 now? Then another round of check after I deploy your S3 updates on DEV to compare.

╒══════════════════════════════════╤═════════════════╤═══════════════╕
│e.uuid                            │e.hubmap_id      │descendants_cnt│
╞══════════════════════════════════╪═════════════════╪═══════════════╡
│"3a0960c7cc8864dcc003165cef9ca040"│"HBM925.HSGR.556"│24798          │
├──────────────────────────────────┼─────────────────┼───────────────┤
│"8b34faba613d743dab7392bd22721ddb"│"HBM742.FCHF.843"│24797          │
├──────────────────────────────────┼─────────────────┼───────────────┤
│"b835cb1865b044aac1220181ff19e2e5"│"HBM344.GLSM.884"│2552           │
├──────────────────────────────────┼─────────────────┼───────────────┤
│"f066e9ae118f0d3cf1fc8c77b8533fb9"│"HBM873.FDBJ.735"│2551           │
├──────────────────────────────────┼─────────────────┼───────────────┤
│"171fc88db789ff330b0d7a420b642a23"│"HBM633.KPHW.963"│1077           │
├──────────────────────────────────┼─────────────────┼───────────────┤
│"01494d3a65e0f5b82c6b4db5a1d8b3cf"│"HBM767.GMFZ.728"│1076           │
├──────────────────────────────────┼─────────────────┼───────────────┤
│"b5b4de30c90e4cb3def65407292942e9"│"HBM853.HXBK.427"│1032           │
├──────────────────────────────────┼─────────────────┼───────────────┤
│"3ad74c9d10ccf828672b2ba990f915a8"│"HBM624.KWHJ.622"│857            │
├──────────────────────────────────┼─────────────────┼───────────────┤
│"aa97d20b4a8f3c1198826091bc5455d9"│"HBM679.GXQW.326"│772            │
├──────────────────────────────────┼─────────────────┼───────────────┤

[kburke]

@yuanzhou Only HBM925.HSGR.556 and HBM742.FCHF.843 returned 303 with an S3 URL against my localhost entity-api. All the others returned 200 and a JSON body. There were no HTTP 500 Responses.

[yuanzhou]

Fix released to PROD and https://entity.api.hubmapconsortium.org/descendants/HBM925.HSGR.556 is returning 303 with the S3 link. The payload is 21 MB.