High response times and system load after adding data sets

[hackdna]

• Specific code commit: v1.6.3
• Version of the web browser and OS: unknown
• Environment where the error occurred: AWS

Steps to reproduce

It appears that this happens after adding several data sets.

Observed behavior

Site responds very slowly. High CPU utilization and system load.

ubuntu@ip-172-30-0-50:~$ uptime
 14:55:31 up 42 days, 17:29,  1 user,  load average: 6.66, 3.74, 1.64

Expected behavior

Site responds without delays. Normal CPU utilization and system load.

Notes

• may be related to Optimize ISA-Tab import performance #463, "Exponential explosion" when trying to upload 2 line CSV #2277, Solr connection timeout error during data set import #2394 and CPU utilization continuously high #2965
• high CPU utilization appears to be caused by Neo4j

[hackdna]

@jkmarx reported the v1.6.5 test site being very slow
@scottx611x reported neo4j process "hogging resources" for an unknown reason
CPU utilization on the web instance was maxed out for several minutes at a time:

This is with only 2-4 users interacting with the site concurrently

[jkmarx]

@scottx611x Can use this issue as the Neo4J issue or should we write a separate issue?

Possible routes

• Configuration possibility (SO look into it)
• Drop per user search?
• Proposal to update annotate at certain times (3am)

[scottx611x]

Potential avenues for a fix from Fritz:

Hi Nils, thanks to Scott we now know what the bottleneck with uploading new docs is: it's the creation of the annotation set hierarchy in Neo4J (https://github.com/refinery-platform/neo4j-ontology). What the tool basically does is creating sub-ontologies for every user based on the annotation of available datasets. Creating this ontology takes about 1sec for ~500 datasets for 1 users. Given that you guys have ~360 users it takes ~6mins. Back in the day there were no users so I didn't spend time optimizing and went the safe route (update the hierarchy for all users whenever a dataset was uploaded, shared, or deleted). Apparently this is no longer feasible given the increase in users.

I see 3 general ways to improve this:

1. Try to optimize my Neo4J plugin. I am sure there are ways to cleverly cache certain parts to cut down the computation time of a user's annotation set hierarchy. The traversal of along the Neo4J network might also be improvable. Someone with basically needs to get improve this tool here: https://github.com/refinery-platform/neo4j-ontology/blob/master/src/main/java/org/neo4j/ontology/server/unmanaged/AnnotationResource.java#L190
2. Depending on whether a new dataset is uploaded or shared not all users's annotation set hierarchy needs to be updated. Depending on the group of users who would have access only those users need to be updated. This might still take a long time because if you make a dataset public everyone needs to be updated. Also, even if everyone needs to be updated, if the dataset doesn't have a any new annotations the update in general is not needed. Hence, there's a lot of room for improvement but one needs to carefully catch every condition. (i.e., which user would be afftected by the action and does the action actually afffect the annotations)
3. A user's annotation set hierarchy doesn't need to be updated until the user is logged in actually. Since the individual computations is pretty fast (I think 1 sec waiting at the beginning is not that bad) it might be better to trigger the update of a user's annotation set hierarchy once the user actually logs in. Special care needs to be taken when the user is already logged in and what to do when a lot of people are logged in at the same time. hence, one could do something like: the user's annotation set hierarchy who uploaded / change the dataset gets updated immediately, everybody who is logged in will see an update button if something changed to update their annotation set hierarchy manually, and everybody else's annotation set hierarchy get updated once they log in.

[scottx611x]

Still an issue during release-1.6.7 testing