sssom-py is much too slow

[matentzn]

We need to figure out why that is first of all, i.e. which functions are so inefficient, and then working on improving efficiency. First goal:

• A small document of 100 mappings should be processable in 1 second.

[hrshdhgd]

Here are my observations:

1. I ran test_conversion.py since that took the most time amongst all other tests. The following is the CPU profiling graph (courtesy: Austin).
   

As noted, as_rdf_graph takes a noticeable amount of time (~1.2 seconds) which is called at least 4 times within the test.

2. On running sssom convert A.json -o A.tsv gave me the following:
   

• Immediate improvements needed:
  - from_sssom_dataframe could be recoded to be more efficient.

- same with `to_mapping_set_document`

- But `init` is common for both above:

command for documentation purposes: sudo austin -i 100 -o /path/to/austin/output.austin python -m pytest

[matentzn]

Thank you @hrshdhgd this is a great analysis! You can put it a bit on the backburner now and we get back to it later!

[joeflack4]

Agree it is too slow, and also a good analysis!

[matentzn]

Writing write_owl(msdf1,f) for large msdfs takes a huge amount of time (100MB tsv ~ > 2 hours)

[matentzn]

(sssom) ➜  ontology git:(master) ✗ wc -l ../mappings/oba-all-phenotype.sssom.tsv
 1712464 ../mappings/oba-all-phenotype.sssom.tsv
(sssom) ➜  ontology git:(master) ✗ sssom filter ../mappings/oba-all-phenotype.sssom.tsv -o ../mappings/oba-all-hp-phenotype.sssom.tsv --subject_id HP:% --object_id OBA:%

This should be basically instant but takes 30 minutes. Maybe bypass linkml for certain operations?

[hrshdhgd]

Closing this for now in favor of #462 . Feel free to re-open a new issue with exact location of latency improvement needed.