Roll out KG2.8.0

[amykglen]

NOTE: To create a new issue based on this template, simply go to: https://github.com/RTXteam/RTX/issues/new?template=kg2rollout.md

1. Build and load KG2c:

• merge master into the kg2integration branch
• update the four hardcoded biolink version numbers in the kg2integration branch (as needed):
  • in code/UI/OpenAPI/python-flask-server/openapi_server/openapi/openapi.yaml
  • in code/UI/OpenAPI/python-flask-server/KG2/openapi_server/openapi/openapi.yaml
  • in code/UI/OpenAPI/python-flask-server/RTX_OA3_TRAPI1.2_ARAX.yaml
  • in code/UI/OpenAPI/python-flask-server/RTX_OA3_TRAPI1.2_KG2.yaml
• build a new KG2c on buildkg2c.rtx.ai from the kg2integration branch (how-to is here)
  • before starting the build:
    • make sure there is enough disk space available on arax.ncats.io (need at least 110G, ideally >130G). delete old KG2 database directories from /data/orangeboard/databases as needed (warn the team on Slack in advance).
    • make sure to choose to build a new synonymizer in kg2c_config.json, as described in the how-to
  • after the build is done, verify it looks ok:
    • the 'build node' (node with ID RTX:KG2c) has the expected version number (KG2.X.Y)
    • node_synonymizer.sqlite should be around 15-20 GB
    • make sure node_synonymizer.sqlite's last modified date is today (or whatever day the build was run)
    • make sure kg2c_lite.json.gz's last modified date is today (or whatever day the build was run)
    • the entire build runtime (synonymizer + KG2c) shouldn't have been more than 24 hours
    • the synonymizer and KG2c artifacts should have been auto-uploaded into the proper directory on arax-databases.rtx.ai (/home/rtxconfig/KG2.X.Y)
• load the new KG2c into neo4j at http://kg2-X-Yc.rtx.ai:7474/browser/ (how to is here)
  • verify the correct KG2 version was uploaded by running this query: match (n {id:"RTX:KG2c"}) return n
  • copy the auto-generated KG2c dump from the Neo4j-hosting instance to the databases server like so:
    • scp /home/ubuntu/kg2-build/kg2c.dump rtxconfig@arax-databases.rtx.ai:/home/rtxconfig/KG2.X.Y/extra_files
  • update the KG2pre and KG2c Neo4j endpoints in RTX/code/config_dbs.json (push to kg2integration branch)
• upload the new kg2c_lite_2.X.Y.json.gz file to the translator-lfs-artifacts repo
• load the new KG2c into Plover (available at http://kg2cplover.rtx.ai:9990)
  • update config_dbs.json to point to this new Plover (all maturity levels should point to it for now)

2. Rebuild downstream databases:

The following databases should be rebuilt and copies of them should be put in /home/rtxconfig/KG2.X.Y on arax-databases.rtx.ai. Please use this kind of naming format: mydatabase_v1.0_KG2.X.Y.sqlite.

• NGD database
• refreshed DTD @chunyuma
• DTD model @chunyuma (may be skipped - depends on the changes in this KG2 version)
• DTD database @chunyuma (may be skipped - depends on the changes in this KG2 version)
• XDTD database

NOTE: As databases are rebuilt, RTX/code/config_dbs.json will need to be updated to point to their new paths! Push these changes to the kg2integration branch, unless the rollout of this KG2 version has already occurred, in which case you should push to master (but first follow the steps described here).

3. Update the ARAX codebase:

All code changes should go in the kg2integration branch!

• regenerate the KG2c test triples file @acevedol
  • ensure the new KG2c Neo4j is currently running
  • check out the kg2integration branch and pull to get the latest changes (this is important for ensuring the correct KG2c Neo4j is used)
  • run this script
  • push the regenerated file to RTX/code/ARAX/KnowledgeSources/RTX_KG2c_test_triples.json (in the kg2integration branch)
• update Expand code as needed
• update any other modules as needed
• test everything together:
  • check out the kg2integration branch and pull to get the latest changes
  • locally set force_local = True in ARAX_expander.py (to avoid using the old KG2 API)
  • then run the entire ARAX pytest suite
  • address any failing tests
• update the KG2 and ARAX version numbers in the appropriate places (openapi.yaml, etc.) @edeutsch
  • Bump version for RTX/code/UI/OpenAPI/python-flask-server/openapi_server/openapi/openapi.yaml in line 12 (version:); the major and minor release numbers are kept synchronous with the TRAPI version; just bump the patch release version (least significant digit)
  • Bump version for RTX/code/UI/OpenAPI/python-flask-server/KG2/openapi_server/openapi/openapi.yaml in line 12 (version:); the first three digits are kept synchronous with the KG2 release version
  • Bump version number in RTX/code/UI/OpenAPI/python-flask-server/RTX_OA3_TRAPI1.3_ARAX.yaml on line 4 (version:); same as for the ARAX openapi.yaml file
  • Bump version number in RTX/code/UI/OpenAPI/python-flask-server/RTX_OA3_TRAPI1.3_KG2.yaml on line 4 (version:); same as for the KG2 openapi.yaml file

4. Do the rollout:

• pre-download new databases to arax.ncats.io (under data/orangeboard/databases/KG2.X.Y; first verify there's enough space)
• upload the new databases (referenced in config_dbs.json) to ITRB's SFTP server using the steps detailed here
  • make sure to do this before proceeding with the below steps; this can be done well in advance of the actual roll-out (it doesn't hurt anything to do it early)
• merge master into kg2integration
• merge kg2integration into master
• roll master out to the various arax.ncats.io endpoints
• run the database manager
• run the pytest suite on the various endpoints
• verify each endpoint is running the new KG2 version by running the following JSON query and inspecting the returned node:
  • {"nodes": {"n00": {"ids": ["RTX:KG2c"]}}, "edges": {}}
• update our CI/CD testing instance with the new databases:
  • ssh ubuntu@cicd.rtx.ai
  • cd RTX
  • git pull origin master
  • python3 code/ARAX/ARAXQuery/ARAX_database_manager.py --mnt --skip-if-exists --remove_unused

5. Final items/clean up:

• generate KGX files and upload them to the KGE Archive @acevedol
• turn off the old KG2c version's neo4j instance
• turn off the old KG2c version's plover instance
• turn off the new KG2pre version's neo4j instance
• upgrade the ITRB Plover endpoint (https://kg2cploverdb.ci.transltr.io) to this KG2 version and make the KG2 API starts using it (instead of our self-hosted endpoint):
  • update kg_config.json in the main branch of the Plover repo to point to the new kg2c_lite_2.X.Y.json.gz file (push this change)
  • while ITRB continuous deployment isn't working: request CI deployment of Plover
  • after the CI Plover has been deployed, run the ARAX pytest suite with the NCATS endpoint plugged in (locally change the URL in config_dbs.json and set force_local = True in Expand)
  • if all tests pass, update config_dbs.json in master to point to the ITRB Plover endpoints (all maturity levels)
  • roll master out to the various endpoints on arax.ncats.io
  • turn off our plover endpoint and verify once more that ARAX is still working ok

[acevedol]

I updated the CName record for kg2canonicalized.rtx.ai to point to kg2-8-0c.rtx.ai
Loaded KG2c into Neo4J and verified the version is 2.8.0
I am unable to copy the kg2c dump file to arax.ncats.io at this time

I uploaded kg2c_lite_2.8.0.json.gz to LFS under the branch kg2.8.0. It's waiting on pull request approval from @amykglen because I can't push directly to main on there.

[amykglen]

awesome, thanks @acevedol! just merged the PR.

[saramsey]

Just an FYI, issue NCATSTranslator/testing 246 is indirectly requesting Biolink 3.0 support in ARAX for a specific query question ("What chemicals [qualified] affect (decreases/downregulates) a given protein/gene (COX2)?"). I am cross-listing the issue here so that we can update the NCATSTranslator/testing issue when KG2.8.0c is rolled out into an ITRB-deployed ARAX.

[amykglen]

hey @acevedol @sundareswarpullela - it sounds like Sundar finished creating the KG2.8.0c rebuild that has the patch in place for #1942 / RTXteam/RTX-KG2#250.

wondering if you two could take care of loading that new KG2c into neo4j and uploading the kg2c_lite.json file to the LFS repo?

[sundareswarpullela]

Hey, sure, will get in touch with @acevedol and get started.

[amykglen]

fyi I just took care of loading the patched KG2c into Neo4j

[amykglen]

hey @chunyuma - are you planning to regenerate the COHD, DTD, and/or XDTD databases for KG2.8.0? if so, I believe all the files you need are on arax-databases.rtx.ai. and the new KG2c neo4j is up at http://kg2-8-0c.rtx.ai:7474/browser/

[chunyuma]

Hi @amykglen, I will start to do it today. Thanks for letting me know all data of KG2.8.0 is ready.

[amykglen]

sounds good - and just fyi, shortly after posting my last comment above I realized that the bug we thought we had patched where some edges were missing qualified_predicates isn't quite fixed yet.

so if you need to use the new qualified_predicate property in any of your builds, you should probably wait for the KG2c rebuild to finish. otherwise you can proceed with the existing KG2c files.

[amykglen]

status update here: I finished Plover work yesterday and today @sundareswarpullela and I will be working on the last piece, which is updating the ARAX code as needed. hoping we'll be ready to roll out at tomorrow's mini-hackathon!

[amykglen]

after I upload databases to the ITRB SFTP server, we need to make a request for ITRB to deploy KG2, ARAX, and Plover CI instances (which all correspond to master in their respective repos)

[amykglen]

all new databases and md5 sums have been uploaded to the ITRB SFTP server; just waiting on the refreshed DTD database before requesting ITRB deployments, which I believe @chunyuma is in the middle of testing.

[amykglen]

all databases have been uploaded to the ITRB SFTP server and I requested CI deployments in slack this evening.

[amykglen]

hey @edeutsch - we could probably use another roll out of master to beta/test when you have a chance. (I fixed a couple bugs - qualified edges weren't being decorated with their publications/sentences and we were losing some qualified edges due to a bug in forming edge keys)

be sure to run the database manager when you do so (a couple refreshed DTD databases have been added since our initial rollout to arax.ncats.io)

[edeutsch]

hi @amykglen okay, beginning the rollout and database managing now...

[edeutsch]

okay, the latest master branch has been rolled out and the databases have been managed for /test, /beta, /devED, /devLM
It appears to be functioning normally but additional testing welcome and advised.

[edeutsch]

@amykglen do we need to re-request a deployment to ITRB CI after this?

[amykglen]

thanks! I don't think we need to re-request, since they haven't done it yet and I only requested that they deploy master (not at a specific commit). I'll let them know today's commits should be included though

[saramsey]

Should we update the "new version of KG2 roll-out" GitHub issue template to include a checklist item for "update smartapi registry entry for RTX-KG2 KP so that it has the new version of KG2c"?

Here is the current RTX-KG2 registry entry in SmartAPI, and it appears to be 2.7.6c:

https://smart-api.info/registry?q=kg2

Tagging @acevedol @amykglen @edeutsch @sundareswarpullela

[amykglen]

@edeutsch can correct me if I'm wrong, but I don't think there's anything to update in our SmartAPI registration this time, since our TRAPI version didn't change.

there are five endpoints listed under our single RTX-KG2 SmartAPI registration - only the /beta endpoint has KG2.8.0 rolled out to it so far:

(we've requested deployment to ITRB CI, aka "staging")

[amykglen]

though it would probably be good to have a checklist item for updating our SmartAPI registration, in the event that the TRAPI version has changed!

[saramsey]

Thank you Amy! That all makes sense. Very helpful.

Second question-- when we eventually deploy RTX-KG2.8.0c to production, shouldn't this line of the Smart API yaml file get updated?

[amykglen]

yeah, I think that line has already been updated in master (here)... so I think that will automatically be updated in our registration when we roll out to the other endpoints? not positive though - @edeutsch knows for sure

[edeutsch]

yes, correct. SmartAPI checks our endpoints once per day and ingests any noticed updates, like endpoint version or Biolink version, or even API updates. The only time we need to make changes at SmartAPI is when we add/change endpoints (e.g. when the endpoint becomes .../1.4/...

[saramsey]

Thank you for educating me, @amykglen and @edeutsch . That is pretty slick (that SmartAPI does that).

[amykglen]

master has successfully been rolled out to both the ARAX and KG2 ITRB CI instances now; I updated the implementation tracker spreadsheet.

will wait to close this issue until we get all instances using the ITRB Plover instead of our self-hosted Plover (just a matter of updating config_dbs.json and testing)

[edeutsch]

outstanding, thanks!

[amykglen]

@acevedol - have you gotten a chance to upload the KG2.8.0 KGX files? or is that not a step we need to do anymore, since I seem to remember hearing KGX has been/will be deprecated?

otherwise I think we can close this issue (our self-hosted Plover has been turned off; all KG2 endpoints are using the ITRB plovers)

[acevedol]

@amykglen You are correct, KGX is no longer a required step because KGX is deprecated.

[amykglen]

great, thanks! closing