-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Roll out KG2.8.0 #1937
Comments
I updated the CName record for kg2canonicalized.rtx.ai to point to kg2-8-0c.rtx.ai I uploaded kg2c_lite_2.8.0.json.gz to LFS under the branch kg2.8.0. It's waiting on pull request approval from @amykglen because I can't push directly to main on there. |
awesome, thanks @acevedol! just merged the PR. |
Just an FYI, issue NCATSTranslator/testing 246 is indirectly requesting Biolink 3.0 support in ARAX for a specific query question ("What chemicals [qualified] affect (decreases/downregulates) a given protein/gene (COX2)?"). I am cross-listing the issue here so that we can update the |
hey @acevedol @sundareswarpullela - it sounds like Sundar finished creating the KG2.8.0c rebuild that has the patch in place for #1942 / RTXteam/RTX-KG2#250. wondering if you two could take care of loading that new KG2c into neo4j and uploading the kg2c_lite.json file to the LFS repo? |
Hey, sure, will get in touch with @acevedol and get started. |
fyi I just took care of loading the patched KG2c into Neo4j |
hey @chunyuma - are you planning to regenerate the COHD, DTD, and/or XDTD databases for KG2.8.0? if so, I believe all the files you need are on |
Hi @amykglen, I will start to do it today. Thanks for letting me know all data of KG2.8.0 is ready. |
sounds good - and just fyi, shortly after posting my last comment above I realized that the bug we thought we had patched where some edges were missing so if you need to use the new |
status update here: I finished Plover work yesterday and today @sundareswarpullela and I will be working on the last piece, which is updating the ARAX code as needed. hoping we'll be ready to roll out at tomorrow's mini-hackathon! |
after I upload databases to the ITRB SFTP server, we need to make a request for ITRB to deploy KG2, ARAX, and Plover CI instances (which all correspond to |
all new databases and md5 sums have been uploaded to the ITRB SFTP server; just waiting on the refreshed DTD database before requesting ITRB deployments, which I believe @chunyuma is in the middle of testing. |
all databases have been uploaded to the ITRB SFTP server and I requested CI deployments in slack this evening. |
hey @edeutsch - we could probably use another roll out of be sure to run the database manager when you do so (a couple refreshed DTD databases have been added since our initial rollout to arax.ncats.io) |
hi @amykglen okay, beginning the rollout and database managing now... |
okay, the latest master branch has been rolled out and the databases have been managed for /test, /beta, /devED, /devLM |
@amykglen do we need to re-request a deployment to ITRB CI after this? |
thanks! I don't think we need to re-request, since they haven't done it yet and I only requested that they deploy |
Should we update the "new version of KG2 roll-out" GitHub issue template to include a checklist item for "update smartapi registry entry for RTX-KG2 KP so that it has the new version of KG2c"? Here is the current RTX-KG2 registry entry in SmartAPI, and it appears to be 2.7.6c: |
@edeutsch can correct me if I'm wrong, but I don't think there's anything to update in our SmartAPI registration this time, since our TRAPI version didn't change. there are five endpoints listed under our single RTX-KG2 SmartAPI registration - only the /beta endpoint has KG2.8.0 rolled out to it so far: (we've requested deployment to ITRB CI, aka "staging") |
though it would probably be good to have a checklist item for updating our SmartAPI registration, in the event that the TRAPI version has changed! |
yes, correct. SmartAPI checks our endpoints once per day and ingests any noticed updates, like endpoint version or Biolink version, or even API updates. The only time we need to make changes at SmartAPI is when we add/change endpoints (e.g. when the endpoint becomes .../1.4/... |
will wait to close this issue until we get all instances using the ITRB Plover instead of our self-hosted Plover (just a matter of updating |
outstanding, thanks! |
@acevedol - have you gotten a chance to upload the KG2.8.0 KGX files? or is that not a step we need to do anymore, since I seem to remember hearing KGX has been/will be deprecated? otherwise I think we can close this issue (our self-hosted Plover has been turned off; all KG2 endpoints are using the ITRB plovers) |
@amykglen You are correct, KGX is no longer a required step because KGX is deprecated. |
great, thanks! closing |
NOTE: To create a new issue based on this template, simply go to: https://github.com/RTXteam/RTX/issues/new?template=kg2rollout.md
1. Build and load KG2c:
master
into thekg2integration
branchkg2integration
branch (as needed):buildkg2c.rtx.ai
from thekg2integration
branch (how-to is here)arax.ncats.io
(need at least 110G, ideally >130G). delete old KG2 database directories from/data/orangeboard/databases
as needed (warn the team on Slack in advance).kg2c_config.json
, as described in the how-toRTX:KG2c
) has the expected version number (KG2.X.Y)node_synonymizer.sqlite
should be around 15-20 GBnode_synonymizer.sqlite
's last modified date is today (or whatever day the build was run)kg2c_lite.json.gz
's last modified date is today (or whatever day the build was run)arax-databases.rtx.ai
(/home/rtxconfig/KG2.X.Y
)match (n {id:"RTX:KG2c"}) return n
scp /home/ubuntu/kg2-build/kg2c.dump rtxconfig@arax-databases.rtx.ai:/home/rtxconfig/KG2.X.Y/extra_files
RTX/code/config_dbs.json
(push tokg2integration
branch)kg2c_lite_2.X.Y.json.gz
file to the translator-lfs-artifacts repoconfig_dbs.json
to point to this new Plover (all maturity levels should point to it for now)2. Rebuild downstream databases:
The following databases should be rebuilt and copies of them should be put in
/home/rtxconfig/KG2.X.Y
onarax-databases.rtx.ai
. Please use this kind of naming format:mydatabase_v1.0_KG2.X.Y.sqlite
.NOTE: As databases are rebuilt,
RTX/code/config_dbs.json
will need to be updated to point to their new paths! Push these changes to thekg2integration
branch, unless the rollout of this KG2 version has already occurred, in which case you should push tomaster
(but first follow the steps described here).3. Update the ARAX codebase:
All code changes should go in the
kg2integration
branch!kg2integration
branch and pull to get the latest changes (this is important for ensuring the correct KG2c Neo4j is used)RTX/code/ARAX/KnowledgeSources/RTX_KG2c_test_triples.json
(in thekg2integration
branch)kg2integration
branch and pull to get the latest changesforce_local = True
inARAX_expander.py
(to avoid using the old KG2 API)openapi.yaml
, etc.) @edeutschRTX/code/UI/OpenAPI/python-flask-server/openapi_server/openapi/openapi.yaml
in line 12 (version:
); the major and minor release numbers are kept synchronous with the TRAPI version; just bump the patch release version (least significant digit)RTX/code/UI/OpenAPI/python-flask-server/KG2/openapi_server/openapi/openapi.yaml
in line 12 (version:
); the first three digits are kept synchronous with the KG2 release versionRTX/code/UI/OpenAPI/python-flask-server/RTX_OA3_TRAPI1.3_ARAX.yaml
on line 4 (version:
); same as for the ARAXopenapi.yaml
fileRTX/code/UI/OpenAPI/python-flask-server/RTX_OA3_TRAPI1.3_KG2.yaml
on line 4 (version:
); same as for the KG2openapi.yaml
file4. Do the rollout:
arax.ncats.io
(under data/orangeboard/databases/KG2.X.Y; first verify there's enough space)config_dbs.json
) to ITRB's SFTP server using the steps detailed heremaster
intokg2integration
kg2integration
intomaster
master
out to the variousarax.ncats.io
endpoints{"nodes": {"n00": {"ids": ["RTX:KG2c"]}}, "edges": {}}
ssh ubuntu@cicd.rtx.ai
cd RTX
git pull origin master
python3 code/ARAX/ARAXQuery/ARAX_database_manager.py --mnt --skip-if-exists --remove_unused
5. Final items/clean up:
kg_config.json
in themain
branch of the Plover repo to point to the newkg2c_lite_2.X.Y.json.gz
file (push this change)config_dbs.json
and setforce_local = True
in Expand)config_dbs.json
inmaster
to point to the ITRB Plover endpoints (all maturity levels)master
out to the various endpoints on arax.ncats.ioThe text was updated successfully, but these errors were encountered: