Patching CEDAR Resources

The Workflow

NOTE: Activate Python 3.4 by typing the command $ source activate py34 in the target server

1. Create a MongoDB backup.

Go to cedar-util module (goutil) and switch the directory to script/mongodb where you will find a script called mongo-rename.py. Follow the instruction below:

(py34)$ python mongo-rename.py

Please choose an option:
1. List my MongoDB databases
2. Create a copy of an existing database
3. Swap the content of two databases
4. Exit

>> 2

Enter the name of the DB that you want to copy:
>> cedar

Enter the name of the new database (copy of 'cedar'):
>> cedar-copy-2018-09-06

Please choose an option:
1. List my MongoDB databases
2. Create a copy of an existing database
3. Swap the content of two databases
4. Exit

>> 4

2. Run the cedar-validator.py and get the validation report.

The command below gives you and example of running the validation script to check all the templates.

(py34)$ pwd
/srv/cedar/cedar-util/scripts
(py34)$ python cedar-validator.py -t template --input-mongodb=cedar > /tmp/template-report.txt

3. Analyze the report and create a list of resource ids that are invalid.

a) At this point, you can send the report from the server to your local computer through the mail application to make the report processing easier. This following step is optional if you are comfortable working on the server machine.

$ mail -s "Validation report" -a /tmp/template-report.txt your.email@stanford.edu
(Press CRTL+D to send)

b) The validation report will contain a progress bar graphic, which you will ignore, and the validation error summary. As an example, you might see some text like below:

...
Validating (242/245): |##########################################################--| 98% Complete [f7b40a2f]
Validating (243/245): |###########################################################-| 99% Complete [f7b8546f]
Validating (244/245): |###########################################################-| 99% Complete [fcd26b1d]
Validating (245/245): |############################################################| 100% Complete [fd1c7f55]
Found 1 validation error(s).
Details: {
  "object has missing required properties (['bibo:status','pav:version']) at /": [
    "https://repo.staging.metadatacenter.net/templates/02edc290-831b-4d6e-bd24-01a2dd330b2a",
    "https://repo.staging.metadatacenter.net/templates/04596ef2-cd54-444a-a4ad-4b7bc3cf4c80",
    "https://repo.staging.metadatacenter.net/templates/055eea88-766c-4f8c-8d30-69b772e7800a",
    "https://repo.staging.metadatacenter.net/templates/06da62ac-cb85-4f7e-a833-dc9742f5132a",
    "https://repo.staging.metadatacenter.net/templates/0869f17e-1ef5-477f-8b91-d9e8dd9ff355",
...

c) Get rid all the progress bar graphics and keep the resource ids under the error messages. What you would like to produce is a list of resource ids as illustrated below:

https://repo.staging.metadatacenter.net/templates/02edc290-831b-4d6e-bd24-01a2dd330b2a
https://repo.staging.metadatacenter.net/templates/04596ef2-cd54-444a-a4ad-4b7bc3cf4c80
https://repo.staging.metadatacenter.net/templates/055eea88-766c-4f8c-8d30-69b772e7800a
https://repo.staging.metadatacenter.net/templates/06da62ac-cb85-4f7e-a833-dc9742f5132a
https://repo.staging.metadatacenter.net/templates/0869f17e-1ef5-477f-8b91-d9e8dd9ff355
...

d) Copy the list from your local computer and paste it back to the server by creating an empty file (e.g., /tmp/template.list)

4. Run cedar-patch.py to fix all those invalid resources

The command below gives you an example of running the patch script to fix all the templates in the staging server by taking into account the invalid templates in /tmp/template.list and storing the fixed templates to /tmp/template-patches

(py34)$ pwd
/srv/cedar/cedar-util/scripts
(py34)$ python cedar-patch.py -t template \
--input-mongodb=cedar --filter=/tmp/template.list \
--output-dir=/tmp/template-patches 

5. Commit the fixes to MongoDB database

The next step is to commit the patched resources to the backend system. The command below is an example of committing all the fixed templates in the /tmp/template-patches directory that start with "template-patched-" in the file name.

(py34)$ pwd
/srv/cedar/cedar-util/scripts
(py34)$ python commit-patch.py -t template \
--patch-dir=/tmp/template-patches \
--patch-list=/tmp/template.list  \
--input-mongodb=cedar \
--use-patches-with-prefix="template-patched-"