Skip to content
Permalink
Browse files

Dm harmoinzation (#49)

* Edits to git repo folder org (#45)

* 2 edit file names (#46)

* Edits to git repo folder org

* Edits made to json schema file names

* Changed micsss_assay template to schema 7

* 3 assay core creation (#47)

* Edits to git repo folder org

* Edits made to json schema file names

* Created an assay core schema

* 4 json reference edits (#48)

* Edits to git repo folder org

* Edits made to json schema file names

* Created an assay core schema

* Edit references within json schemas

* Additional changes to file references

* re-applied changes after botched merge

* re-generated docs
  • Loading branch information...
jim-bo committed Jun 5, 2019
1 parent d5e7251 commit c9c71b5c8bcd353ed3c20cf97251b9dee33597a1
Showing with 7,867 additions and 325 deletions.
  1. +28 −21 README.md
  2. +87 −39 docs/docs/index.html
  3. +333 −0 docs/docs/schemas.artifacts.artifact.html
  4. +143 −0 docs/docs/schemas.assays.assay_core.html
  5. +20 −0 docs/docs/schemas.assays.components.image.html
  6. +474 −0 docs/docs/schemas.assays.components.micsss_antibody.html
  7. +362 −0 docs/docs/schemas.assays.micsss_assay.html
  8. +342 −0 docs/docs/schemas.assays.mif_assay.html
  9. +437 −0 docs/docs/schemas.assays.ngs_assay.html
  10. +427 −0 docs/docs/schemas.assays.olink_assay.html
  11. +168 −0 docs/docs/schemas.assays.rna_expression_assay.html
  12. +138 −0 docs/docs/schemas.assays.wes_assay.html
  13. +826 −0 docs/docs/schemas.templates.analyte_template.html
  14. +809 −0 docs/docs/schemas.templates.hne_template.html
  15. +85 −0 docs/docs/schemas.templates.metadata.mIF_template.html
  16. +85 −0 docs/docs/schemas.templates.metadata.micsss_template.html
  17. +86 −0 docs/docs/schemas.templates.metadata.olink_template.html
  18. +86 −0 docs/docs/schemas.templates.metadata.rna_expression_template.html
  19. +86 −0 docs/docs/schemas.templates.metadata.wes_template.html
  20. +877 −0 docs/docs/schemas.templates.pbmc_template.html
  21. +843 −0 docs/docs/schemas.templates.plasma_template.html
  22. +775 −0 docs/docs/schemas.templates.whole_blood_template.html
  23. 0 schemas/{ → artifacts}/artifact.json
  24. +27 −0 schemas/assays/assay_core.json
  25. 0 schemas/assays/{micsss.json → micsss_assay.json}
  26. 0 schemas/assays/{mif.json → mif_assay.json}
  27. +40 −16 schemas/{ngs_artifact.json → assays/ngs_assay.json}
  28. +95 −0 schemas/assays/olink_assay.json
  29. +6 −6 schemas/{rna_expression_artifact.json → assays/rna_expression_assay.json}
  30. +5 −10 schemas/{wes_artifact.json → assays/wes_assay.json}
  31. +0 −90 schemas/olink_artifact.json
  32. 0 schemas/templates/{analyte.json → analyte_template.json}
  33. 0 schemas/templates/{hne.json → hne_template.json}
  34. +15 −6 schemas/templates/{ → metadata}/mIF_template.json
  35. +17 −8 schemas/templates/{ → metadata}/micsss_template.json
  36. +89 −0 schemas/templates/metadata/olink_template.json
  37. +28 −25 schemas/templates/{rna_expression.json → metadata/rna_expression_template.json}
  38. +26 −23 schemas/templates/{wes.json → metadata/wes_template.json}
  39. +0 −79 schemas/templates/olink.json
  40. 0 schemas/templates/{pbmc.json → pbmc_template.json}
  41. 0 schemas/templates/{plasma.json → plasma_template.json}
  42. 0 schemas/templates/{whole_blood.json → whole_blood_template.json}
  43. BIN template_examples/analyte.xlsx
  44. BIN template_examples/analyte_template.xlsx
  45. BIN template_examples/hne.xlsx
  46. BIN template_examples/hne_template.xlsx
  47. BIN template_examples/mIF_template.xlsx
  48. BIN template_examples/micsss_template.xlsx
  49. BIN template_examples/olink.xlsx
  50. BIN template_examples/olink_template.xlsx
  51. BIN template_examples/pbmc.xlsx
  52. BIN template_examples/pbmc_template.xlsx
  53. BIN template_examples/plasma.xlsx
  54. BIN template_examples/plasma_template.xlsx
  55. BIN template_examples/rna_expression.xlsx
  56. BIN template_examples/rna_expression_template.xlsx
  57. BIN template_examples/tissue.xlsx
  58. BIN template_examples/wes.xlsx
  59. BIN template_examples/wes_template.xlsx
  60. BIN template_examples/whole_blood.xlsx
  61. BIN template_examples/whole_blood_template.xlsx
  62. +1 −1 tests/conftest.py
  63. +1 −1 tests/test_template_reader.py
@@ -77,10 +77,11 @@ cidc_schemas generate_template -m schemas/templates/pbmc.json -s schemas -o pbmc
Check that a populated template file is valid with respect to a template specification.

```bash
cidc_schemas validate_template -m schemas/templates/pbmc.json -s schemas -x template_examples/pbmc.xlsx
cidc_schemas validate_template -m schemas/templates/pbmc.json -s schemas -x template_examples/pbmc_template.xlsx
```

### Validate JSON schemas

Check that a JSON schema conforms to the JSON Schema specifications.

```bash
@@ -100,41 +101,45 @@ cidc_schemas convert --to_json <some_yaml_file>
### Workflow

A new manifest or assay template has been "added" to the repository once these three things are true:
* A file `schemas/templates/<TEMPLATE NAME>.json` exists specifying the template schema.
* A file `template_examples/<TEMPLATE NAME>.xlsx` exists containing a populated example Excel template corresponding to the template schema.
* Running `pytest tests/test_templates.py` generates no errors related to this template.

- A file `schemas/templates/<TEMPLATE NAME>.json` exists specifying the template schema.
- A file `template_examples/<TEMPLATE NAME>.xlsx` exists containing a populated example Excel template corresponding to the template schema.
- Running `pytest tests/test_templates.py` generates no errors related to this template.

Here's the recommended workflow for achieving those three things:

1) Create a new git branch and switch to it (naming your branch something like `template-dev-<TEMPLATE NAME>`):
1. Create a new git branch and switch to it (naming your branch something like `template-dev-<TEMPLATE NAME>`):
```bash
git checkout -b <YOUR BRANCH NAME>
```
2) On this branch, create a new template schema called `<TEMPLATE NAME>.json` in the `schemas/templates` directory. See the template schema structure section below for guidance.
3) Use the CLI to generate an empty Excel template from your schema, and visually verify that the generated template accords with your expectations. Iteratively edit the schema and regenerate the Excel template until you are satisfied.
4) Fill out the generated Excel template with some valid sample values, and place that file in `template_examples` with the name `<TEMPLATE NAME>.xlsx`.
- **Note**: by this point, you should have created two files:
1) `schemas/templates/<TEMPLATE NAME>.json`
2) `template_examples/<TEMPLATE NAME>.xlsx`
5) Ensure that `pytest tests/test_templates.py` raises no errors related to this template.
6) Commit and push your changes.
2. On this branch, create a new template schema called `<TEMPLATE NAME>.json` in the `schemas/templates` directory. See the template schema structure section below for guidance.
3. Use the CLI to generate an empty Excel template from your schema, and visually verify that the generated template accords with your expectations. Iteratively edit the schema and regenerate the Excel template until you are satisfied.
4. Fill out the generated Excel template with some valid sample values, and place that file in `template_examples` with the name `<TEMPLATE NAME>.xlsx`.
- **Note**: by this point, you should have created two files:
1. `schemas/templates/<TEMPLATE NAME>.json`
2. `template_examples/<TEMPLATE NAME>.xlsx`
5. Ensure that `pytest tests/test_templates.py` raises no errors related to this template.
6. Commit and push your changes.
```bash
# Add the two files you've created
git add schemas/templates/<TEMPLATE NAME>.json template_examples/<TEMPLATE NAME>.xlsx
git commit -m "Added template for <TEMPLATE NAME>"
git push -u origin <YOUR BRANCH NAME>
```
7) Navigate to GitHub and create a pull request. Get feedback on your template.
8) Once your pull request is approved, merge your changes into master. All done!
7. Navigate to GitHub and create a pull request. Get feedback on your template.
8. Once your pull request is approved, merge your changes into master. All done!

### Template Schema Structure

The current template generator can create empty Excel workbooks with arbitrarily many worksheets in them from JSON schemas. Every worksheet in the template has the same high level structure made up of two sections:
1) **`preamble_rows`**: a set of key-value rows appearing at the top of the worksheet. This is a good place to have template users input data that applies to, e.g., the entire batch of samples.
2) **`data_columns`**: a data table that appears below `preamble_rows` in each worksheet, containing data headers with columns beneath them with multiple data entries for each header. Data columns are grouped into subtables, where a set of column headers will have one shared header above them (e.g., the shared header "Filled by Biorepository" should appear above all data columns that the biorepository will fill out). This is a good place to have template users input data that will be different for, e.g., each sample.

1. **`preamble_rows`**: a set of key-value rows appearing at the top of the worksheet. This is a good place to have template users input data that applies to, e.g., the entire batch of samples.
2. **`data_columns`**: a data table that appears below `preamble_rows` in each worksheet, containing data headers with columns beneath them with multiple data entries for each header. Data columns are grouped into subtables, where a set of column headers will have one shared header above them (e.g., the shared header "Filled by Biorepository" should appear above all data columns that the biorepository will fill out). This is a good place to have template users input data that will be different for, e.g., each sample.

**Note**: Either of these sections can be omitted from a given worksheet.

The template generator expects JSON schemas with the following structure:

```
{
"$schema": "http://json-schema.org/draft-07/schema#",
@@ -166,8 +171,10 @@ The template generator expects JSON schemas with the following structure:
}
}
```

Template elements:
* **`<worksheet name>`**: the name for the worksheet as it will appear in Excel.
* **`<field name>`**: a human-readable label for a template field as it should appear in the spreadsheet. In the preamble section, these will be row keys. In the data section, these will be column headers.
* **`<value schema>`**: a reference to a property in another JSON schema in the data model, or a custom JSON schema. The `<value schema>`'s `"type"` and `"format"` properties will be used for cell data validation in the generated Excel spreadsheet.
* **`<subtable header>`**: A shared header that appears in a merged cell above the data column headers specified as properties of this subtable. `<subtable header>`s are only valid as a property of the `"data_columns"` section of the template schema.

- **`<worksheet name>`**: the name for the worksheet as it will appear in Excel.
- **`<field name>`**: a human-readable label for a template field as it should appear in the spreadsheet. In the preamble section, these will be row keys. In the data section, these will be column headers.
- **`<value schema>`**: a reference to a property in another JSON schema in the data model, or a custom JSON schema. The `<value schema>`'s `"type"` and `"format"` properties will be used for cell data validation in the generated Excel spreadsheet.
- **`<subtable header>`**: A shared header that appears in a merged cell above the data column headers specified as properties of this subtable. `<subtable header>`s are only valid as a property of the `"data_columns"` section of the template schema.
@@ -51,31 +51,11 @@ <h5>Schemas</h5>
<td>Pertaining to a portion (volume or weight) of the whole.</td>
</tr>

<tr>
<td width=25%><a href='schemas.rna_expression_artifact.html'>RNA Gene Expression Artifact</a></td>
<td>Additional properties to describe a RNA Artifact for expression analysis.</td>
</tr>

<tr>
<td width=25%><a href='schemas.wes_artifact.html'>Whole Exome Sequencing Artifact</a></td>
<td>Additional properties to describe a Whole Exome Sequencing Artifact.</td>
</tr>

<tr>
<td width=25%><a href='schemas.shipping_core.html'>Core Shipping Details</a></td>
<td>Core shipping details for a manifest.</td>
</tr>

<tr>
<td width=25%><a href='schemas.olink_artifact.html'>olink Artifact</a></td>
<td>Additional properties to describe a olink Artifact.</td>
</tr>

<tr>
<td width=25%><a href='schemas.ngs_artifact.html'>NGS Sequencing Artifact</a></td>
<td>Additional properties to describe a NGS Sequencing Artifact.</td>
</tr>

<tr>
<td width=25%><a href='schemas.user.html'>User</a></td>
<td>Any user of the CIDC platform.</td>
@@ -104,8 +84,21 @@ <h5>Schemas</h5>
</td>
</tr>

</table>

<h5>Schemas > Artifacts</h5>
<p>

<table class="table table-striped">
<thead>
<tr>
<th>Entity</th>
<th>Description</th>
</tr>
</thead>

<tr>
<td width=25%><a href='schemas.artifact.html'>Artifact</a></td>
<td width=25%><a href='schemas.artifacts.artifact.html'>Artifact</a></td>
<td>Any file artifact associated with a clinical trial.</td>
</tr>

@@ -123,11 +116,42 @@ <h5>Schemas > Assays</h5>
</thead>

<tr>
<td width=25%><a href='schemas.assays.mif.html'>MIF</a></td>
<td width=25%><a href='schemas.assays.micsss_assay.html'>MICSSS</a></td>
<td>Base information about the aquisition and analysis of a quantitative pathology image
</td>
</tr>

<tr>
<td width=25%><a href='schemas.assays.ngs_assay.html'>NGS Sequencing Assay</a></td>
<td>Additional properties to describe a NGS Sequencing Experiment.</td>
</tr>

<tr>
<td width=25%><a href='schemas.assays.wes_assay.html'>Whole Exome Sequencing Assay</a></td>
<td>Additional properties to describe a Whole Exome Sequencing Assay.</td>
</tr>

<tr>
<td width=25%><a href='schemas.assays.mif_assay.html'>MIF</a></td>
<td>Base information about the aquisition and analysis of a quantitative pathology image
</td>
</tr>

<tr>
<td width=25%><a href='schemas.assays.assay_core.html'>Assay Core</a></td>
<td>Main properties of the assay experiment template</td>
</tr>

<tr>
<td width=25%><a href='schemas.assays.olink_assay.html'>olink Assay</a></td>
<td>Additional properties to describe an olink Experiment.</td>
</tr>

<tr>
<td width=25%><a href='schemas.assays.rna_expression_assay.html'>RNA Gene Expression Assay</a></td>
<td>Additional properties to describe a RNA assay for expression analysis.</td>
</tr>

</table>

<h5>Schemas > Assays > Components</h5>
@@ -147,6 +171,12 @@ <h5>Schemas > Assays > Components</h5>
</td>
</tr>

<tr>
<td width=25%><a href='schemas.assays.components.micsss_antibody.html'>Multiplexed immunohistochemical consecutive staining on single slide</a></td>
<td>Data specific to antibody prepration on the multiplex immunohistochemical platform
</td>
</tr>

<tr>
<td width=25%><a href='schemas.assays.components.image.html'>Image</a></td>
<td>Base information about the aquisition and analysis of a quantitative pathology image
@@ -178,58 +208,76 @@ <h5>Schemas > Templates</h5>
</thead>

<tr>
<td width=25%><a href='schemas.templates.wes.html'>Metadata file for WES</a></td>
<td>Metadata information for whole exome sequencing analysis.
<td width=25%><a href='schemas.templates.hne_template.html'>Shipping manifest for H&E slides</a></td>
<td>Shipping / receiving manifest for transferring H&E slides.
</td>
</tr>

<tr>
<td width=25%><a href='schemas.templates.olink.html'>Metadata file for olink</a></td>
<td>Metadata information for olink analysis.
<td width=25%><a href='schemas.templates.whole_blood_template.html'>Shipping manifest for Whole Blood</a></td>
<td>Shipping / receiving manifest for transferring Whole Blood.
</td>
</tr>

<tr>
<td width=25%><a href='schemas.templates.rna_expression.html'>Metadata file for RNA expression</a></td>
<td>Metadata information for rna expression analysis.
<td width=25%><a href='schemas.templates.pbmc_template.html'>Shipping manifest for PBMC samples</a></td>
<td>Shipping / receiving manifest for transferring PBMC samples.
</td>
</tr>

<tr>
<td width=25%><a href='schemas.templates.plasma.html'>Shipping manifest for Plasma samples</a></td>
<td width=25%><a href='schemas.templates.plasma_template.html'>Shipping manifest for Plasma samples</a></td>
<td>Shipping / receiving manifest for transferring Plasma samples.
</td>
</tr>

<tr>
<td width=25%><a href='schemas.templates.hne.html'>Shipping manifest for H&E slides</a></td>
<td>Shipping / receiving manifest for transferring H&E slides.
<td width=25%><a href='schemas.templates.analyte_template.html'>Shipping manifest for Analyte</a></td>
<td>Shipping / receiving manifest for transferring Aanlyte.
</td>
</tr>

</table>

<h5>Schemas > Templates > Metadata</h5>
<p>

<table class="table table-striped">
<thead>
<tr>
<td width=25%><a href='schemas.templates.analyte.html'>Shipping manifest for Analyte</a></td>
<td>Shipping / receiving manifest for transferring Aanlyte.
</td>
<th>Entity</th>
<th>Description</th>
</tr>
</thead>

<tr>
<td width=25%><a href='schemas.templates.whole_blood.html'>Shipping manifest for Whole Blood</a></td>
<td>Shipping / receiving manifest for transferring Whole Blood.
<td width=25%><a href='schemas.templates.metadata.micsss_template.html'>Metadata file for micsss</a></td>
<td>Metadata information for multiplexed immunohistochemical consecutive staining on single slide.</td>
</tr>

<tr>
<td width=25%><a href='schemas.templates.metadata.olink_template.html'>Metadata file for olink</a></td>
<td>Metadata information for olink analysis.
</td>
</tr>

<tr>
<td width=25%><a href='schemas.templates.pbmc.html'>Shipping manifest for PBMC samples</a></td>
<td>Shipping / receiving manifest for transferring PBMC samples.
<td width=25%><a href='schemas.templates.metadata.wes_template.html'>Metadata file for WES</a></td>
<td>Metadata information for whole exome sequencing analysis.
</td>
</tr>

<tr>
<td width=25%><a href='schemas.templates.mIF_template.html'>Metadata file for mIF</a></td>
<td width=25%><a href='schemas.templates.metadata.mIF_template.html'>Metadata file for mIF</a></td>
<td>Metadata information for multiplex immunoflouresence(mIF).</td>
</tr>

<tr>
<td width=25%><a href='schemas.templates.metadata.rna_expression_template.html'>Metadata file for RNA expression</a></td>
<td>Metadata information for rna expression analysis.
</td>
</tr>

</table>

</div>

0 comments on commit c9c71b5

Please sign in to comment.
You can’t perform that action at this time.