Merge pull request #158 from 4dn-dcic/use_openpyxl

Use openpyxl
4dn-dcic · May 3, 2022 · 1455635 · 1455635
2 parents c7a3f81 + d91d330
commit 1455635
Show file tree

Hide file tree

Showing 89 changed files with 438 additions and 465 deletions.
diff --git a/Data_Files/MicroscopyCalibration/1calibration.xls b/Data_Files/MicroscopyCalibration/1calibration.xls
diff --git a/Data_Files/MicroscopyCalibration/1calibration_Release.xls b/Data_Files/MicroscopyCalibration/1calibration_Release.xls
diff --git a/Data_Files/Rao_et_al_2014/Protocols/DilutionHiC.docx b/Data_Files/Rao_et_al_2014/Protocols/DilutionHiC.docx
diff --git a/Data_Files/Rao_et_al_2014/Protocols/DilutionHiC.pdf b/Data_Files/Rao_et_al_2014/Protocols/DilutionHiC.pdf
diff --git a/Data_Files/Rao_et_al_2014/Protocols/In situ HiC.docx b/Data_Files/Rao_et_al_2014/Protocols/In situ HiC.docx
diff --git a/Data_Files/Rao_et_al_2014/Protocols/In situ HiC_3day.docx b/Data_Files/Rao_et_al_2014/Protocols/In situ HiC_3day.docx
diff --git a/Data_Files/Rao_et_al_2014/Protocols/In situ HiC_agar.docx b/Data_Files/Rao_et_al_2014/Protocols/In situ HiC_agar.docx
diff --git a/Data_Files/Rao_et_al_2014/Protocols/In situ HiC_sn_pellet.docx b/Data_Files/Rao_et_al_2014/Protocols/In situ HiC_sn_pellet.docx
diff --git a/Data_Files/Rao_et_al_2014/Protocols/In situ HiC_tethered.docx b/Data_Files/Rao_et_al_2014/Protocols/In situ HiC_tethered.docx
diff --git a/Data_Files/Rao_et_al_2014/Protocols/In situ HiC_wo_crosslink.docx b/Data_Files/Rao_et_al_2014/Protocols/In situ HiC_wo_crosslink.docx
diff --git a/Data_Files/Rao_et_al_2014/Protocols/InsituHiC.pdf b/Data_Files/Rao_et_al_2014/Protocols/InsituHiC.pdf
diff --git a/Data_Files/Rao_et_al_2014/Protocols/InsituHiC_3day.pdf b/Data_Files/Rao_et_al_2014/Protocols/InsituHiC_3day.pdf
diff --git a/Data_Files/Rao_et_al_2014/Protocols/InsituHiC_agar.pdf b/Data_Files/Rao_et_al_2014/Protocols/InsituHiC_agar.pdf
diff --git a/Data_Files/Rao_et_al_2014/Protocols/InsituHiC_sn_pellet.pdf b/Data_Files/Rao_et_al_2014/Protocols/InsituHiC_sn_pellet.pdf
diff --git a/Data_Files/Rao_et_al_2014/Protocols/InsituHiC_tethered.pdf b/Data_Files/Rao_et_al_2014/Protocols/InsituHiC_tethered.pdf
diff --git a/Data_Files/Rao_et_al_2014/Protocols/InsituHiC_wo_crosslink.pdf b/Data_Files/Rao_et_al_2014/Protocols/InsituHiC_wo_crosslink.pdf
diff --git a/Data_Files/Rao_et_al_2014/Protocols/b5964093-eb15-4ab3-bd7f-0135b5c7d7c6.png b/Data_Files/Rao_et_al_2014/Protocols/b5964093-eb15-4ab3-bd7f-0135b5c7d7c6.png
diff --git a/Data_Files/Rao_et_al_2014/Protocols/the_most_interesting_coder_in_the_world.png b/Data_Files/Rao_et_al_2014/Protocols/the_most_interesting_coder_in_the_world.png
diff --git a/Data_Files/Rao_et_al_2014/RaoRepSetDiff.xlsx b/Data_Files/Rao_et_al_2014/RaoRepSetDiff.xlsx
diff --git a/Data_Files/Rao_et_al_2014/del_test.xlsx b/Data_Files/Rao_et_al_2014/del_test.xlsx
diff --git a/Data_Files/Rao_et_al_2014/fieldsJinSelvaraj.xls b/Data_Files/Rao_et_al_2014/fieldsJinSelvaraj.xls
diff --git a/Data_Files/Rao_et_al_2014/fieldsJinSelvaraj_release.xls b/Data_Files/Rao_et_al_2014/fieldsJinSelvaraj_release.xls
diff --git a/Data_Files/Rao_et_al_2014/fieldsRao.xls b/Data_Files/Rao_et_al_2014/fieldsRao.xls
diff --git a/Data_Files/Rao_et_al_2014/fieldsRaoRelease.xls b/Data_Files/Rao_et_al_2014/fieldsRaoRelease.xls
diff --git a/Data_Files/Rao_et_al_2014/testing_images.xlsx b/Data_Files/Rao_et_al_2014/testing_images.xlsx
diff --git a/Data_Files/SOPs/Documents/4DN_F121-9_SOP_170314.pdf b/Data_Files/SOPs/Documents/4DN_F121-9_SOP_170314.pdf
diff --git a/Data_Files/SOPs/Documents/4DN_F123_F121-9_MEF_KOSR_SOP_161101.pdf b/Data_Files/SOPs/Documents/4DN_F123_F121-9_MEF_KOSR_SOP_161101.pdf
diff --git a/Data_Files/SOPs/Documents/4DN_GM12878_SOP_170314.pdf b/Data_Files/SOPs/Documents/4DN_GM12878_SOP_170314.pdf
diff --git a/Data_Files/SOPs/Documents/4DN_H1_SOP_170305.pdf b/Data_Files/SOPs/Documents/4DN_H1_SOP_170305.pdf
diff --git a/Data_Files/SOPs/Documents/4DN_HAP1_SOP_170314.pdf b/Data_Files/SOPs/Documents/4DN_HAP1_SOP_170314.pdf
diff --git a/Data_Files/SOPs/Documents/4DN_HEK293_SOP_170314.pdf b/Data_Files/SOPs/Documents/4DN_HEK293_SOP_170314.pdf
diff --git a/Data_Files/SOPs/Documents/4DN_HFFc6_SOP_161216.pdf b/Data_Files/SOPs/Documents/4DN_HFFc6_SOP_161216.pdf
diff --git a/Data_Files/SOPs/Documents/4DN_IMR90_SOP_161101.pdf b/Data_Files/SOPs/Documents/4DN_IMR90_SOP_161101.pdf
diff --git a/Data_Files/SOPs/Documents/4DN_K562_SOP_170314.pdf b/Data_Files/SOPs/Documents/4DN_K562_SOP_170314.pdf
diff --git a/Data_Files/SOPs/Documents/4DN_U2OS_SOP_170314.pdf b/Data_Files/SOPs/Documents/4DN_U2OS_SOP_170314.pdf
diff --git a/Data_Files/SOPs/Documents/4DN_WTC-11_SOP_150309.pdf b/Data_Files/SOPs/Documents/4DN_WTC-11_SOP_150309.pdf
diff --git a/Data_Files/SOPs/Documents/4DN_hTERT-RPE_SOP_170314.pdf b/Data_Files/SOPs/Documents/4DN_hTERT-RPE_SOP_170314.pdf
diff --git a/Data_Files/SOPs/cell_lines_and_SOPs_170522.xls b/Data_Files/SOPs/cell_lines_and_SOPs_170522.xls
diff --git a/Data_Files/Sanborn_et_al_2015/Sanborn_2015_metadata_170712.xls b/Data_Files/Sanborn_et_al_2015/Sanborn_2015_metadata_170712.xls
diff --git a/Data_Files/Sanborn_et_al_2015/Sanborn_FASTA_upload.xls b/Data_Files/Sanborn_et_al_2015/Sanborn_FASTA_upload.xls
diff --git a/Data_Files/Submission Pack/Example_rao_etal2014.xls b/Data_Files/Submission Pack/Example_rao_etal2014.xls
diff --git a/Data_Files/Submission Pack/Metadata_entry_form.xls b/Data_Files/Submission Pack/Metadata_entry_form.xls
diff --git a/Data_Files/Submission Pack/Metadata_entry_form_V2.xls b/Data_Files/Submission Pack/Metadata_entry_form_V2.xls
diff --git a/Data_Files/Submission Pack/Metadata_entry_form_V3.xls b/Data_Files/Submission Pack/Metadata_entry_form_V3.xls
diff --git a/Data_Files/Submission Pack/keypairs.json b/Data_Files/Submission Pack/keypairs.json
diff --git a/Data_Files/reference_files/file_reference.json b/Data_Files/reference_files/file_reference.json
diff --git a/Data_Files/reference_files/file_reference2.json b/Data_Files/reference_files/file_reference2.json
diff --git a/Data_Files/workflow_mapping/workflow_mapping_combined.json b/Data_Files/workflow_mapping/workflow_mapping_combined.json
diff --git a/Data_Files/workflows/combine_workflow.json b/Data_Files/workflows/combine_workflow.json
diff --git a/Data_Files/workflows/workflow.json b/Data_Files/workflows/workflow.json
diff --git a/Data_Files/workflows/workflow2.json b/Data_Files/workflows/workflow2.json
diff --git a/Data_Files/workflows/workflow3.json b/Data_Files/workflows/workflow3.json
diff --git a/Data_Files/workflows/workflow4.json b/Data_Files/workflows/workflow4.json
diff --git a/README.md b/README.md
@@ -58,7 +58,7 @@ If you do not yet have access, please contact [4DN Data Wranglers](mailto:suppor
 to get an account and [learn how to generate and save a key](https://data.4dnucleome.org/help/submitter-guide/getting-started-with-submissions#getting-connection-keys-for-the-4dn-dcic-servers).
 
 ## Generating data submission forms
-To create the data submission xls forms, you can use `get_field_info`.
+To create the data submission excel workbook, you can use `get_field_info`.
 
 It will accept the following parameters:
 ~~~~
@@ -68,7 +68,7 @@ It will accept the following parameters:
     --nodesc         do not add the descriptions in the second line (by default they are added)
     --noenums        do not add the list of options for a field if they are specified (by default they are added)
     --comments       adds any (usually internal) comments together with enums (by default False)
-    --outfile        change the default file name "fields.xls" to a specified one
+    --outfile        change the default file name "fields.xlsx" to a specified one
     --debug          to add more debugging output
     --noadmin        if you have admin access to 4DN this option lets you generate the sheet as a non-admin user
 ~~~~
@@ -77,19 +77,19 @@ Examples generating a single sheet:
 ~~~~
 get_field_info --type Biosample
 get_field_info --type Biosample --comments
-get_field_info --type Biosample --comments --outfile biosample.xls
+get_field_info --type Biosample --comments --outfile biosample.xlsx
 ~~~~
 
 Example Workbook with all sheets:
 ~~~~
-get_field_info --outfile MetadataSheets.xls
+get_field_info --outfile MetadataSheets.xlsx
 ~~~~
 
 Examples for Workbooks using a preset option:
 ~~~~
-get_field_info --type HiC --comments --outfile exp_hic_generic.xls
-get_field_info --type ChIP-seq --comments --outfile exp_chipseq_generic.xls
-get_field_info --type FISH --comments --outfile exp_fish_generic.xls
+get_field_info --type HiC --comments --outfile exp_hic_generic.xlsx
+get_field_info --type ChIP-seq --comments --outfile exp_chipseq_generic.xlsx
+get_field_info --type FISH --comments --outfile exp_fish_generic.xlsx
 ~~~~
 
 Current presets include: `Hi-C, ChIP-seq, Repli-seq, ATAC-seq, DamID, ChIA-PET, Capture-C, FISH, SPT`
@@ -100,15 +100,15 @@ Please refer to the [submission guidelines](https://data.4dnucleome.org/help/sub
 
 After you fill out the data submission forms, you can use `import_data` to submit the metadata. The method can be used both to create new metadata items and to patch fields of existing items.
 ~~~~
-	import_data filename.xls
+	import_data filename.xlsx
 ~~~~
 
 #### Uploading vs Patching
 
 Runnning `import_data` without one of the flags described below will perform a dry run submission that will include several validation checks.
 It is strongly recommended to do a dry run prior to actual submission and if necessary work with a Data Wrangler to correct any errors.
 
-If there are uuid, alias, @id, or accession fields in the xls form that match existing entries in the database, you will be asked if you want to PATCH each object.
+If there are uuid, alias, @id, or accession fields in the excel form that match existing entries in the database, you will be asked if you want to PATCH each object.
 You can use the `--patchall` flag, if you want to patch ALL objects in your document and ignore that message.
 
 If no object identifiers are found in the document, you need to use `--update` for POSTing to occur.
@@ -131,7 +131,7 @@ Note if you are attempting to run the scripts in the wranglertools directory wit
 
 ```
   python -m wranglertools.get_field_info —-type Biosource
-	python -m wranglertools.import_data filename.xls
+	python -m wranglertools.import_data filename.xlsx
 ```
 
 pypi page is - https://pypi.python.org/pypi/Submit4DN
@@ -158,10 +158,5 @@ To run the mark tests, or exclude them from the tests you can use the following
     # Run only tests with file_operation
     py.test -m file_operation
 
-For a better testing experienece that also check to ensure sufficient coverage and runs linters use invoke:
-
-```
-   invoke test
-```
-
-This will first run linters, if linters pass, tests will be run and if tests achieve specified minimum coverage (89% as of time of writting) pass the tests.
+    # skip tests that use ftp (do this when testing locally)
+    py.test -m "not ftp"