code reorg: separate data

[toloudis]

Put all data subdirs into a data/ directory, and all source code into a src/ to make it very obvious which parts of this repo are code vs data.

[meganrm]

Refactoring suggestions:

put datasets in data folder:

1. change temp folder: const TEMP_FOLDER = "./data/" + id; should be changed to const TEMP_FOLDER = "./tmp/" + id;
2. In .gitignore: data/**/cell-feature-analysis.json and data/**/file-info.json need to be changed to tmp/**/cell-feature-analysis.json and tmp/**/file-info.json
3. data should include the 5 directories that start with dataset-
4. line 77 of check-input-datasets.js should be changed from fsPromises.readdir("./").then(async (files) => { to fsPromises.readdir("./data").then(async (files) => {

Verify these changes:

npm run validate-datasets: should print out a bunch of "passed" print statements
npm run process-dataset data/dataset-cellsystems-2021 true: should run, shouldn't create any new files that aren't git ignored.

put command line scripts in bin folder

6. new folder: bin with process-dataset, upload-dataset-image and release-dataset
7. change package.json to have

"process-dataset": "node bin/process-dataset",
    "upload-image": "node bin/upload-dataset-image",
    "release-dataset": "node bin/release-dataset",

Verify these changes:

npm run process-dataset data/dataset-cellsystems-2021 true: should run

Put rest of the code in src

8. rest of the code, aws, data-validation, firebase, utils should be moved into new directory src
9. new directory src/process-single-dataset/ should have the file currently named process-single-dataset.js but renamed index.js. Also steps/ directory and constants.js

/bin/process-dataset/index.js
/src/process-single-dataset/index.js (was process-dataset/process-single-dataset.js)
/src/process-single-dataset/steps/ (moved but not renamed)
/src/process-single-dataset/constants.js (moved but not renamed)

11. The Makefile in data-validation could also be moved to bin (can pair-program if have trouble with the references) verify: cd bin && make docs should create git ignored html files
12. resolve any remaining broken references