pyDataverse_nesstar

Scripts, files and documentation for the pyDataverse NESSTAR data migration with pyDataverse.

INSTALL

Requirements:

• pyDataverse
• pydantic

git clone git@github.com:AUSSDA/pyDataverse_nesstar.git
cd pyDataverse_nesstar
pipenv install

RUN

Set up

pipenv shell
export INSTANCE="INSTANCE_NAME"

Place .env files in expected location with needed variables set: see src/settings.py to find out more.

Workflow

In general, the workflow for data imports is always as this:

1. Prepare the CSV file
2. Prepare the script
3. Prepare Dataverse installations
4. Prepare Dataverse
5. Create Dataverse
6. Test on local Dataverse installation
7. Upload 1 dataset with datafiles
8. Review: Developer
9. Publish the dataset with datafiles
10. Review: Developer
11. Upload 10 datasets with datafiles
12. Publish the datasets
13. Review: Developer
14. Upload all datasets with datafiles
15. Publish the datasets
16. Review: Developer
17. Review
18. Test on development Dataverse installation
19. Upload 1 dataset with datafiles
20. Review: Developer + Ingest
21. Upload all datasets with datafiles
22. Review: Developer + Ingest
23. Import on production Dataverse installation
24. Upload 1 dataset with datafiles
25. Review: Developer + Ingest
26. Upload all datasets with datafiles
27. Review: Developer + Ingest
28. Publish all datasets
29. Review: Developer + Ingest
30. Clean up the Dataverse installations
31. Delete/Destroy datasets
32. Delete Dataverses

Execute Script

Before you run the script, adapt the data pipeline control flags in src/nesstar.py.

cd src
python -m nesstar

DEVELOPMENT

Install

git clone git@github.com:AUSSDA/pyDataverse_nesstar.git
cd pyDataverse_nesstar
pipenv install --dev
pre-commit install