A tool converting Darwin Core Archive into Frictionless Data Package.
- datapackage.json: Ensure your DarwinCore archive complies with Frictionless specifications
- README.md: Add human readable metadata from EML
- Support all standards DarwinCore terms
- Support default values in DarwinCore schema
- Fields constraints: Enable further data validation, with goodtables
- URL: Accept DarwinCore Archive from local path or URL
- Command line interface
pip install FrictionlessDarwinCore
fdwca --help
Usage: fdwca [OPTIONS] DWCA OUTPATH
Options:
-f, --format [json|md|csv] Output format
--help Show this message and exit.
# convert from local DwC archive
fdwca myDwC.zip myDP.zip
# convert from URL (archive accessible on internet)
fdwca https://ipt.biodiversity.be/archive.do?r=rbins_saproxilyc_beetles S1dp.zip
# only generates JSON descriptor (datapackage.json)
fdwca -f json https://ipt.biodiversity.be/archive.do?r=rbins_saproxilyc_beetles datapackage.json
# only generates markdown human readable metadata (readme.md)
fdwca -f md https://ipt.biodiversity.be/archive.do?r=rbins_saproxilyc_beetles readme.md
# only converts data as zipped CSV files
fdwca -f csv https://ipt.biodiversity.be/archive.do?r=rbins_saproxilyc_beetles beetles.zip
Alternatively, you can use DwCArchive Python object like this:
from FrictionlessDarwinCore import DwCArchive
# load DarwinCore archive from URL
da = DwCArchive('https://ipt.biodiversity.be/archive.do?r=rbins_saproxilyc_beetles')
# infer Data Package structure from DarwinCore files
da.infer()
if da.valid:
# save it as Data Package locally
da.save('BeetlesDP.zip')
# ... or generates separate JSON descriptor
da.to_json('datpackage.json')
# ... or generates separate markdown human readable metadata
da.to_markdown('readme.md')
# ... or generated zip with data files only
da.to_csv('data.zip')
DarwinCore standard, created and maintained by Biodivesity Informatics Standards(aka TDWG), is used to publish Life Sciences data about observations, collections specimens, species checklists and sampling events. DarwinCore Archive(DwCA), a bundle of biodiversity data and metadata files, is well established mechanism for publishing or using data in Global Biodiversity Information Facility and other Life Sciences networks.
Frictionless Data Package is an emerging, domain agnostic, data standard that offers a variety of cross technology tools.
Bridging these two data ecosystems is our vision. This project is supported by Open Knowledge Foundation and funded under the Frictionless Data Tool Fund.
DarwinCore archives consist of:
- a core data file
- optionally, 1 or more extension data file(s)
- eml.xml: metadata written in Ecological Metadata Language
- meta.xml: the structure of the DarwinCore data files
Basically, this conversion tool appends two files to the archive, see diagram below:
- datapackage.json: data package descriptor of the data files
- readme.md: markdown, human readable, metadata
┌─────────────────────────────────────────────────────────────────┐ │ ┌──────────────────────────────────────────────────────────┐ │ │ │DarwinCore Archive │ │ │ │ │ │ │ │ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ │ │ ┌ ─ ─ Extension 1 │ │ │ │ │ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ │ │ ┌──────────────────┐ │ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ │ │ │ Core file │─ ─ ─ ─ Extension 2 │ │ │ │ │ └──────────────────┘ │ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ │ │ │ │ │ │ │ │ │ │ │ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ │ │ └ ─ ─ Extension n │ │ │ │ │ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ │ │ │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ │ │ │ │ │ meta.xml │ │ eml.xml │ │ │ │ │ └──────────────────┘ └──────────────────┘ │ │ │ │ │ │ │ │ │ └─────────────┼────────────────────────────┼───────────────┘ │ │ ▼ ▼ │ │ ┌──────────────────┐ ┌──────────────────┐ │ │ │ datapackage.json │ │ readme.md │ │ │ └──────────────────┘ └──────────────────┘ │ │ │ │ FrictionlessDarwinCore│ │ (=Data Package)│ └─────────────────────────────────────────────────────────────────┘
The tool can also generate these two files as separate outputs without touching the archive.
Additionally, the tool also converts the Core and Extension(s) files, when needed.
Darwin Core is a very persmissive standard some recommandations but almost no constraining rules. This table assigns Frictionless Data Package's type, format and constraints to every Darwin Core term. Values that do not comply with these Frictionless DarwinCore rules will automatically raise warnings.
The initial test cases suite covers a wide variety of Darwin Core usages. It should give enough confidence that basic incompatibilities are identified, reported and solved but it will not guarantee that all possible DwC Archives will automatically translate into valid Data Packages.
Here described only breaking and the most important changes.
You are encouraged to contribute by identifying/reporting issues or incompatiblities and helping to solve them.
Have a look at these online DarwinCore pages:
- Darwin Core Archives – How-to Guide
- iDigBio's Darwin Core Hour Webinar Series.
More information available under the project blog