Skip to content

Latest commit

 

History

History
60 lines (51 loc) · 3.12 KB

HOWTO.md

File metadata and controls

60 lines (51 loc) · 3.12 KB
Raw

How to create a cloud-native fiboa dataset using fiboa CLI

WORK IN PROGRESS

Once when starting with fiboa

The following steps you usually only need to do once:

  1. Install Python 3.9 or later, GDAL 3.8 or later, tippecanoe, and AWS CLI
  2. Clone the fiboa CLI repository: git clone https://github.com/fiboa/cli and cd cli to switch into the new folder.
  3. Install the dependencies of the fiboa cli repo and the CLI itself: pip install -e .
  4. Check whether the CLI works: fiboa --version

Once when creating a new dataset

  1. Create a converter using fiboa CLI converters, see the template. Create a PR for it.
  2. Register at Source Cooperative and email hello@source.coop for permission to publish in the fiboa organization.
  3. Create a new repository in the fiboa organization, e.g. @fiboa/xx-yy. You'll find it at https://beta.source.coop/fiboa/xx-yy/
  4. Create a new folder for your data, e.g. data mkdir data
  5. Create a README file at data/README.md and a license file at data/LICENSE.txt An example repository with a README etc. can be found here: https://beta.source.coop/fiboa/de-nrw/

Each time you update the dataset

  1. Go to the parent folder of the folder that contains your data (e.g. data) in CLI
  2. Run the converter, e.g. xx_yy: fiboa convert xx_yy -o data/xx-yy.parquet -h https://beta.source.coop/fiboa/xx-yy/ --collection
  3. Validate the result, e.g. fiboa validate data/xx-yy.parquet --data
  4. Move the collection.json into a stac folder: mkdir data/stac and mv data/collection.json data/stac
  5. Update the README file at data/README.md
  6. Create PMTiles file: ogr2ogr -t_srs EPSG:4326 geo.json data/xx-yy.parquet and tippecanoe -zg --projection=EPSG:4326 -o data/xx-yy.pmtiles -l xx-yy geo.json --drop-densest-as-needed
  7. Edit the STAC Collection, update the paths, and everything else that you want to customize. Also don't forget to add a link to the PMTiles file using the corresponding STAC extension.
  8. Create new credentials for S3, at: https://beta.source.coop/repositories/fiboa/xx-yy/download/
  9. Copy the credentials. Windows users may need to change the commands slightly. Use e.g. $env:AWS_DEFAULT_REGION="us-west-2" instead of export AWS_DEFAULT_REGION=us-west-2
  10. Upload to AWS: aws s3 sync data s3://us-west-2.opendata.source.coop/fiboa/xx-yy/

If you've created and published a STAC Collection for your dataset, make sure to add it to the STAC catalog that combines all datasets into a single STAC catalog It will also publish your dataset to the fiboa data overview page. Create a PR to add your STAC Collection as child link to the following file: https://github.com/fiboa/fiboa.github.io/blob/main/stac/catalog.json See also the README for an alternative.