Backend — Setup Your OSM Harvester

The Pumps Harvester is also Python script, packed as an Docker image that is run as a GitHub action from our giessdenkiez-de repository. See our current active action here. It harvests pumps data from OSM and outputs the harvested file. Subsequently, for our use case, the harvested pumps data is pushed to Supabase storage.

Refer to the following diagram to understand the relationship between Frontend, Backend and the OSM Harvester. We are aware that this is overly complex and should be simplified. It grew historically into this.

In the example below we omit all comments that are the same as in the DWD harvester example. Create the file .github/workflows/pumps-harvester.yml and add the following code to it.

See https://github.com/technologiestiftung/giessdenkiez-de/blob/master/.github/workflows/pumps.yml for reference.

name: Pumps CI

on:
  workflow_dispatch:
  schedule:
    - cron: "0 4 * * 0"

jobs:
  pumps:
    runs-on: ubuntu-latest
    name: Aggregate pumps data from open street maps
    steps:
      # We want the pumps.geojson file to live in the repository therefore
      # you must check out the repository,
      - name: Checkout
        uses: actions/checkout@v2
        # Use the custom action from our repository
      - name: Pumps data generate step
        uses: technologiestiftung/giessdenkiez-de-osm-pumpen-harvester@v1.4.1
        id: pumps
        # The file where we want to write to
        with:
          outfile-path: "public/data/pumps.geojson"
      # Use the output from the `pumps`
      # This step is just for showing how to pass strings around
      - name: File output
        run: echo "The file was written to ${{ steps.pumps.outputs.file }}"
      # Push the harvested file to Supabase storage
      - name: Upload file to supabase
        run: |
          getStatusCode=$(curl -s -o /dev/null -w "%{http_code}" \
            -X GET \
            ${{ vars.SUPABASE_URL }}/storage/v1/object/info/public/${{ vars.SUPABASE_DATA_ASSETS_BUCKET }}/pumps.geojson)
          if [ "$getStatusCode" = "200" ]; then
            putStatusCode=$(curl -s -o /dev/null -w "%{http_code}" \
              -X PUT \
              -H "Authorization: Bearer ${{ secrets.SUPABASE_ACCESS_TOKEN }}" \
              -H "Content-Type: application/geo+json" \
              -d "@${{ steps.pumps.outputs.file }}" \
              ${{ vars.SUPABASE_URL }}/storage/v1/object/${{ vars.SUPABASE_DATA_ASSETS_BUCKET }}/pumps.geojson)
            if [ "$putStatusCode" = "200" ]; then
              echo "Uploading to Supabase successful"
            else
              echo "Uploading to Supabase failed"
              exit 1
            fi
          else
            postStatusCode=$(curl -s -o /dev/null -w "%{http_code}" \
              -X POST \
              -H "Authorization: Bearer ${{ secrets.SUPABASE_ACCESS_TOKEN }}" \
              -H "Content-Type: application/geo+json" \
              -d "@${{ steps.pumps.outputs.file }}" \
              ${{ vars.SUPABASE_URL }}/storage/v1/object/${{ vars.SUPABASE_DATA_ASSETS_BUCKET }}/pumps.geojson)
            if [ "$postStatusCode" = "200" ]; then
              echo "Uploading to Supabase successful"
            else
              echo "Uploading to Supabase failed"
              exit 1
            fi
          fi