datagov-brokerpak-harvester

Why this project

This is a cloud-service-broker plugin that makes services needed by the data.gov team brokerable via the Open Service Broker API (compatible with Cloud Foundry and Kubernetes), using Terraform.

For more information about the brokerpak concept, here's a 5-minute lightning talk from the 2019 Cloud Foundry Summit. You may also want to check out the brokerpak introduction and specification docs.

Huge props go to @josephlewis42 of Google for publishing and publicizing the brokerpak concept, and to the Pivotal team running with the concept!

Prerequisites

1. make is used for executing docker commands in a meaningful build cycle.
2. jq is used for running certain tests
3. Docker Desktop (for Mac or Windows) or Docker Engine (for Linux) is used for building, serving, and testing the brokerpak.
4. terraform 1.1.5 is used for local development.
5. AWS account credentials (as environment variables) are used for actual service provisioning. The corresponding user must have at least the permissions described in permission-policies.tf. Set at least these variables:
   • AWS_ACCESS_KEY_ID
   • AWS_SECRET_ACCESS_KEY
   • AWS_DEFAULT_REGION

For the Demo:

1. Ensure ssb-development AWS Credentials are set up.
2. Start broker

   make clean build up

3. Create demo infrastructure

   ./test.sh

4. Log into AWS Console (ssb-development) and go to SQS to send a test message... OR

   aws sqs send-message --queue-url https://sqs.us-west-2.amazonaws.com/<account-id>/<queue-name> --message-body "raw data" --delay-seconds 2
   aws sqs send-message --queue-url https://sqs.us-west-2.amazonaws.com/<account-id>/<queue-name> --message-body "anything else" --delay-seconds 2

5. Export cloud.gov S3 Credentials to inspect exported data

   aws s3 ls s3://${BUCKET_NAME}/raw/
   aws s3 ls s3://${BUCKET_NAME}/clean/

6. Destroy demo infrastructure.

   DESTROY=1 ./test.sh

Contributing

See CONTRIBUTING for additional information.

Public domain

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.