Cloud Functions that fetch data from public APIs and store it in Google Cloud Storage.
Google Cloud Functions Gen2 Golang environment is supported.
- Fetch data is stored in a path based on the API's URL. ( like ghq )
- Therefore, even if you get various API data, you do not need to manage the bucket's path.
- Execute Cloud Pub/Sub as a trigger.
- Therefore, periodic execution is also possible with Cloud Scheduler.
- Fetch data is managed by Object Versioning.
- GCS price are optimized by Object Lifecycle Management.
- The storage class is set to change from Standard to Coldline after 7 days from object creation and from Coldline to Archive after 30 days (easy to change).
- zl (zap based logger) for logging by level.
- Logs in JSON format so it can check the element contents in detail with Cloud Logging.
- You can also check the contents of CloudEvents triggered by Functions in the log.
If you haven't already, install and set up the Cloud SDK.
Create GCP resources
make init
Run test
make test
Deploy to google cloud functions
make deploy
Send pub/sub event
make send URL="https://api.github.com/users/github"
Open resources in console
make open
The api data saved in GCS can be used for various purposes such as data analysis and machine learning by loading it into BigQuery.
Of course, it can also be used in applications.
List of public APIs.
GitHub - public-apis/public-apis: A collective list of free APIs