Skip to content

datagovsg/ckanext-s3-resources

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

65 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

CKAN Extension to upload resources to AWS S3

This extension can be used to upload your CKAN resources to AWS S3 instead of the CKAN server. You can blacklist certain resource types (e.g. APIs) from being uploaded to S3. A paster command is provided to migrate resources to S3.

Requirements

  • boto3 - for connecting to S3

Setup/Configuration

For the extension to work, you need to ensure that the following configuration options have been set in your configuration file *.ini:

  • ckan.datagovsg_s3_resources.s3_aws_access_key_id - AWS access key ID. Obtained from AWS.
  • ckan.datagovsg_s3_resources.s3_aws_secret_access_key - AWS secret access key ID. Obtained from AWS.
  • ckan.datagovsg_s3_resources.s3_bucket_name - The name of the bucket on S3 to upload the resources to.
  • ckan.datagovsg_s3_resources.s3_url_prefix - Base URL (https://s3.data.gov.sg)
    • Resource URLs will be in the form of <base_url><package_name>/<resource_filename>
      • e.g. https://s3.data.gov.sg/package-123/resource-123.csv
    • Package zip URLs will be in the form of <base_url><package_name>/<package_name>.zip
      • e.g. https://s3.data.gov.sg/package-123/package-123.csv
    • e.g. ckan.datagovsg_s3_resources.s3_url_prefix = https://bucket-name.s3.amazonaws.com/
  • ckan.datagovsg_s3_resources.upload_filetype_blacklist (optional) - A space separated list of file formats to ignore.
    • e.g. ckan.datagovsg_s3_resources.upload_filetype_blacklist = csv pdf xls
  • ckan.datagovsg_s3_resources.s3_aws_region_name (optional) - Specify which AWS region to use.
    • e.g. ap-southeast-1

Migration

The extension includes a paster command to help migrate the existing resources to S3. The command can be run by doing:

paster --plugin=plugin_name migrate_s3

About

CKAN Extension to upload resources to AWS S3 https://data.gov.sg

Resources

License

Security policy

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages