Microsoft Azure Blob Storage file output plugin for Embulk
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
config/checkstyle Upgraded embulk to v0.8.2 by embulk migrate Feb 2, 2016
gradle/wrapper Upgrade Gradle from 2.10 to 3.2.1 Jul 24, 2017
lib/embulk/output first commit Oct 11, 2015
src
.gitignore first commit Oct 11, 2015
.travis.yml Only build for Java 8 Oct 18, 2018
CHANGELOG.md v0.1.7 Oct 19, 2018
README.md
appveyor.yml Add comment about encrypted value for Appveyor Oct 18, 2018
build.gradle v0.1.7 Oct 19, 2018
gradlew first commit Oct 11, 2015
gradlew.bat first commit Oct 11, 2015

README.md

Azure Blob Storage file output plugin for Embulk

Build Status

Embulk file output plugin stores files on Microsoft Azure Blob Storage

Overview

  • Plugin type: file output
  • Resume supported: no
  • Cleanup supported: no

Configuration

First, create Azure Storage Account.

  • account_name: storage account name (string, required)
  • account_key: primary access key (string, required)
  • container: container name (string, required)
  • path_prefix: prefix of target keys (string, required) (string, required)
  • file_ext: e.g. "csv.gz, json.gz" (string, required)

Auto create container

container will create automatically when container doesn't exists.

When a container was deleted, a container with same name cannot be created for at least 30 seconds. It's a service specification of Azure Blob Storage.

Example

out:
  type: azure_blob_storage
  account_name: myaccount
  account_key: myaccount_key
  container: my-container
  path_prefix: logs/csv-
  file_ext: csv.gz
  formatter:
    type: csv
    header_line: false
  encoders:
  - {type: gzip}

Build

$ ./gradlew gem  # -t to watch change of files and rebuild continuously

Test

$ ./gradlew test  # -t to watch change of files and rebuild continuously

To run unit tests, we need to configure the following environment variables.

When environment variables are not set, skip some test cases.

AZURE_ACCOUNT_NAME
AZURE_ACCOUNT_KEY
AZURE_CONTAINER
AZURE_CONTAINER_DIRECTORY (optional, if needed)

If you're using Mac OS X El Capitan and GUI Applications(IDE), like as follows.

$ vi ~/Library/LaunchAgents/environment.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>my.startup</string>
  <key>ProgramArguments</key>
  <array>
    <string>sh</string>
    <string>-c</string>
    <string>
      launchctl setenv AZURE_ACCOUNT_NAME my-account-name
      launchctl setenv AZURE_ACCOUNT_KEY my-account-key
      launchctl setenv AZURE_CONTAINER my-container
      launchctl setenv AZURE_CONTAINER_DIRECTORY unittests
    </string>
  </array>
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

$ launchctl load ~/Library/LaunchAgents/environment.plist
$ launchctl getenv AZURE_ACCOUNT_NAME //try to get value.

Then start your applications.