Skip to content
master
Switch branches/tags
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
src
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Avro parser plugin for Embulk

Avro parser plugin for Embulk.

Overview

  • Plugin type: parser
  • Guess supported: yes

Configuration

  • type: Specify this parser as avro
  • avsc: Specify avro schema file.
  • columns: Specify column name and type. See below (array, optional)
    • timestamp_unit: Specify unit of time. (This config is effective only if avro value is long, int, float, double)
  • default_timezone: Default timezone of the timestamp (string, default: UTC)
  • default_timestamp_format: Default timestamp format of the timestamp (string, default: %Y-%m-%d %H:%M:%S.%N %z)

If columns is not set, this plugin detect schema automatically by using avsc schema.

support timestamp_unit type is below.

  • "Second"
  • "second"
  • "sec"
  • "s"
  • "MilliSecond"
  • "millisecond"
  • "milli_second"
  • "milli"
  • "msec"
  • "ms"
  • "MicroSecond"
  • "microsecond"
  • "micro_second"
  • "micro"
  • "usec"
  • "us"
  • "NanoSecond"
  • "nanosecond"
  • "nano_second"
  • "nano"
  • "nsec"
  • "ns"

Example

in:
  type: file
  path_prefix: "items"
  parser:
    type: avro
    avsc : "./item.avsc"
    columns:
      - {name: "id", type: "long"}
      - {name: "code", type: "string"}
      - {name: "name", type: "string"}
      - {name: "description", type: "string"}
      - {name: "flag", type: "boolean"}
      - {name: "price", type: "long"}
      - {name: "item_type", type: "string"}
      - {name: "tags", type: "json"}
      - {name: "options", type: "json"}
      - {name: "spec", type: "json"}
      - {name: "created_at", type: "timestamp", format: "%Y-%m-%dT%H:%M:%S%:z"}
      - {name: "created_at_utc", type: "timestamp", timestamp_unit: "second"}

out:
  type: stdout
// item.avsc

{
  "type" : "record",
  "name" : "Item",
  "namespace" : "example.avro",
  "fields" : [
    {"name": "id", "type": "int"},
    {"name": "code", "type": "long"},
    {"name": "name", "type": "string"},
    {"name": "description", "type": ["string", "null"]},
    {"name": "flag", "type": "boolean"},
    {"name": "created_at", "type": "string"},
    {"name": "created_at_utc", "type": "float"},
    {"name": "price", "type": ["double", "null"]},
    {"name": "spec", "type": {
      "type": "record",
      "name": "item_spec",
      "fields" : [
        {"name" : "key", "type" : "string"},
        {"name" : "value", "type" : ["string", "null"]}
      ]}
    },
    {"name": "tags", "type": [{"type": "array", "items": "string"}, "null"]},
    {"name": "options", "type": {"type": "map", "values": ["string", "null"]}},
    {"name": "item_type", "type": {"name": "item_type_enum", "type": "enum", "symbols": ["D", "M"]}},
    {"name": "dummy", "type": "null"}
  ]
}

You don't have to write parser: section in the configuration file. After writing in: section, you can let embulk guess parser: section using this command:

$ embulk gem install embulk-parser-avro
$ embulk guess -g avro config.yml -o guessed.yml

Build

$ ./gradlew gem  # -t to watch change of files and rebuild continuously

About

Avro parser plugin for Embulk.

Resources

License

Packages

No packages published