Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
src
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Distinct filter plugin for Embulk

filter returns distinct records by columns you configured.

Overview

  • Plugin type: filter

Configuration

  • columns: column name list to distinguish records (array of string, required)

Example

filters:
  - type: distinct
    columns: [c0, c1]

Run Example

$ ./gradlew classpath
$ embulk run -I lib example/config.yml

Note

this plugin uses a lot of memory because of having distinct column values.

TODO

  • lessen further the amount of memory by filter. i.e. use crc32 of values as distinct key?
    • want ideas!
  • test

Build

$ ./gradlew gem  # -t to watch change of files and rebuild continuously

About

No description, website, or topics provided.

Resources

License

Packages

No packages published
You can’t perform that action at this time.