No description or website provided.
Kotlin Other
Latest commit 1ae25e3 Feb 16, 2017 @kamatama41 committed on GitHub Merge pull request #32 from kamatama41/fix-checkstyle
Fix checkstyle warns
Permalink
Failed to load latest commit information.
config/checkstyle Update to current plugin format Feb 14, 2017
gradle/wrapper Update gradle Feb 2, 2017
lib/embulk/input
src Fix checkstyle warns Feb 16, 2017
.gitignore
.travis.yml
Dockerfile Replace using image to reduce build time Jan 12, 2017
LICENSE.txt First commit. Apr 14, 2015
README.md
build.gradle Fix checkstyle warns Feb 16, 2017
docker-compose.yml Use docker-compose instead of docker Jan 22, 2017
gradlew Update gradle to the latest Jan 30, 2017
gradlew.bat Update gradle to the latest Jan 30, 2017

README.md

Build Status

Remote file input plugin for Embulk

This plugin load data from Remote hosts by SCP

Overview

  • Plugin type: file input
  • Resume supported: yes
  • Cleanup supported: yes

Configuration

  • hosts: Target hosts, format should be host or host:port (overrides default_port) (list, default: [])
  • hosts_command: Command for getting hosts(Windows not supported). If given the option, "hosts" is overwritten. (string, default: null)
  • hosts_separator: Separator for "hosts_command" result (string, default: " ")
  • default_port: Default port number for SSH (integer, default: 22)
  • path: Path of remote host (File or Directory) (string, default: "")
  • path_command: Command for getting path (Windows not supported). If given the option "path" is overwritten. (string, default: null)
  • ignore_not_found_hosts: If the option is true, Hosts which meet the following conditions are skipped. (Means they are not included into resume target.) (boolean, default: false)
    • Target file (or directory) isn't found
    • Occurred SSH error
  • auth: SSH authentication setting (hash, default: {})
    • user: SSH username (string, default: execute user)
    • type: public_key or password (string, default: public_key)
    • key_path: Path of secret key (If you choose type "public_key") (string, default: "~/.ssh/id_rsa or id_dsa")
    • password: SSH password (If you choose type "password") (string)
    • skip_host_key_verification: If the option is true, HostKey verification will be skipped (boolean, default: false)

Example

in:
  type: remote
  hosts:
    - host1
    - host2:10022
#  hosts_command: echo 'host1,host2'
#  hosts_separator: ','
  path: /some/path/20150414125923
#  path_command: echo /some/path/`date "+%Y%m%d%H%M%S"`
  ignore_not_found_hosts: true
  auth:
    user: {username}
    type: public_key
    key_path: /usr/home/.ssh/id_rsa
#    type: password
#    password: {password}

Note

When this plugin run on Linux, a task might be blocked.
The cause is java.security.SecureRandom. Please try one of the followings.

set JVM_OPTION "-Djava.security.egd"

$ export JAVA_TOOL_OPTIONS="-Djava.security.egd=file:/dev/./urandom"
$ embulk run config.yml

rewrite ${JAVA_HOME}/jre/lib/security/java.security

# securerandom.source=file:/dev/random # before
securerandom.source=file:/dev/./urandom # after

see also

http://stackoverflow.com/questions/137212/how-to-solve-performance-problem-with-java-securerandom

Development on local machine

  • Install Docker and then we can create SSH-able containers
$ ssh-keygen -t ecdsa -f ./id_rsa_test -N ''
$ docker-compose up -d
$ docker-compose ps
          Name                 Command       State           Ports         
--------------------------------------------------------------------------
embulkinputremote_host1_1   /entrypoint.sh   Up      0.0.0.0:10022->22/tcp 
embulkinputremote_host2_1   /entrypoint.sh   Up      0.0.0.0:10023->22/tcp 

Build

$ ./gradlew gem