How to run ETL jobs with Kiba

⚠️ The Kiba built-in CLI (bundle exec kiba) has been removed in v3 (see release notes for more information).

Once the job is declared, you can execute it by calling Kiba.run:

job = ETL::SyncJob.setup(some_config)
Kiba.run(job)

Please note that in general, it is not a good practice to reuse the job instance for multiple calls to Kiba.run. By doing so, you may keep some state between runs. It is better to re-parse the job before each run.

You can trigger the execution of your jobs in various places:

• From a background job (e.g. Sidekiq): use this option if your jobs are long and must be created dynamically (to react to some form of HTTP event / web-hook, for instance)
• From a rake task: use this option if you need system-based cron jobs, or one-off commands which you can launch manually
• It is also possible to run your job inside a web server query, but only do so if the job is extremely quick to run, to avoid blocking your web server resources (it is usually better to use a background job instead, though)

Next: Implementing ETL sources