bq2es-tool is an agnostic CLI to expose commands which allows you to import data (and other related actions) between BigQuery and Elastic Search.
Format:
bq2es-tool [command] [--flags]
- Repository: https://github.com/GlobalFishingWatch/bq2es-tool
There are available the following commands:
- Import
The import command allows you to import data from BigQuery to Elastic Search.
--project-id=
the project id where we want to run the query.--query=
SQL query to get rows from BigQuery.--index-name=
The destination name index.--elastic-search-url=
The Elasticsearch's URL.
--import-mode=
The import mode allows you to define if you want to recreate the index or append the data | values: [ recreate | append ]. Default: recreate.--on-error=
The flag allows you to define if delete all previous data or not in case an error happens. Values: [ delete | keep ]. Default: delete
Here an example of this command:
bq2es-tool import \
--project-id=world-fishing-827 \
--query="SELECT * FROM scratch_megan.peru_track_data" \
--elastic-search-url="https://gfw:****@elasticsearch.globalfishingwatch.org" \
--index-name=test-track-data
When you execute this command, under the hood happens the followings steps:
- The CLI executes the SQL query and gets the rows
- The CLI parses the results from RowIterator to JSON files. The keys are the name of each column.
- The CLI imports the parsed data to Elasticsearch creating a default mapping and using the bulk method. The index's name is provided by the flag --index-name