target-clickhouse
is a Singer target for clickhouse.
Build with the Meltano Target SDK.
Setting | Required | Default | Description |
---|---|---|---|
sqlalchemy_url | False | None | SQLAlchemy connection string |
table_name | False | None | The name of the table to write to. |
engine_type | False | MergeTree | The engine type to use for the table. This must be one of the following engine types: MergeTree, ReplacingMergeTree, SummingMergeTree, AggregatingMergeTree, ReplicatedMergeTree, ReplicatedReplacingMergeTree, ReplicatedSummingMergeTree, ReplicatedAggregatingMergeTree. |
table_path | False | None | The table path for replicated tables. This is required when using any of the replication engines. Check out the documentation for more information |
replica_name | False | None | The replica_name for replicated tables. This is required when using any of the replication engines. |
cluster_name | False | None | The cluster to create tables in. This is passed as the clickhouse_cluster argument when creating a table. Documentation can be found here. |
default_target_schema | False | None | The default target database schema name to use for all streams. |
stream_maps | False | None | Config object for stream maps capability. For more information check out Stream Maps. |
stream_map_config | False | None | User-defined config values to be used within map expressions. |
flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
flattening_max_depth | False | None | The max depth to flatten schemas. |
A full list of supported settings and capabilities is available by running: target-clickhouse --about
This Singer target will automatically import any environment variables within the working directory's
.env
if the --config=ENV
is provided, such that config values will be considered if a matching
environment variable is set either in the terminal context or in the .env
file.
You can easily run target-clickhouse
by itself or in a pipeline using Meltano.
target-clickhouse --version
target-clickhouse --help
# Test using the "Carbon Intensity" sample:
tap-carbon-intensity | target-clickhouse --config /path/to/target-clickhouse-config.json
Follow these instructions to contribute to this project.
pipx install poetry
poetry install
In order to run the tests locally, you must have a Docker daemon running on your host machine.
You can start the Clickhouse container by running:
./docker_run_clickhouse.sh
Create tests within the tests
subfolder and
then run:
poetry run pytest
You can also test the target-clickhouse
CLI interface directly using poetry run
:
poetry run target-clickhouse --help
Testing with Meltano
Note: This target will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.
Next, install Meltano (if you haven't already) and any needed plugins:
# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd target-clickhouse
meltano install
Now you can test and orchestrate using Meltano:
# Test invocation:
meltano invoke target-clickhouse --version
# OR run a test `elt` pipeline with the Carbon Intensity sample tap:
meltano elt tap-carbon-intensity target-clickhouse
See the dev guide for more instructions on how to use the Meltano Singer SDK to develop your own Singer taps and targets.