Example JDBC gRPC Connector
This connector exemplifies how a JDBC gRPC Connector can be written. It contains an example that includes a MySQL Database, a PostgreSQL Database, a QIX Engine and the JDBC gRPC connector.
Go to the examples folder and run the following:
ACCEPT_EULA=<yes/no> docker-compose up --build -d
Then go to the reload-runner directory and install NodeJs dependencies:
Use the following command to run the example:
To run integration tests, use the following command:
The perfomance of the JDBC gRPC connector can be tweaked with a few different environment settings.
You can use the
DATABASE_FETCH_SIZE command to limit the memory consumption in the connector when fetching data from the database.
DATABASE_FETCH_SIZE sets the amount of rows fetched from the database loaded into memory in batches.
DATABASE_FETCH_SIZE is 100000.
DATABASE_FETCH_SIZE is not set, the entire database query is loaded into the memory of the connector.
You can use the
MAX_DATA_CHUNK_SIZE command to tweak the size of the data chunks sent over gRPC to QIX Engine.
MAX_DATA_CHUNK_SIZE represents how many fields can be batched together in one package.
This setting is highly dependant on the content of the fields and the package should be keept below the default 4MB gRPC package size limit.
MAX_DATA_CHUNK_SIZE is set to 300
These settings can be changed in the example in docker-compose.yml file.
- Java JDK 8.0
- Maven 3.3.9
mvn install java -jar ./target/core-grpc-jdbc-connector.jar
Add other JDBC Drivers
Other JDBC Drivers can be added to the pom.xml file in the following section:
<dependencies> <dependency> <groupId>org.postgresql</groupId> <artifactId>postgresql</artifactId> <version>42.2.2</version> </dependency> </dependencies>
This repository is licensed under MIT but components used in the Dockerfile examples are under other licenses. Make sure that you are complying with those licenses when using the built images.