Skip to content

Commit

Permalink
Added variable KAFKA_COMPRESSION_TYPE to optionally add compression t…
Browse files Browse the repository at this point in the history
…ype when sending Metastore events to Kafka (#115)

* Added LIMIT_PARTITION_REQUEST_NUMBER variable to protect metastore abuse

* changelog

* Added variable KAFKA_COMPRESSION_TYPE to optionally add compression type when sending Metastore events to Kafka

* add readme and fix script

* comments
  • Loading branch information
javsanbel2 committed Jul 25, 2023
1 parent e2dca92 commit cb5623f
Show file tree
Hide file tree
Showing 5 changed files with 17 additions and 2 deletions.
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@ All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## [3.0.11] - 2023-07-25
### Added
- Added variable `KAFKA_COMPRESSION_TYPE` to optionally add compression type when sending Metastore events to Kafka through apiary-metastore-listener library.
- Upgraded `APIARY_EXTENSIONS_VERSION` to `7.3.7` (was `7.3.4`).
- Upgraded `APIARY_GLUESYNC_LISTENER_VERSION` to `7.3.7` (was `7.3.6`).

## [3.0.10] - 2023-06-28
### Added
- Added variable `LIMIT_PARTITION_REQUEST_NUMBER` to protect the cluster, this controls how many partitions can be scanned for each partitioned table. The default value "-1" means no limit. The limit on partitions does not affect metadata-only queries.
Expand Down
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
from amazonlinux:2

ENV RANGER_VERSION 2.0.0
ENV APIARY_EXTENSIONS_VERSION 7.3.4
ENV APIARY_GLUESYNC_LISTENER_VERSION 7.3.6
ENV APIARY_EXTENSIONS_VERSION 7.3.7
ENV APIARY_GLUESYNC_LISTENER_VERSION 7.3.7
ENV APIARY_RANGER_PLUGIN_VERSION 5.0.1
ENV APIARY_METASTORE_METRICS_VERSION 4.2.0
ENV APIARY_METASTORE_AUTH_VERSION 4.2.0
Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ For more information please refer to the main [Apiary](https://github.com/Expedi
|HMS_MIN_THREADS|No (defaults to `200`)|Minimum size of the Hive metastore thread pool.|
|HMS_MAX_THREADS|No (defaults to `1000`)|Maximum size of the Hive metastore thread pool.|
|INSTANCE_NAME|Yes|Apiary instance name, will be used as prefix on most AWS resources to allow multiple Apiary instance deployments.|
|KAFKA_BOOTSTRAP_SERVERS|No|Kafka Bootstrap Servers to enable Kafka Metastore listener and send Metastore events to Kafka.|
|KAFKA_CLIENT_ID|No|Kafka label you define that names the Kafka producer.|
|KAFKA_COMPRESSION_TYPE|No|Kafka Compression type, if none is specified there is no compression enabled. Values available are gzip, lz4 and snappy.|
|LDAP_BASE|No|LDAP base DN used to search for user groups.|
|LDAP_CA_CERT|Base64 encoded Certificate Authority Bundle to validate LDAP SSL connection.|
|LDAP_SECRET_ARN|No|LDAP bind DN SecretsManager secret ARN.|
Expand Down
5 changes: 5 additions & 0 deletions files/hive-site.xml
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,11 @@
<value>KAFKA_CLIENT_ID</value>
</property>

<property>
<name>com.expediagroup.apiary.extensions.events.metastore.kafka.messaging.compression.type</name>
<value>none</value>
</property>

<property>
<name>hive.metastore.disallow.incompatible.col.type.changes</name>
<value>true</value>
Expand Down
1 change: 1 addition & 0 deletions files/startup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ if [[ ! -z $KAFKA_BOOTSTRAP_SERVERS ]]; then
[[ -n $ECS_CONTAINER_METADATA_URI ]] && export KAFKA_CLIENT_ID=$(wget -q -O - ${ECS_CONTAINER_METADATA_URI}/task|jq -r .TaskARN|awk -F/ '{ print $NF }')
[[ -n $KUBERNETES_SERVICE_HOST ]] && export KAFKA_CLIENT_ID="$HOSTNAME"
[[ -n $KAFKA_CLIENT_ID ]] && sed "s/KAFKA_CLIENT_ID/$KAFKA_CLIENT_ID/" -i /etc/hive/conf/hive-site.xml
[[ -n $KAFKA_COMPRESSION_TYPE ]] && update_property.py com.expediagroup.apiary.extensions.events.metastore.kafka.messaging.compression.type "$KAFKA_COMPRESSION_TYPE" /etc/hive/conf/hive-site.xml
fi

APIARY_S3_INVENTORY_SCHEMA=s3_inventory
Expand Down

0 comments on commit cb5623f

Please sign in to comment.