Added LIMIT_PARTITION_REQUEST_NUMBER variable to protect metastore ab…

…use (#113)

* Added LIMIT_PARTITION_REQUEST_NUMBER variable to protect metastore abuse

* changelog

CHANGELOG.md

@@ -3,6 +3,10 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). 
 
+## [3.0.10] - 2023-06-28
+### Added
+- Added variable `LIMIT_PARTITION_REQUEST_NUMBER` to protect the cluster, this controls how many partitions can be scanned for each partitioned table. The default value "-1" means no limit. The limit on partitions does not affect metadata-only queries.
+
 ## [3.0.9] - 2022-11-23
 ### Added
 - Variable `MYSQL_SECRET_USERNAME_KEY` for pulling aws credentials where the key is set to something other than `username`. Defaults to `username`.  

README.md

@@ -20,6 +20,7 @@ For more information please refer to the main [Apiary](https://github.com/Expedi
 |GLUE_PREFIX|No|Prefix added to Glue databases to handle database name collisions when synchronizing multiple Hive Metastores to the Glue catalog.|
 |HADOOP_HEAPSIZE|No|Hive Metastore Java process heapsize.|
 |HMS_AUTOGATHER_STATS|No (default is `true`)|Whether or not to create basic statistics on table/partition creation. Valid values are `true` or `false`.|
+|LIMIT_PARTITION_REQUEST_NUMBER|No (default is `-1`)|To protect the cluster, this controls how many partitions can be scanned for each partitioned table. The default value "-1" means no limit. The limit on partitions does not affect metadata-only queries.|
 |HIVE_METASTORE_ACCESS_MODE|No|Hive Metastore access mode, applicable values are: readwrite, readonly|
 |HIVE_DB_NAMES|No|comma separated list of Hive database names, when specified Hive databases will be created and mapped to corresponding S3 buckets.|
 |HIVE_METASTORE_LOG_LEVEL|No|Hive Metastore service Log4j log level.|

files/hive-site.xml

@@ -28,10 +28,10 @@
   <value>true</value>
 </property>
 
-  <property>
-    <name>datanucleus.connectionPool.maxPoolSize</name>
-    <value>10</value>
-  </property>
+<property>
+  <name>datanucleus.connectionPool.maxPoolSize</name>
+  <value>10</value>
+</property>
 
 <property>
   <name>hive.metastore.uris</name>
@@ -98,4 +98,9 @@
   <value>true</value>
 </property>
 
+<property>
+  <name>hive.metastore.limit.partition.request</name>
+  <value>-1</value>
+</property>
+
 </configuration>

files/startup.sh

@@ -31,6 +31,10 @@ if [[ -n ${HMS_AUTOGATHER_STATS} ]]; then
   update_property.py hive.stats.autogather "${HMS_AUTOGATHER_STATS}" /etc/hive/conf/hive-site.xml
 fi
 
+if [[ -n ${LIMIT_PARTITION_REQUEST_NUMBER} ]]; then
+  update_property.py hive.metastore.limit.partition.request "${LIMIT_PARTITION_REQUEST_NUMBER}" /etc/hive/conf/hive-site.xml
+fi
+
 #configure LDAP group mapping, required for ranger authorization
 if [[ -n $LDAP_URL ]] ; then
     if [[ -n $LDAP_SECRET_ARN ]] ; then