DRILL-8221: Collect bug fixes from master for Drill 1.20.1 (#2545)

* Prepare for the next bugfix iteration. * SAS Reader fixes (#2472) Co-authored-by: pseudomo <pseudomo@yandex.ru> * Add jackson-bom (#2454) * [DRILL-8150] log4j 2.17.2 in format-excel (#2476) * DRILL-8151: Add support for more ElasticSearch and Cassandra data types (#2477) * DRILL-8154: Upgrade to poi 5.2.1 (#2480) * DRILL-8145: Fix flaky TestDrillbitResilience#memoryLeaksWhenCancelled test case (#2471) * Set Brotli codec jar and test to occur only on Linux amd64. * DRILL-8145: Fix flaky TestDrillbitResilience#memoryLeaksWhenCancelled test case - changing timeout for TestDrillbitResilience tests - timing tuning for memoryLeaksWhenCancelled - update TestContainers version - -DforkCount=1 for Travis Maven build - directMemoryMb: 2500 -> 4500 leads to less occasinal test failures Co-authored-by: James Turton <james@somecomputer.xyz> * [MINOR UPDATE] Add Stalebot Config (#2487) * [MINOR UPDATE] Fix license for Stalebot Config (#2488) * DRILL-8156: Declare and chown a /data VOLUME in the Drill Dockerfile (#2491) * Add a mountpoint and VOLUME for local storage to Dockerfile. * Address review comments concerning layer ordering. * Fix image size blowup by moving chmod to intermediate container. * Combine RUN commands in Dockerfile. * DRILL-8168: Do not duplicate attempts to impersonate a user in the REST API (#2495) * DRILL-8172: Use the specified memory usage for Travis CI (#2500) * DRILL-8165: Upgrade liquibase because of CVE-2022-0839 (#2497) * Create codeql-analysis.yml * Update codeql-analysis.yml Removed cpp from code analysis * [MINOR UPDATE] Add license to CodeQL YAML (#2501) * DRILL-8176: Upgrade Jackson Due to CVE-2020-36518 (#2504) * DRILL-8164: Upgrade metadata-extractor because of CVE-2022-24613 (#2493) * DRILL-8164: Upgrade metadata-extractor because of CVE-2022-24613 * Update the ProfileCopyright tag name * Update the mov format name * Add the QuickTime.Rotation tag * Bump metadata-extractor to 2.17.0 * DRILL-8178: Bump AWS Libraries to Latest Version (#2506) * DRILL-8175: Update Drill release script after 1.20 (#2503) * Set DRILL_PID_DIR in Dockerfile to writable location for distributed mode. Some users of the images built from this Dockerfile customise them so that they launch Drill in distributed mode instead of embedded mode. This change saves them from having to set DRILL_PID_DIR themselves in order to succeed. * Update release script and instructions after the release of 1.20. - Add support for specifying a build profile such as "hadoop-2". - Update instuctions for the Drill web site. - Update instructions for uploading RCs (no more home.apache) - Some fixes. * DRILL-8176: minor issue in previous jackson bom (#2508) * minor issue in previous jackson bom * Update pom.xml * DRILL-8187: Dialect factory returns ANSI SQL dialect for BigQuery (#2513) * DRILL-8192: Cassandra queries fail when enabled Mongo plugin (#2518) * DRILL-8013: Drill attempts to push "$SUM0" to JDBC storage plugin for AVG (#2521) * DRILL-8194: Fix the function of REPLACE throws IndexOutOfBoundsException If text's length is more than previously applied (#2522) * DRILL-8200: Update Hadoop libs to ≥ 3.2.3 for CVE-2022-26612 (#2525) * Remove pointless Buffer casts. Compiling Drill with JDK > 8 will still result in ByteBuffer <-> Buffer cast exceptions at runtime when running on JDK 8 even though maven.target.version is set to 8. Setting maven.compiler.release to 8 solves the Buffer casts but raises a compilation error of package sun.security.jgss does not exist for JDK 8. There were a few handwritten casts to avoid the Buffer casting issue but many instances are not covered so the few reverted in this commit achieve nothing. * Update Hadoop to 3.2.3. * [MINOR UPDATE] Update AWS Java SDK to 1.12.211 * DRILL-8219: Handle null catalog names returned by DB2 in storage-jdbc. (#2542) Co-authored-by: pseudomo <yura_levchenko@mail.ru> Co-authored-by: pseudomo <pseudomo@yandex.ru> Co-authored-by: Rymar Maksym <rim.maxim+dev@gmail.com> Co-authored-by: PJ Fanning <pjfanning@users.noreply.github.com> Co-authored-by: Volodymyr Vysotskyi <vvovyk@gmail.com> Co-authored-by: Vitalii Diravka <vitalii@apache.org> Co-authored-by: Charles S. Givre <cgivre@apache.org> Co-authored-by: luoc <luoc@apache.org> Co-authored-by: xurenhe <xurenhe19910131@gmail.com>
apache · May 12, 2022 · 2fa849d · 2fa849d
1 parent d198789
commit 2fa849d
Show file tree

Hide file tree

Showing 112 changed files with 1,339 additions and 429 deletions.
diff --git a/.github/stale.yml b/.github/stale.yml
@@ -0,0 +1,35 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Number of days of inactivity before an issue becomes stale
+daysUntilStale: 120
+# Number of days of inactivity before a stale issue is closed
+daysUntilClose: 14
+# Issues with these labels will never be considered stale
+exemptLabels:
+  - pinned
+  - security
+# Label to use when marking an issue as stale
+staleLabel: wontfix
+# Comment to post when marking an issue as stale. Set to `false` to disable
+markComment: >
+  This issue has been automatically marked as stale because it has not had
+  recent activity. It will be closed if no further activity occurs. Thank you
+  for your contributions.
+# Comment to post when closing a stale issue. Set to `false` to disable
+closeComment: false
diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# For most projects, this workflow file will not need changing; you simply need
+# to commit it to your repository.
+#
+# You may wish to alter this file to override the set of languages analyzed,
+# or to provide custom queries or build logic.
+#
+# ******** NOTE ********
+# We have attempted to detect the languages in your repository. Please check
+# the `language` matrix defined below to confirm you have the correct set of
+# supported CodeQL languages.
+#
+name: "CodeQL"
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    # The branches below must be a subset of the branches above
+    branches: [ master ]
+  schedule:
+    - cron: '33 21 * * 5'
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+    permissions:
+      actions: read
+      contents: read
+      security-events: write
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: [ 'java', 'javascript' ]
+        # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
+        # Learn more about CodeQL language support at https://git.io/codeql-language-support
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+
+    # Initializes the CodeQL tools for scanning.
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v1
+      with:
+        languages: ${{ matrix.language }}
+        # If you wish to specify custom queries, you can do so here or in a config file.
+        # By default, queries listed here will override any specified in a config file.
+        # Prefix the list here with "+" to use these queries and those in the config file.
+        # queries: ./path/to/local/query, your-org/your-repo/queries@main
+
+    # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
+    # If this step fails, then you should remove it and run the build manually (see below)
+    - name: Autobuild
+      uses: github/codeql-action/autobuild@v1
+
+    # ℹ️ Command-line programs to run using the OS shell.
+    # 📚 https://git.io/JvXDl
+
+    # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
+    #    and modify them (or add more) to build your code if your project
+    #    uses a compiled language
+
+    #- run: |
+    #   make bootstrap
+    #   make release
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v1
diff --git a/.travis.yml b/.travis.yml
@@ -45,6 +45,9 @@ cache:
 before_install:
   - export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-arm64"
   - export PATH="$JAVA_HOME/bin:$PATH"
+  - export MEMORYMB=2048
+  - export DIRECTMEMORYMB=5120
+  - free -m
   - java -version
   - mvn -version
   - git fetch --unshallow
@@ -71,7 +74,7 @@ install:
   # For protobuf phase: builds Drill project, performs license checkstyle goal and regenerates Java and C++ Protobuf files
   - |
     if [ $PHASE = "tests" ]; then \
-      mvn install --batch-mode --no-transfer-progress \
+      mvn install --batch-mode --no-transfer-progress -DforkCount=1 -DmemoryMb=$MEMORYMB -DdirectMemoryMb=$DIRECTMEMORYMB \
         -DexcludedGroups="org.apache.drill.categories.SlowTest,org.apache.drill.categories.UnlikelyTest,org.apache.drill.categories.SecurityTest"; \
     elif [ $PHASE = "build_checkstyle_protobuf" ]; then \
       MAVEN_OPTS="-Xms1G -Xmx1G" mvn install --no-transfer-progress -Drat.skip=false -Dlicense.skip=false --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -DskipTests=true -Dmaven.javadoc.skip=true -Dmaven.source.skip=true && \

diff --git a/Dockerfile b/Dockerfile
@@ -49,25 +49,33 @@ RUN mvn -Dmaven.artifact.threads=5 -T1C clean install -DskipTests
 # Get project version and copy built binaries into /opt/drill directory
 RUN VERSION=$(mvn -q -Dexec.executable=echo -Dexec.args='${project.version}' --non-recursive exec:exec) \
  && mkdir /opt/drill \
- && mv distribution/target/apache-drill-${VERSION}/apache-drill-${VERSION}/* /opt/drill
+ && mv distribution/target/apache-drill-${VERSION}/apache-drill-${VERSION}/* /opt/drill \
+ && chmod -R +r /opt/drill
 
 # Target image
 
 # Set the BASE_IMAGE build arg when you invoke docker build.  
 FROM $BASE_IMAGE
 
-ENV DRILL_HOME=/opt/drill DRILL_USER=drilluser
+# Starts Drill in embedded mode and connects to Sqlline
+ENTRYPOINT $DRILL_HOME/bin/drill-embedded
 
-RUN mkdir $DRILL_HOME
+ENV DRILL_HOME=/opt/drill
+ENV DRILL_USER=drilluser
+ENV DRILL_USER_HOME=/var/lib/drill
+ENV DRILL_PID_DIR=$DRILL_USER_HOME
+ENV DRILL_LOG_DIR=$DRILL_USER_HOME/log
+ENV DATA_VOL=/data
 
-RUN groupadd -g 999 $DRILL_USER \
- && useradd -r -u 999 -g $DRILL_USER $DRILL_USER -m -d /var/lib/drill \
- && chown -R $DRILL_USER: $DRILL_HOME
+RUN mkdir $DRILL_HOME $DATA_VOL \
+ && groupadd -g 999 $DRILL_USER \
+ && useradd -r -u 999 -g $DRILL_USER $DRILL_USER -m -d $DRILL_USER_HOME \
+ && chown $DRILL_USER: $DATA_VOL
 
-USER $DRILL_USER
+# A Docker volume where users may store persistent data, e.g. persistent Drill
+# config by specifying a Drill BOOT option of sys.store.provider.local.path: "/data".
+VOLUME $DATA_VOL
 
-COPY --from=build --chown=$DRILL_USER /opt/drill $DRILL_HOME
-
-# Starts Drill in embedded mode and connects to Sqlline
-ENTRYPOINT $DRILL_HOME/bin/drill-embedded
+COPY --from=build /opt/drill $DRILL_HOME
 
+USER $DRILL_USER
diff --git a/common/pom.xml b/common/pom.xml
@@ -24,7 +24,7 @@
   <parent>
     <artifactId>drill-root</artifactId>
     <groupId>org.apache.drill</groupId>
-    <version>1.20.0</version>
+    <version>1.20.1-SNAPSHOT</version>
   </parent>
 
   <artifactId>drill-common</artifactId>

diff --git a/contrib/data/pom.xml b/contrib/data/pom.xml
@@ -23,7 +23,7 @@
   <parent>
     <artifactId>drill-contrib-parent</artifactId>
     <groupId>org.apache.drill.contrib</groupId>
-    <version>1.20.0</version>
+    <version>1.20.1-SNAPSHOT</version>
   </parent>
 
   <groupId>org.apache.drill.contrib.data</groupId>

diff --git a/contrib/data/tpch-sample-data/pom.xml b/contrib/data/tpch-sample-data/pom.xml
@@ -23,7 +23,7 @@
   <parent>
     <artifactId>drill-contrib-data-parent</artifactId>
     <groupId>org.apache.drill.contrib.data</groupId>
-    <version>1.20.0</version>
+    <version>1.20.1-SNAPSHOT</version>
   </parent>
 
   <artifactId>tpch-sample-data</artifactId>

diff --git a/contrib/format-esri/pom.xml b/contrib/format-esri/pom.xml
@@ -24,7 +24,7 @@
   <parent>
     <artifactId>drill-contrib-parent</artifactId>
     <groupId>org.apache.drill.contrib</groupId>
-    <version>1.20.0</version>
+    <version>1.20.1-SNAPSHOT</version>
   </parent>
 
   <artifactId>drill-format-esri</artifactId>

diff --git a/contrib/format-excel/pom.xml b/contrib/format-excel/pom.xml
@@ -24,14 +24,15 @@
   <parent>
     <artifactId>drill-contrib-parent</artifactId>
     <groupId>org.apache.drill.contrib</groupId>
-    <version>1.20.0</version>
+    <version>1.20.1-SNAPSHOT</version>
   </parent>
 
   <artifactId>drill-format-excel</artifactId>
   <name>Drill : Contrib : Format : Excel</name>
 
   <properties>
-    <poi.version>5.2.0</poi.version>
+    <poi.version>5.2.1</poi.version>
+    <log4j.version>2.17.2</log4j.version>
   </properties>
   <dependencies>
     <dependency>
@@ -52,17 +53,17 @@
     <dependency>
       <groupId>com.github.pjfanning</groupId>
       <artifactId>excel-streaming-reader</artifactId>
-      <version>3.4.0</version>
+      <version>3.6.0</version>
     </dependency>
     <dependency>
       <groupId>org.apache.logging.log4j</groupId>
       <artifactId>log4j-api</artifactId>
-      <version>2.17.1</version>
+      <version>${log4j.version}</version>
     </dependency>
     <dependency>
       <groupId>org.apache.logging.log4j</groupId>
       <artifactId>log4j-to-slf4j</artifactId>
-      <version>2.17.1</version>
+      <version>${log4j.version}</version>
     </dependency>
     <!-- Test dependencies -->
     <dependency>

diff --git a/contrib/format-hdf5/pom.xml b/contrib/format-hdf5/pom.xml
@@ -24,7 +24,7 @@
   <parent>
     <artifactId>drill-contrib-parent</artifactId>
     <groupId>org.apache.drill.contrib</groupId>
-    <version>1.20.0</version>
+    <version>1.20.1-SNAPSHOT</version>
   </parent>
 
   <artifactId>drill-format-hdf5</artifactId>

diff --git a/contrib/format-httpd/pom.xml b/contrib/format-httpd/pom.xml
@@ -23,7 +23,7 @@
   <parent>
     <artifactId>drill-contrib-parent</artifactId>
     <groupId>org.apache.drill.contrib</groupId>
-    <version>1.20.0</version>
+    <version>1.20.1-SNAPSHOT</version>
   </parent>
   <artifactId>drill-format-httpd</artifactId>
   <name>Drill : Contrib : Format : Httpd/Nginx Access Log</name>

diff --git a/contrib/format-iceberg/pom.xml b/contrib/format-iceberg/pom.xml
@@ -23,7 +23,7 @@
   <parent>
     <artifactId>drill-contrib-parent</artifactId>
     <groupId>org.apache.drill.contrib</groupId>
-    <version>1.20.0</version>
+    <version>1.20.1-SNAPSHOT</version>
   </parent>
 
   <artifactId>drill-iceberg-format</artifactId>

diff --git a/...berg/src/main/java/org/apache/drill/exec/store/iceberg/plan/IcebergPluginImplementor.java b/...berg/src/main/java/org/apache/drill/exec/store/iceberg/plan/IcebergPluginImplementor.java
@@ -32,6 +32,8 @@
 import org.apache.drill.exec.planner.logical.DrillOptiq;
 import org.apache.drill.exec.planner.logical.DrillParseContext;
 import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.apache.drill.exec.store.StoragePlugin;
+import org.apache.drill.exec.store.dfs.FileSystemPlugin;
 import org.apache.drill.exec.store.iceberg.IcebergGroupScan;
 import org.apache.drill.exec.store.plan.AbstractPluginImplementor;
 import org.apache.drill.exec.store.plan.rel.PluginFilterRel;
@@ -134,6 +136,11 @@ public boolean artificialLimit() {
     return true;
   }
 
+  @Override
+  protected Class<? extends StoragePlugin> supportedPlugin() {
+    return FileSystemPlugin.class;
+  }
+
   @Override
   public boolean splitProject(Project project) {
     return true;

diff --git a/contrib/format-image/pom.xml b/contrib/format-image/pom.xml
@@ -24,7 +24,7 @@
   <parent>
     <artifactId>drill-contrib-parent</artifactId>
     <groupId>org.apache.drill.contrib</groupId>
-    <version>1.20.0</version>
+    <version>1.20.1-SNAPSHOT</version>
   </parent>
 
   <artifactId>drill-format-image</artifactId>
@@ -39,6 +39,7 @@
     <dependency>
       <groupId>com.drewnoakes</groupId>
       <artifactId>metadata-extractor</artifactId>
+      <version>2.17.0</version>
     </dependency>
 
     <!-- Test dependencies -->

diff --git a/...b/format-image/src/main/java/org/apache/drill/exec/store/image/GenericMetadataReader.java b/...b/format-image/src/main/java/org/apache/drill/exec/store/image/GenericMetadataReader.java
@@ -145,17 +145,17 @@ public void read(@NotNull FileType fileType, @NotNull FileStatus fileStatus, @No
           try {
             int numOfComponent = 1;
             int colorType = pngDir.getInt(PngDirectory.TAG_COLOR_TYPE);
-            if (colorType == PngColorType.IndexedColor.getNumericValue()) {
+            if (colorType == PngColorType.INDEXED_COLOR.getNumericValue()) {
               directory.setColorMode("Indexed");
-            } else if (colorType == PngColorType.Greyscale.getNumericValue()) {
+            } else if (colorType == PngColorType.GREYSCALE.getNumericValue()) {
               directory.setColorMode("Grayscale");
-            } else if (colorType == PngColorType.GreyscaleWithAlpha.getNumericValue()) {
+            } else if (colorType == PngColorType.GREYSCALE_WITH_ALPHA.getNumericValue()) {
               numOfComponent = 2;
               directory.setColorMode("Grayscale");
               directory.setAlpha(true);
-            } else if (colorType == PngColorType.TrueColor.getNumericValue()) {
+            } else if (colorType == PngColorType.TRUE_COLOR.getNumericValue()) {
               numOfComponent = 3;
-            } else if (colorType == PngColorType.TrueColorWithAlpha.getNumericValue()) {
+            } else if (colorType == PngColorType.TRUE_COLOR_WITH_ALPHA.getNumericValue()) {
               numOfComponent = 4;
               directory.setAlpha(true);
             }

diff --git a/contrib/format-image/src/test/resources/image/eps.json b/contrib/format-image/src/test/resources/image/eps.json
@@ -62,7 +62,7 @@
     "RenderingIntent" : "Media-Relative Colorimetric",
     "XYZValues" : "0.964 1 0.825",
     "TagCount" : "10",
-    "Copyright" : "(c) 1999 Adobe Systems Inc.",
+    "ProfileCopyright" : "(c) 1999 Adobe Systems Inc.",
     "ProfileDescription" : "GBR",
     "MediaWhitePoint" : "(0.9505, 1, 1.0891)",
     "MediaBlackPoint" : "(0, 0, 0)",

diff --git a/contrib/format-image/src/test/resources/image/jpeg.json b/contrib/format-image/src/test/resources/image/jpeg.json
@@ -141,7 +141,7 @@
     "DeviceModel" : "sRGB",
     "XYZValues" : "0.964 1 0.825",
     "TagCount" : "17",
-    "Copyright" : "Copyright (c) 1998 Hewlett-Packard Company",
+    "ProfileCopyright" : "Copyright (c) 1998 Hewlett-Packard Company",
     "ProfileDescription" : "sRGB IEC61966-2.1",
     "MediaWhitePoint" : "(0.9505, 1, 1.0891)",
     "MediaBlackPoint" : "(0, 0, 0)",

diff --git a/contrib/format-image/src/test/resources/image/mov.json b/contrib/format-image/src/test/resources/image/mov.json
@@ -1,5 +1,5 @@
 {
-  "Format" : "MOV",
+  "Format" : "QUICKTIME",
   "Duration" : "01:32:3650",
   "PixelWidth" : "560",
   "PixelHeight" : "320",
@@ -31,7 +31,8 @@
     "SelectionTime" : "0",
     "SelectionDuration" : "0",
     "CurrentTime" : "0",
-    "NextTrackID" : "3"
+    "NextTrackID" : "3",
+    "Rotation" : "0"
   },
   "QuickTimeVideo" : {
     "CreationTime" : "Fri Jan 01 00:00:00 +00:00 1904",

diff --git a/contrib/format-image/src/test/resources/image/tiff.json b/contrib/format-image/src/test/resources/image/tiff.json
@@ -114,7 +114,7 @@
     "DeviceModel" : "sRGB",
     "XYZValues" : "0.964 1 0.825",
     "TagCount" : "17",
-    "Copyright" : "Copyright (c) 1998 Hewlett-Packard Company",
+    "ProfileCopyright" : "Copyright (c) 1998 Hewlett-Packard Company",
     "ProfileDescription" : "sRGB IEC61966-2.1",
     "MediaWhitePoint" : "(0.9505, 1, 1.0891)",
     "MediaBlackPoint" : "(0, 0, 0)",