title | slug | parent |
---|---|---|
Apache Drill 1.12.0 Release Notes |
Apache Drill 1.12.0 Release Notes |
Release Notes |
Release date: December 15, 2017
Today, we're happy to announce the availability of Drill 1.12.0. You can download it here.
This release of Drill provides the following new features and improvements:
- Kafka and OpenTSDB storage plugins (DRILL-4779, DRILL-5337)
- SSL/TLS support (DRILL-5431)
- Network encryption support (DRILL-5682)
- Queue-based memory assignment for buffering operators (DRILL-5716)
- A collection of networking functions that facilitate network analysis using Drill (DRILL-5834)
- Support for the libpam4j PAM authenticator (DRILL-5820)
- Filter pushdown for Parquet can handle files with multiple rowgroups (DRILL-5795)
- UTF-8 is enabled in the query string by default (DRILL-5772)
- IF NOT EXISTS support for CREATE TABLE and CREATE VIEWS (DRILL-5952)
- Geometry functions,
ST_AsGeoJSON
andST_AsJSON
, that return GeoJSON and JSON representations (DRILL-5962, DRILL-5960) - JMX metrics for failed and canceled queries (DRILL-5909)
- Syntax highlighting and error checking for storage plugin configurations (DRILL-5981)
- System options improvements, including a new internal system options table (DRILL-5723)
- Ability to prevent users from accessing a path outside the current workspace (DRILL-5964)
- Ability to put the server in quiescent mode for a graceful shutdown (DRILL-4286)
- The Drill Web UI lists the completion of successfully completed queries as "successful" (DRILL-5923)
The following sections list additional bug fixes and improvements:
- [DRILL-5253] - External sort fails with OOM error (Fails to allocate sv2)
- [DRILL-5801] - Label the Gantt chart that indicates the lifespan details of the various minor fragments
- [DRILL-5802] - Provide a sortable table for tables within a query profile
- [DRILL-5803] - Show the hostname for each minor fragment in operator table
- [DRILL-5867] - List profiles in pages rather than a long verbose listing
- [DRILL-5881] - Java Client: [Threat Modeling] Drillbit may be spoofed by an attacker and this may lead to data being written to the attacker's target instead of Drillbit
- [DRILL-5882] - C++ Client: [Threat Modeling] Drillbit may be spoofed by an attacker and this may lead to data being written to the attacker's target instead of Drillbit
- [DRILL-6031] - Document support for dots in field names
- [DRILL-1051] - Casting timestamp as date gives wrong result for dates earlier than 1883
- [DRILL-1499] - Different column order could appear in the result set for a schema-less select * query, even there are no changing schemas.
- [DRILL-3241] - Query with window function runs out of direct memory and does not report back to client that it did
- [DRILL-3407] - CTAS Auto Partition : The plan for count(*) should show the list of files scanned
- [DRILL-3449] - When Foreman node dies, the FragmentExecutor still tries to send status updates to Foreman
- [DRILL-3827] - Empty metadata file causes queries on the table to fail
- [DRILL-3829] - Metadata Caching : Drill should ignore a corrupted cache file
- [DRILL-4139] - Fix parquet partition pruning for BIT, INTERVAL and DECIMAL types
- [DRILL-4255] - SELECT DISTINCT query over JSON data returns UNSUPPORTED OPERATION
- [DRILL-4640] - Unable to submit physical plan from Web UI on Windows
- [DRILL-4686] - Aggregation query over HBase table results in IllegalStateException: Failure while reading vector
- [DRILL-4734] - Query against HBase table on a 5 node cluster fails with SchemaChangeException
- [DRILL-4735] - Count(dir0) on parquet returns 0 result
- [DRILL-5002] - Using hive's date functions on top of date column gives wrong results for local time-zone
- [DRILL-5146] - Unnecessary spilling to disk by sort when we only have 5000 rows with one column
- [DRILL-5166] - Select with options returns NPE
- [DRILL-5185] - Union all not passing type info when the output contains 0 rows
- [DRILL-5268] - SYSTEM ERROR: UnsupportedOperationException: Unable to get size for minor type [MAP] and mode [REQUIRED]
- [DRILL-5269] - SYSTEM ERROR: JsonMappingException: No suitable constructor found for type [simple type, class org.apache.drill.exec.store.direct.DirectSubScan]
- [DRILL-5327] - Hash aggregate can return empty batch which can cause schema change exception
- [DRILL-5339] - Web UI: flaws in query profile display on error
- [DRILL-5341] - Web UI: remove duplicating link to documentation in Options page
- [DRILL-5346] - Web UI: remove link on user in query profile list
- [DRILL-5357] - Partition pruning information not available in query plan for COUNT aggregate query
- [DRILL-5416] - Vectors read from disk report incorrect memory sizes
- [DRILL-5442] - Managed Sort: IndexOutOfBounds with a join over an inlist
- [DRILL-5443] - Managed External Sort fails with OOM while spilling to disk
- [DRILL-5445] - Assertion Error in Managed External Sort when dealing with repeated maps
- [DRILL-5447] - Managed External Sort : Unable to allocate sv2 vector
- [DRILL-5464] - Fix JSON reader when it deals with empty file
- [DRILL-5465] - Managed external sort results in an OOM
- [DRILL-5478] - Spill file size parameter is not honored by the managed external sort
- [DRILL-5480] - Empty batch returning from HBase may cause SchemaChangeException even when data does not have different schema
- [DRILL-5507] - Millions of "Failure finding Drillbit running on host" info messages in foreman logs
- [DRILL-5513] - Managed External Sort : OOM error during the merge phase
- [DRILL-5519] - Sort fails to spill and results in an OOM
- [DRILL-5525] - Inconsistent, unhelpful semantics for batch, field schema comparison
- [DRILL-5546] - Schema change problems caused by empty batch
- [DRILL-5547] - Drill config options and session options do not work as intended
- [DRILL-5564] - IllegalStateException: allocator[op:21:1:5:HashJoinPOP]: buffer space (16674816) + prealloc space (0) + child space (0) != allocated (16740352)
- [DRILL-5582] - [Threat Modeling] Drillbit may be spoofed by an attacker and this may lead to data being written to the attacker's target instead of Drillbit
- [DRILL-5594] - Excessive buffer reallocations during merge phase of external sort
- [DRILL-5597] - Incorrect "bits" vector allocation in nullable vectors allocateNew()
- [DRILL-5602] - Repeated List Vector fails to initialize the offset vector
- [DRILL-5617] - Spill file name collisions when spill file is on a shared file system
- [DRILL-5645] - negation of expression causes null pointer exception
- [DRILL-5660] - Drill 1.10 queries fail due to Parquet Metadata "corruption" from DRILL-3867
- [DRILL-5663] - Drillbit fails to start when only keystore path is provided without keystore password.
- [DRILL-5675] - Drill C++ Client Date Time Literals Support Metadata Mapping is Incorrect
- [DRILL-5686] - Warning for sasl.max_wrapped_size contain incorrect syntax
- [DRILL-5687] - Disable TestMergeJoinWithSchemaChanges#testMissingAndNewColumns
- [DRILL-5694] - hash agg spill to disk, second phase OOM
- [DRILL-5698] - Drill should start in embedded mode using java 1.8.0_144
- [DRILL-5699] - Drill Web UI Page Source Has Links To External Sites
- [DRILL-5701] - drill.connections.rpc.<user/control/data>.<encrypted/unencrypted> metric not behaving correctly
- [DRILL-5702] - Jdbc Driver Class not found
- [DRILL-5710] - drill-config.sh incorrectly exits with Java 1.7 or later is required to run Apache Drill
- [DRILL-5714] - Fix NPE when mapr-db plugin is used in table function
- [DRILL-5715] - Performance of refactored HashAgg operator regressed
- [DRILL-5717] - change some date time unit cases with specific timezone or Local
- [DRILL-5721] - Query with only root fragment and no non-root fragment hangs when Drillbit to Drillbit Control Connection has network issues
- [DRILL-5725] - Update Jackson version to 2.7.8
- [DRILL-5727] - Update release profile to generate SHA-512 checksum.
- [DRILL-5729] - Fix Travis Checks
- [DRILL-5737] - Hash Agg uses more than the allocated memory under certain low memory conditions
- [DRILL-5740] - hash agg fail to read spill file
- [DRILL-5743] - Using order by clause in a select * query on hbase table returns only the row_key and order by field(s)
- [DRILL-5744] - External sort fails with OOM error
- [DRILL-5745] - Invalid "location" information in Drill web server
- [DRILL-5746] - Pcap PR manually edited Protobuf files, values lost on next build
- [DRILL-5749] - Foreman and Netty threads occure deadlock
- [DRILL-5751] - Fix unit tests to use local file system even if it is not set by default
- [DRILL-5753] - Managed External Sort: One or more nodes ran out of memory while executing the query.
- [DRILL-5755] - TOP_N_SORT operator does not free memory while running
- [DRILL-5757] - CONVERT_TO_JSON function is failed while using non-existence field as a parameter.
- [DRILL-5761] - Disable Lilith ClassicMultiplexSocketAppender by default
- [DRILL-5763] - Fix NPE during MapRDBSubScan deserialization
- [DRILL-5765] - Json query profile is not shown on Web UI
- [DRILL-5766] - Stored XSS in APACHE DRILL
- [DRILL-5771] - Fix serDe errors for format plugins
- [DRILL-5775] - Select * query on a maprdb binary table fails
- [DRILL-5790] - PCAP format explicitly opens local file
- [DRILL-5792] - CONVERT_FROM_JSON on an empty file throws runtime exception
- [DRILL-5804] - External Sort times out, may be infinite loop
- [DRILL-5811] - Large number of "Failure finding Drillbit" messages when using MFS
- [DRILL-5812] - Restore the minor fragment Gantt chart Canvas
- [DRILL-5816] - Hash function produces skewed results on String values with same leading prefix
- [DRILL-5819] - Default value of security.admin.user_groups and security.admin.users is "true"
- [DRILL-5822] - The query with "SELECT *" with "ORDER BY" clause and `planner.slice_target`=1 doesn't preserve column order
- [DRILL-5824] - 1st phase Hash Aggregate allocates more memory than the limit
- [DRILL-5830] - Resolve regressions to MapR DB from DRILL-5546
- [DRILL-5838] - Fix MaprDB filter pushdown for the case of nested field (reg. of DRILL-4264)
- [DRILL-5839] - Handle Empty Batches in Merge Receiver
- [DRILL-5840] - A query that includes sort completes, and then loses Drill connection. Drill becomes unresponsive, and cannot restart because it cannot communicate with Zookeeper
- [DRILL-5845] - Columns returned by select with "ORDER BY" and "LIMIT" clauses are not in correct order.
- [DRILL-5849] - Add freemarker lib to dependencyManagement to ensure proper version is used when resolving dependency version conflicts
- [DRILL-5853] - Sort removal based on NULL direction
- [DRILL-5854] - IllegalStateException when empty batch is received.
- [DRILL-5857] - Fix NumberFormatException in Hive unit tests
- [DRILL-5859] - Time for query queuing timeout not display correctly in WebUI
- [DRILL-5863] - Sortable table incorrectly sorts minor fragments and time elements lexically instead of sorting by implicit value
- [DRILL-5864] - Selecting a non-existing field from a MapR-DB JSON table fails with NPE
- [DRILL-5872] - Deserialization of profile JSON fails due to totalCost being reported as "NaN"
- [DRILL-5874] - NPE in AnonWebUserConnection.cleanupSession()
- [DRILL-5878] - TableNotFound exception is being reported for a wrong storage plugin.
- [DRILL-5887] - Display process user/ groups in Drill UI
- [DRILL-5888] - jdbc-all-jar unit tests broken because of dependency on hadoop.security
- [DRILL-5890] - Tests Leak Many Open File Descriptors
- [DRILL-5895] - Fix unit tests for mongo storage plugin
- [DRILL-5896] - Handle vector creation in HbaseRecordReader to avoid NullableInt vectors later
- [DRILL-5898] - Query returns columns in the wrong order
- [DRILL-5906] - java.lang.NullPointerException while quering Hive ORC tables on MapR cluster.
- [DRILL-5923] - State of a successfully completed query shown as "COMPLETED"
- [DRILL-5941] - Skip header / footer logic works incorrectly for Hive tables when file has several input splits
- [DRILL-5972] - Slow performance for query on INFORMATION_SCHEMA.TABLE
- [DRILL-5979] - unnable to access temporary table from drill web interface
- [DRILL-5986] - Update jackson-databind version to 2.7.9.1
- [DRILL-5987] - Two versions of javassist on the classpath
- [DRILL-6006] - Label current is missing on Web UI since the same drillbits with different status are considered to be different
- [DRILL-6007] - web ui index page is refreshed at high pace
- [DRILL-6017] - Fix for SHUTDOWN button being visible for non Admin users
- [DRILL-6019] - Only admin should be able to access shutdown resources
- Order by doesn't sort columns when window function is involved in the query
- MIN MAX DIR tests fail and return incorrect results
- header.line.count does not work for Hive table with data file size > chunk size
- lang.NoClassDefFoundError exception not handled in org.apache.drill.exec.rpc.security.ClientAuthenticatorProvider
- Investigate jackson-databind vulnerabilities CVE-2017-15095 & CVE-2017-7525
- Null Pointer Exception with query using table function
- Fix XSS vulnerabilities in Drill
- Some queries in concurrent execution get stuck in STARTING phase
- hashagg.num_partitions cannot be set by "alter session" or "alter system"
- Query with 2-way JOIN fails with "Hash join does not support schema changes"
- RowKeyJoin is generated instead of IndexScan for queries with derived table/ table functions
- ConvertCountToDirectScan rule enhancements
- Drill wrong old date-time values displaying
- Warning message "Exception while trying to prune partition" when query on view
- Drill not able to read MapR-DB when FQDN is longer than 64 characters
- Pam authentication with Centrify doesn't work by JPam restriction
- Hash aggregate does not support schema changes
- Simple query with only one join condition failed with "Hash join does not support schema changes"
- UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL].
- Query on TPC-H SF100 dataset fails with "Hash join does not support schema changes" [MapR-DB JSON Tables]
- [DRILL-1691] - ConvertCountToDirectScan rule should be applicable for 2 or more COUNT aggregates
- [DRILL-4264] - Allow field names to include dots
- [DRILL-5089] - Skip initializing all enabled storage plugins for every query
- [DRILL-5106] - Refactor SkipRecordsInspector to exclude check for predefined file formats
- [DRILL-5259] - Allow listing a user-defined number of profiles
- [DRILL-5338] - Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile that they are part of total time duration
- [DRILL-5704] - Improve error message on client side when queries fail with "Failed to create schema tree." when Impersonation is enabled and logins are anonymous
- [DRILL-5709] - Provide a value vector method to convert a vector to nullable
- [DRILL-5716] - Queue-based memory assignment for buffering operators
- [DRILL-5726] - Support Impersonation without authentication for REST API
- [DRILL-5795] - Filter pushdown for parquet handles multi rowgroup file
- [DRILL-5808] - Reduce memory allocator strictness for "managed" operators
- [DRILL-5815] - Provide option to set query memory as percent of total
- [DRILL-5832] - Migrate OperatorFixture to use SystemOptionManager rather than mock
- [DRILL-5834] - Add Networking Functions
- [DRILL-5842] - Refactor and simplify the fragment, operator contexts for testing
- [DRILL-5862] - Update project parent pom xml to the latest ASF version
- [DRILL-5893] - Maven forkCount property is too aggressive causing builds to fail on some machines.
- [DRILL-5899] - Simple pattern matchers can work with DrillBuf directly
- [DRILL-5909] - need new JMX metrics for (FAILED and CANCELED) queries
- [DRILL-5910] - Logging exception when custom AuthenticatorFactory not found
- [DRILL-5921] - Counters metrics should be listed in table
- [DRILL-5943] - Avoid the strong check introduced by DRILL-5582 for PLAIN mechanism
- [DRILL-5952] - Implement "CREATE TABLE IF NOT EXISTS"
- [DRILL-5960] - Add function STAsGeoJSON to extend GIS support
- [DRILL-5962] - Add function STAsJSON to extend GIS support
- [DRILL-5964] - Do not allow queries to access paths outside the current workspace root
- [DRILL-5980] - Make queryType param for REST API case insensitive
- [DRILL-5981] - Add Syntax Highlighting and Error Checking to Storage Plugin Config Page
- [DRILL-5989] - Run Smoke Tests On Travis
- [DRILL-4286] - Have an ability to put server in quiescent mode of operation
- [DRILL-4779] - Kafka storage plugin support
- [DRILL-5337] - OpenTSDB storage plugin
- [DRILL-5431] - Support SSL
- [DRILL-5682] - Apache Drill should support network encryption
- [DRILL-5723] - Support System/Session Internal Options And Additional Option System Fixes
- [DRILL-5601] - Rollup of External Sort memory management fixes
- [DRILL-5685] - Provide a way to set common environment variable between sqlline and Drillbit differently.
- [DRILL-5758] - Rollup of external sort fixes to issues found by QA
- [DRILL-5772] - Enable UTF-8 support in query string by default
- [DRILL-5781] - Fix unit test failures to use tests config even if default config is available
- [DRILL-5820] - Add support for libpam4j Pam Authenticator
- [DRILL-5911] - Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json library
- [DRILL-5917] - Ban org.json:json library in Drill
- [DRILL-6000] - Graceful Shutdown Unit Tests Should Not Be Run On Travis
- [DRILL-6011] - Update log info message in DrillClient