Skip to content

1.0.0

Latest
Latest

Choose a tag to compare

View all tags
@andygrove andygrove released this 07 Aug 20:11
· 40 commits to main since this release
1.0.0
3a7a2c4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

DataFusion Comet 1.0.0 Changelog

This release consists of 244 commits from 23 contributors. See credits at the end of this changelog for more information.

Fixed bugs:

  • fix: decline native V1 scans on object_store-unsupported filesystem schemes #4525 (schenksj)
  • fix: exclude release scratch dirs from RAT and license skill docs #4685 (andygrove)
  • fix(shuffle): tolerate non-UTF-8 bytes in get_string (lossy decode) #4524 (schenksj)
  • fix: decline CreateArray with struct-nullability-divergent children #4533 (schenksj)
  • fix: propagate nested cast errors #4675 (manuzhang)
  • fix: reject Parquet INT96 as TimestampNTZ on Spark 3.x #4357 (andygrove)
  • fix: gate str_to_map collations #4701 (manuzhang)
  • fix: repair broken rust-test build on main (init_datasource_exec arg mismatch) #4712 (andygrove)
  • fix: correct stale unix_timestamp NTZ and date_format codegen-default doc text (#4502) #4645 (andygrove)
  • fix: fall back for decimal SUM/AVG over sliding window frames (window audit) #4732 (andygrove)
  • fix: date_trunc schema mismatch and DST handling in non-UTC timezones #4761 (andygrove)
  • fix: size Iceberg delete files in the native scan to avoid dropping deletes #4760 (mbutrovich)
  • fix: gate non-default collations for Spark 4 datetime expressions #4693 (0lai0)
  • fix: ArrayInsert evaluation for null source arrays #4726 (peterxcli)
  • fix: support Spark 4 decimal window avg #4749 (manuzhang)
  • fix: array expression audit follow-ups (#4503) #4713 (andygrove)
  • fix: prevent wrong results from Iceberg native scan exchange reuse with different pushed filters #4812 (mbutrovich)
  • fix: count returning zero when scan is disabled and going through CometSparkColumnarToColumnar #4795 (Dummk0pf)
  • fix: handle null sub-arrays in flatten #4822 (michaelmitchell-bit)
  • fix: correct user guide URL in compatibility fallback messages #4854 (andygrove)
  • fix: report FromUnixTime non-default format as Unsupported #4847 (andygrove)
  • fix: emit JDK libjvm search path from core build script to fix -ljvm CI link failures #4868 (andygrove)
  • fix: register partitioning scalar subqueries for native shuffle to avoid "Subquery N not found" #4869 (mbutrovich)
  • fix: match Spark percentile interpolation precision #4792 (manuzhang)
  • fix: restrict array_filter array_compact fast path to the lambda variable #4848 (andygrove)
  • fix: suppress spurious WriteFilesExec fallback reason on native writes #4928 (andygrove)
  • fix: decode CAST(binary AS string) JVM-compatibly instead of unsafe reinterpret #4763 (andygrove)
  • fix: widen local table scan child nullability to match native kernels #4843 (andygrove)
  • fix: apply 1.11 Iceberg diff changes from #4991 to other versions #5020 (mbutrovich)
  • fix: raise CAST_INVALID_INPUT for invalid calendar dates in ANSI cast to date #5014 (andygrove)
  • fix: materialize ConstantColumnVector on Comet's serialize/export paths #4532 (schenksj)
  • fix: make native pow implementation compatible with Spark #5033 (andygrove)
  • fix: resolve Comet jar without hardcoding the SNAPSHOT qualifier #5108 (andygrove)
  • fix: honour Spark's legacy null IN () behavior #5127 (andygrove)
  • fix: honor fail_on_error in native make_decimal #5080 (andygrove)
  • fix: work around DataFusion 54.1.0 Parquet page-index regression #5132 (mbutrovich)
  • fix: throw ARITHMETIC_OVERFLOW for Long.MinValue div -1 under ANSI mode #5084 (andygrove)
  • fix: seed native Parquet scan reader options from session config #5107 (mbutrovich)
  • fix: raise REMAINDER_BY_ZERO for Float/Double under ANSI mode #5081 (andygrove)
  • fix: reduce log verbosity at per-task logging callsites #5155 (mbutrovich)
  • fix: round on Int64 with scale <= -19 now overflows correctly #5082 (andygrove)
  • fix: match Spark's whitespace trim semantics for casts from string to boolean, integral, float/double and decimal #5150 (andygrove)
  • fix: disambiguate Iceberg scans that share a metadata_location #5180 (mbutrovich)
  • fix: use per-expression eval mode for decimal promotion #5171 (peterxcli)
  • fix: codegen dispatcher null short-circuit swallowed ANSI errors, plus two latent TIME-type gaps #5219 (andygrove)
  • fix: count ReusedSubquery and CometSubqueryBroadcast correctly in extended explain #5206 (andygrove)

Performance related:

  • perf: O(1) PlanDataInjector lookup by op kind #4535 (schenksj)
  • perf: cache full Parquet metadata (incl. page index) via DataFusion's CachedParquetFileReaderFactory #4707 (mbutrovich)
  • perf: add metadata size hint to Parquet reader to match Iceberg path #4717 (mbutrovich)
  • perf(parquet): revise filter pushdown configuration #4722 (mbutrovich)
  • perf: unwrap identity casts in schema adapter to enable Parquet stats pruning #4730 (mbutrovich)
  • perf: avoid excessive timer calls #4739 (wForget)
  • perf: optimize spark_size in spark-expr #4877 (andygrove)
  • perf: optimize spark_unhex in spark-expr #4876 (andygrove)
  • perf: optimize parse_url (50x speedup) #4893 (andygrove)
  • perf: optimize to_json (2x faster) #4902 (andygrove)
  • perf: optimize spark_cast_int_to_int (100x faster) #4920 (andygrove)
  • perf: optimize try_arithmetic_kernel (up to 4x faster) #4910 (andygrove)
  • perf: optimize spark_arrays_overlap (up to 18x faster) #4906 (andygrove)
  • perf: optimize spark_lpad (up to 2x faster) #4919 (andygrove)
  • perf: optimize CheckOverflow with a shared no-overflow fast path (ANSI and non-ANSI) #4937 (andygrove)
  • perf: skip the null-masking pass in DecimalRescaleCheckOverflow when nothing overflows #4938 (andygrove)
  • perf: vectorize floating-point-to-decimal cast #4940 (andygrove)
  • perf: optimize spark_get_json_object (4x faster) #4907 (andygrove)
  • perf: optimize spark_regexp_extract #4894 (andygrove)
  • perf: vectorize spark_unscaled_value (9x faster) #4972 (u70b3)
  • perf: optimize cast_binary_to_string binary-format styles (up to 27x faster) #4912 (andygrove)
  • perf: optimize parse_string_to_decimal (30-40% faster) #4916 (andygrove)
  • perf: optimize date_parser / cast string to date (up to 2x faster) #4917 (andygrove)
  • perf: optimize cast_decimal128_to_utf8 in datafusion-comet-spark-expr #4924 (andygrove)
  • perf: optimize date_trunc (>2x faster) #4915 (andygrove)
  • perf: optimize spark_cast_float64_to_utf8 (~40% faster) #4918 (andygrove)
  • perf: dedupe Iceberg residuals and delete files in native scan serde #4982 (mbutrovich)
  • perf: optimize spark_ceil (3x faster) #4926 (andygrove)
  • perf: bypass shuffle BatchCoalescer for already-sized batches #5003 (andygrove)
  • perf: encode shuffle IPC schema once per writer instead of per block #5006 (andygrove)
  • refactor: drop redundant concat layer in single-partition shuffle #5004 (andygrove)
  • fix: make native cast from float/double to decimal compatible with Spark #5136 (andygrove)
  • perf: intern QueryContext SQL text into a per-plan pool (up to 20x smaller serialized plans for TPC-DS) #5204 (andygrove)
  • perf: avoid rebuilding untouched operators in PlanDataInjector.injectPlanData #5220 (andygrove)

Implemented enhancements:

  • feat: surface native parquet read failures as FAILED_READ_FILE #4536 (schenksj)
  • feat: opt concat into codegen dispatch for non-UTF8_BINARY collations #4640 (andygrove)
  • feat: opt sort_array into codegen dispatch under strict floating-point mode #4637 (andygrove)
  • feat: support MapType input for ElementAt #4697 (0lai0)
  • feat: extend native windows support #4209 (comphead)
  • feat: add array_prepend support #4716 (andygrove)
  • feat: surface native opt-in expressions as compatible-by-default with a COMET-INFO plan hint #4721 (andygrove)
  • feat: support StringSplitSQL for split_part #4592 (michaelmitchell-bit)
  • feat: support exact percentile and median aggregates natively #4542 (andygrove)
  • feat: support TimestampNTZ inputs natively for hour/minute/second #4753 (andygrove)
  • feat: add unsupported metadata column names to fallback reasons #4758 (hsiang-c)
  • feat: Add support for base64 #4778 (andygrove)
  • feat: support interval types and make_ym_interval / make_dt_interval #4541 (andygrove)
  • feat: support PreciseTimestampConversion for native batch time-window grouping #4784 (andygrove)
  • feat: support Azure authentication for native Parquet scan #4783 (Dummk0pf)
  • feat: support shuffle array expression #4797 (andygrove)
  • feat: rebalance associative bitwise/Add/Multiply chains to avoid protobuf recursion limit #4588 (schenksj)
  • feat: support grouping() and grouping_id() indicator functions #4815 (andygrove)
  • feat: Stage based fallback #4519 (karuppayya)
  • feat: release tokio runtime on driver/executor exit #4734 (wForget)
  • feat: route Unsupported through codegen dispatch for opt-in serdes #4728 (andygrove)
  • docs: comet docs design overhaul- phase 1 #4353 (pranamya123)
  • feat: name incompatible aggregate functions in mixed-execution fallback reason #4750 (andygrove)
  • feat: implement native empty2null spark inner function #4683 (kazantsev-maksim)
  • feat: enable mixed partial/final execution for sum and non-decimal avg #4861 (andygrove)
  • feat: Add experimental support for accelerated PyArrow UDFs #4234 (andygrove)
  • feat: support Iceberg 1.11, audit existing Iceberg diffs, bump iceberg-rust dep, add run-iceberg-tests CI trigger #4840 (mbutrovich)
  • feat: support approx_percentile / percentile_approx aggregate #4801 (andygrove)
  • feat: remove constraint on array of nested elements #4714 (hsiang-c)
  • feat: core SPI for contrib leaf scans (CometScanWithPlanData) [Delta contrib split, part 1] #4700 (schenksj)
  • feat: support gzip compression in native Parquet writes #4930 (andygrove)
  • feat: support size() for MapType input #4580 (marvelshan)
  • feat: hint at native pyarrow UDF path when the feature is disabled #4892 (andygrove)
  • feat: Implement TimeType support - Infrastructure - shuffle #4398 (YutaLin)
  • feat: fall back to Spark for collated predicate operands #4948 (comphead)
  • feat: add withAlternative alias mechanism for CometConf #4979 (andygrove)
  • feat: add spark.comet.shuffle.maxBufferBytes to cap native shuffle writer memory #4989 (andygrove)
  • feat: support approx_count_distinct aggregate expression #4819 (andygrove)
  • feat: Iceberg table format V3: native table decryption, fall back for other V3 features #4991 (mbutrovich)
  • feat: support multiply_dt_interval with codegen dispatch #4900 (peterxcli)
  • feat: Support Spark levenshtein expression in native execution #4105 (Myx778)
  • feat: native collect_list / array_agg aggregate #4720 (andygrove)
  • feat: add CalendarIntervalType support #4898 (peterxcli)
  • feat: native randstr implementation compatible with Spark #5035 (andygrove)
  • feat: support interval codegen dispatch for nested values and native shuffle #4976 (peterxcli)
  • feat: add codegen dispatch fallback for CometCast incompatible/unsupported cases (including legacy config paths) #5079 (comphead)
  • feat: support SampleExec natively for sampling without replacement #5110 (andygrove)
  • feat: disable native columnar-to-row conversion by default #5114 (andygrove)
  • feat: expose Comet version as spark.comet.version runtime config #5049 (andygrove)
  • feat: native uuid() implementation compatible with Spark #5034 (andygrove)
  • feat: support Iceberg metadata columns _pos, _spec, _file, and _partition #4752 (parthchandra)
  • feat: report native vs codegen-dispatch expression coverage in extended explain #5201 (andygrove)

Documentation updates:

  • docs: update release_process for changelog #4668 (mbutrovich)
  • docs: restore 0.13 user guide dropped in 0.17 release #4704 (andygrove)
  • docs: fix prettier check error #4748 (wForget)
  • docs: refresh expression audit notes for resolved correctness issues #4762 (andygrove)
  • docs: mark try_avg and try_sum as natively supported #4776 (andygrove)
  • docs: update Iceberg docs to clarify S3 storage credentials #4767 (mbutrovich)
  • docs: render captioned sidebar sections for versioned user guides #4699 (andygrove)
  • docs: triage bug vs enhancement and apply type labels consistently #4768 (andygrove)
  • docs: mark make_dt_interval and make_ym_interval as supported #4790 (andygrove)
  • docs: correct window function support status and limitations #4833 (andygrove)
  • docs: minor docs site improvements #4855 (andygrove)
  • docs: remove references to closed and fixed issues from compatibility guide #4856 (andygrove)
  • docs: fix sidebar drawer mis-tap, missing aria-expanded, homepage permalink, and ASF links page #4858 (pranamya123)
  • docs: announce JDK 11 and Spark 3.4 deprecation for 1.1.0 removal #4857 (andygrove)
  • docs: note invalid UTF-8 scan limitation and FFI import risk #4846 (andygrove)
  • docs: update Gluten comparison with AWS Labs benchmark and Comet strengths #4873 (andygrove)
  • docs: add scalar expression optimization guide, skill, and performance audits #4933 (andygrove)
  • doc: Document scan tuning for spark.sql.files.maxPartitionBytes #4931 (comphead)
  • docs: add 0.17.1 changelog #4961 (mbutrovich)
  • docs: document Spark version adoption and support-lifetime policy #4977 (andygrove)
  • docs: show implementation kind for each expression #5028 (andygrove)
  • docs: add blog posts and talks page #5043 (andygrove)
  • docs: avoid "not a blocker" comments in reviews for review skill #5057 (mbutrovich)
  • docs: refresh stale issue references and normalize issue link format #5062 (andygrove)
  • docs: fix references to configuration keys that do not exist #5063 (andygrove)
  • docs: define post-1.0 versioning policy and legacy config process #5056 (andygrove)
  • docs: expand tuning guide with performance and memory configs #4908 (andygrove)
  • docs: document known correctness issues in the compatibility guide #5085 (andygrove)
  • docs: update post 1.0.0 roadmap #5064 (mbutrovich)
  • docs: drop compatibility notes for bugs that are now fixed #5154 (andygrove)
  • docs: document ReusedExchange caveat and both CometSparkToColumnar names in operator-count exclusions #5240 (andygrove)

Other:

  • chore: fix generate-release-docs.sh for per-Spark-version doc layout #4662 (mbutrovich)
  • chore: add branch protection to release branches, update release_process.md #4665 (mbutrovich)
  • chore: start 0.18.0 development #4664 (mbutrovich)
  • test: cover nested complex casts #4608 (manuzhang)
  • refactor: move string expression support checks to getSupportLevel #4676 (andygrove)
  • refactor: move arithmetic and math support checks to getSupportLevel #4674 (andygrove)
  • refactor: move aggregate expression support checks to getSupportLevel #4678 (andygrove)
  • refactor: move array expression support checks to getSupportLevel #4677 (andygrove)
  • chore: add array tests with NaN handling #4686 (comphead)
  • chore: tweak CI execution memory params #4687 (comphead)
  • chore: add ordering tests for array_union #4688 (comphead)
  • chore: fix ConstantFolding rule exclusion for benchmarks #4689 (comphead)
  • chore(deps): bump actions/checkout from 6 to 7 #4690 (dependabot[bot])
  • chore(deps): bump the all-other-cargo-deps group in /native with 2 updates #4691 (dependabot[bot])
  • chore(deps): bump itertools from 0.14.0 to 0.15.0 in /native #4692 (dependabot[bot])
  • chore: update documentation links for 0.17.0 release #4698 (andygrove)
  • chore: add optional CI flow for parquet writes #4696 (comphead)
  • deps: Upgrade to DataFusion 54.0.0 #4062 (andygrove)
  • chore: apply same SBT profile for Spark 4.0 #4709 (comphead)
  • refactor: remove map_contains_key serde #4703 (manuzhang)
  • chore: Spark 4.0 restore DEDICATED_JVM_SBT_TESTS #4711 (comphead)
  • refactor: share Spark 4.1+ shim sources to remove duplication #4710 (andygrove)
  • chore(deps): bump object_store from 0.13.2 to 0.14.0 in /native #4737 (dependabot[bot])
  • chore(deps): bump the all-other-cargo-deps group in /native with 3 updates #4736 (dependabot[bot])
  • chore(deps): bump actions/cache from 5 to 6 #4735 (dependabot[bot])
  • deps: revert object_store to 0.13.2 #4740 (mbutrovich)
  • chore: fix 4.0.2 diff to handle CometWindowsExec properly #4743 (comphead)
  • chore: surface DataFusion 54 PruningMetrics and Ratio in CometNativeScan metrics #4733 (mbutrovich)
  • refactor: gate CometUnaryMinus input types in getSupportLevel #4759 (andygrove)
  • test: enable literal sha2 now that native engine handles scalar args #4773 (andygrove)
  • ci: republish docs site when generated-doc sources change #4777 (andygrove)
  • test: enable SparkSessionExtensionSuite with Comet #4772 (andygrove)
  • chore: start 1.0.0 development #4794 (andygrove)
  • chore(deps): bump rand from 0.10.1 to 0.10.2 in /native in the all-other-cargo-deps group #4805 (dependabot[bot])
  • deps: bump iceberg-rust version to a rev with ArrowReader fix #4826 (sandugood)
  • chore: Use native impl of soundex function #4824 (kazantsev-maksim)
  • refactor(shuffle): Introduce PartitionWriter interface to decouple shuffle partitioning from storage #4779 (wForget)
  • chore: fallback for spark.sql.mapKeyDedupPolicy == LAST_WIN #4863 (comphead)
  • chore: remove hdfs Comet crate #4904 (comphead)
  • test: cover invalid unhex inputs in SQL file tests #4890 (andygrove)
  • chore: Codegen fallback explain for plan #4891 (comphead)
  • test: add version check on an Iceberg test for Variant type #4935 (mbutrovich)
  • chore: use Datafusion substring #4161 (comphead)
  • chore: Support native writer Overwrite and ErrorIfExists modes #4946 (comphead)
  • refactor: move AttributeReference and KnownFloatingPointNormalized expression support checks to getSupportLevel #4745 (peterxcli)
  • chore(deps): bump actions/setup-node from 6 to 7 #4955 (dependabot[bot])
  • chore: bump spark-4.2 profile to the released 4.2.0 #4960 (andygrove)
  • chore(deps): bump the all-other-cargo-deps group across 1 directory with 6 updates #4956 (dependabot[bot])
  • chore(deps): bump lz4_flex from 0.13.1 to 0.14.0 in /native #4958 (dependabot[bot])
  • chore: codeql dependabot fix #4962 (comphead)
  • chore(deps): bump the codeql-actions group with 2 updates #4970 (dependabot[bot])
  • test: promote try_to_date/try_to_timestamp SQL tests to native coverage #4973 (u70b3)
  • chore: remove dead Parquet parallel-IO configs #4981 (andygrove)
  • refactor: remove untested async columnar shuffle #4985 (andygrove)
  • deps: bump to datafusion 54.1.0 and latest iceberg-rust #4996 (mbutrovich)
  • chore: use DF array_repeat and array_compact #4741 (comphead)
  • chore: remove legacy use.lazyMaterialization #4998 (kazuyukitanimura)
  • chore(deps): bump actions/stale from 10.3.0 to 10.4.0 #4879 (dependabot[bot])
  • chore(deps): bump base64 from 0.22.1 to 0.23.0 in /native #5018 (dependabot[bot])
  • chore(deps): bump actions/setup-python from 6 to 7 #5017 (dependabot[bot])
  • chore(deps): bump the codeql-actions group with 2 updates #5016 (dependabot[bot])
  • chore(deps): bump actions/cache from 5 to 6 #4882 (dependabot[bot])
  • chore(deps): bump actions/checkout from 6 to 7 #4881 (dependabot[bot])
  • refactor: group orphan spark.comet.explain.* configs under one prefix #5026 (andygrove)
  • refactor: unify shuffle configs under spark.comet.shuffle.* prefix #4986 (andygrove)
  • test: add isolated and small-batch benchmarks for columnar-to-row conversion #5113 (andygrove)
  • chore: stop enabling incompatible casts in plan stability suite #5139 (andygrove)
  • chore: replace deprecated symbol literals with $"col" in tests #5168 (andygrove)
  • test: cover ArrayExists three-valued logic config #5000 (manuzhang)
  • chore(deps): bump actions/stale from 10.4.0 to 11.0.0 #5163 (dependabot[bot])
  • chore: remove Array -> Seq round trips flagged by the 2.13 copy deprecation #5170 (andygrove)
  • chore: supply explicit empty argument lists for 2.13 #5175 (andygrove)
  • chore: clean up Scala compiler warnings #5141 (andygrove)
  • chore: drop deprecated .toIterable from protobuf builder calls #5173 (andygrove)
  • chore: bump Spark 3.5 to 3.5.9 #5181 (andygrove)
  • chore: bump Spark 4.1 to 4.1.3 #5183 (andygrove)
  • refactor: use arrow make_comparator for nested structural equality in arrays_overlap and array_position #5176 (peterxcli)
  • chore: bump Spark 4.0 to 4.0.4 #5182 (andygrove)
  • chore: remove unused spark.comet.exceptionOnDatetimeRebase config #5221 (andygrove)
  • refactor: rename pyarrowUdf config to pyarrowUDF and fix stale config docs #5197 (andygrove)
  • chore: cargo update for 1.0 release #5226 (mbutrovich)

Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

   132	Andy Grove
    28	Matt Butrovich
    19	Oleks V
    18	dependabot[bot]
     8	Scott Schenkein
     7	Manu Zhang
     7	Peter Lee
     4	Zhen Wang
     2	ChenChen Lai
     2	Kazantsev Maksim
     2	Prames Maanikam
     2	Pranamya Vadlamani
     2	hsiang-c
     2	kid
     1	Bolin Lin
     1	KAZUYUKI TANIMURA
     1	Karuppayya
     1	Mitchell
     1	Myx778
     1	Parth Chandra
     1	Zaki
     1	alexander domenti
     1	michaelmitchell-bit

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.

Assets 2
Loading