[CALCITE-4673] Correlate variables deduplication is not triggered when custom table function presented. #2448
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…er COLLECTION_TABLE
zstan
changed the title
|
looks good |
|
@julianhyde @zabetak Checker failed with no additional info, plz help with it ? locally tests are ok. |
@zstan I restarted the jobs to see if it passes but in general you can run the CheckerFramework verifications locally by passing the
|
|
I'm doing final review. I made some cosmetic changes in https://github.com/julianhyde/calcite/tree/4673-correlated-table and am testing now. Will merge shortly. |
hannerwang
pushed a commit
to hannerwang/calcite
that referenced
this pull request
Aug 17, 2021
…bles, SqlToRelConverter should eliminate duplicate variables (Evgeniy Stanilovskiy) Close apache#2448
keatoooon
added a commit
to Bodo-inc/calcite
that referenced
this pull request
May 20, 2022
* [CALCITE-4606] In Elasticsearch adapter, translate SEARCH RexCall to termsQuery (Jacky Yin)
Translate SEARCH RexCall(In/NotIn) to termsQuery in ES.
Close apache/calcite#2420
* [CALCITE-4619] `Full join` plan cannot be executed in MySQL (Jiasen Sheng)
Close apache/calcite#2430
* [CALCITE-4636] Switch out of agg mode when constructing RelCollation for aggregate functions (Rafay)
When we create RelCollation for WITHIN GROUP clause, we try to find the node in the group set first, which returns a RexNode which references the current group set instead of the field in the input. If the field is present in the group set, its index may not match with the field index in the input node, hence we get a mismatch, and then we wrongly add a new expression in SqlToRelConverter.AggConverter#lookupOrCreateGroupExpr method.
The solution is to disable the agg mode, which will force the SqlToRelConverter.Blackboard#convertExpression function to return the correct RexNode referencing from the input, without looking in the grouping set.
* [CALCITE-4655] JdbcTable.scan throws NullPointerException (Ulrich Kramer)
Close apache/calcite#2441
* [CALCITE-4158] In generated SQL, "*" should be followed by space (NobiGo)
Close apache/calcite#2442
* Site: Add Vladimir Ozerov as committer
* Site: Deploy using .asf.yml
* [CALCITE-4614] Exasol dialect implementation (TJ Banghart)
Formatting; import SqlDialect.DatabaseProduct to make code more concise.
Close apache/calcite#2431
* [CALCITE-4644] Add PERCENTILE_CONT and PERCENTILE_DISC functions (Rafay)
In this change, we can parse and validate those functions,
but execution will be [CALCITE-4666].
Validation is based on new method SqlAggFunction.isPercentile()
(experimental and subject to change without notice).
There are connections between the RANGE clause in windowed
aggregate functions and interpolation used by PERCENTILE_CONT.
Therefore use the same logic to determine whether "x" is a
valid type in OVER (ORDER BY x RANGE r)" and
"PERCENTILE_CONT(f) WITHIN GROUP (ORER BY x)"
Close apache/calcite#2444
* [CALCITE-4638] VolcanoPlanner fails to recognize transformation rule correctly in the top-down mode (Vladimir Ozerov)
* [CALCITE-4668] RelBuilder.join should convert Correlate to Join if correlation variable is unused (James Starr)
Reworking the checking and validating if join is correlated in
RelBuilder.join to check if the correlated id is used on the right
side.If the correlated id is not used on the right side, then the join
is not correlated and a normal Join is emitted.
1. Updating ToLogicalConverterTest to generate a non trival Correlate
2. Adding tests for generating non trivial correlate joins to
RelBuilderTest.
3. Updating plans in sub-query.iq, that are now simplified.
Close apache/calcite#2446
* RelOptRulesTest improvements:
- Provide relFn pattern mechanism to test a RelNode function instead of a sql string
- Refactor several tests to use relFn
- Refactor similar tests by using common auxiliary methods
- Correct auxiliary methods names (use "check" prefix instead of "test" prefix)
* [CALCITE-4640] Propagate table scan hints to JDBC
close apache/calcite#2426
* [CALCITE-3775] Implicit lookup methods in SimpleCalciteSchema ignore
case sensitivity parameter (Alon Eldar)
rebased original PR #1792 from Feb2020.
* [CALCITE-4674] Excess quotes in generated SQL when STAR is a column alias
* [CALCITE-4675] Error executing query with SUM and multiplication via JdbcAdapter
Use EXPR$ prefix for naming columns when pushing projections
to avoid creating aliases with weird symbols such as '*'.
Such aliases may create problems in some DBMS such as Redshift
when the plan is converted to SQL via RelToSqlConverter.
* [CALCITE-4420] Some simple arithmetic operations can be simplified
* [CALCITE-4690] Error when executing query with CHARACTER SET in Redshift
Close apache/calcite#2457
* [CALCITE-4623] SemiJoinRule should not match SEMI join
* [CALCITE-4692] Redshift does not support DOUBLE/TINYINT datatypes (Narayanan Venkateswaran)
Close apache/calcite#2459
* Enable JdbcTest#testBushy and update expected plan (NobiGo)
The test runs reasonably fast and the current bushy plan is
the expected one.
Close apache/calcite#2462
* Site: Sort table of aggregate functions
* [CALCITE-4661] Add MODE aggregate function (NobiGo)
Close apache/calcite#2447
* [CALCITE-4642] Use RelDataTypeSystem from Config in Planner (Nick Riasanovsky)
Close apache/calcite#2435
* [CALCITE-4673] If arguments to a table function are correlation variables, SqlToRelConverter should eliminate duplicate variables (Evgeniy Stanilovskiy)
Close apache/calcite#2448
* Refactor RelToSqlConverterTest
* [CALCITE-4485] JDBC adapter generates invalid SQL when one of the joins is "INNER JOIN ... ON TRUE"
If the join tree is a mixture of cross-joins and outer joins,
RelToSqlConverter wrongly generates comma-join syntax.
Consider the (pseudo) RelNode tree
CrossJoin(a, LeftJoin(b, CrossJoin(c, d)))
Before this bug was fixed, RelToSqlConvert would generate
FROM a, b LEFT JOIN c, d
Because LEFT JOIN has higher precedence than ',', this is parsed to
LeftJoin(CrossJoin(a, b), CrossJoin(c, d))
which is incorrect. The fix is to only generate comma-join syntax
if all joins are cross-joins. In this case, we will generate
(pseudo SQL)
FROM a CROSS JOIN b LEFT JOIN c CROSS JOIN d
which is safe.
* [CALCITE-4706] JDBC adapter generates casts exceeding Redshift's data types bounds
1. Add Redshift type system ensuring precision/scale bounds are
satisfied when casting.
2. Consider max precision for DECIMAL and CHAR data types in
SqlDialect#getCastSpec.
3. Consider max scale for DECIMAL data types in SqlDialect#getCastSpec.
4. Extend RelToSqlConverterTest.Sql with custom type system to allow
writting tests with non-default type system (e.g., simulate systems with
large numeric precision).
5. Set typesystem explicitly in SqlDialectFactoryImpl since it cannot
be derived from the metadata and the default is not appropriate.
6. Add unit tests casting to DECIMAL, CHAR, VARCHAR, with
precision/scale exceeding Redshift's bounds.
Close apache/calcite#2470
* [CALCITE-4716] ClassCastException converting SARG in RelNode to SQL
Close apache/calcite#2474
* [CALCITE-4652] AggregateExpandDistinctAggregatesRule must cast top aggregates to original type (Taras Ledkov)
Close #2439
* Cosmetic
In RelToSqlConverterTest,
merge methods that test the same query in several dialects;
merge '.withXxx().ok(expectedXxx)' into one line,
and sort such lines alphabetically;
normalize 'expectedPostgreSQL' and 'expectedPostgreSql' to
'expectedPostgresql', and so forth.
* [CALCITE-4724] In JDBC adapter for ClickHouse, implement Values by generating SELECT without FROM (Liu Enze)
Fix by making SqlDialect.supportsAliasedValues return false
in the ClickHouse dialect, similar to MySQL.
Close apache/calcite#2480
* [CALCITE-4741] AbstractRelNode#getId can overflow into a negative value causing CompileException in certain Enumerable implement methods
* [CALCITE-4665] Allow Aggregate.groupKey to be a strict superset of Aggregate.groupKeys (NobiGo)
Generate correct SQL (by filtering using GROUPING_ID) when
the group key is a superset of the union of the group keys.
Close apache/calcite#2458
* [CALCITE-4748] If there are duplicate GROUPING SETS, Calcite should return duplicate rows (NobiGo)
Close apache/calcite#2503
* [CALCITE-4734] If there are duplicate RexNode in MutableCalc, SubstitutionVisitor should return right rebuild RexNode (xurenhe)
* [CALCITE-4584] Using function in partition by list of over window cause converting exception (Wang Yanlin)
* [CALCITE-4726] Support aggregate calls with a FILTER clause in AggregateExpandWithinDistinctRule (Will Noble)
Close apache/calcite#2483
* [CALCITE-4760] RelBuilder creation fails with error 'No suitable driver found for jdbc:calcite:' in shaded Calcite
This commit does not add tests; it's not easy to reproduce
issues caused by jar-packaging/shading.
* [CALCITE-4747]: clean all outdated edges from graph.edges
* Site: Add upcoming talk at ApacheCon'21 and info about tutorial at BOSS21
* [CALCITE-4766] Remove unreachable code from SqlValidatorImpl#performUnconditionalRewrites for VALUES node (Jianhui Dong)
Adapt the respective comment to reflect the current status and explain
briefly the history (by Stamatis Zampetakis).
Close apache/calcite#2511
* Site is not published due to bad yaml file suffix
* Add RelBuilder.lessThan, and use RelBuilder shorthands
* [CALCITE-4742] Implement "SOME <>" sub-query (NobiGo)
Close apache/calcite#2512
* [CALCITE-4774] Materialized view recognition fails for equivalent predicates (dz)
Close #2514
* [CALCITE-4767] JDBC adapter wrongly quotes backticks inside BigQuery identifiers (Jack Scott)
Currently escapes a backtick with another backtick, but
should escape a backtick with a backslash. Add method
SqlDialect.Config.withIdentifierEscapedQuoteString() so
that this is configurable.
Close apache/calcite#2527
* [CALCITE-4177] RelJson should throw if asked to deserialize a call to an unknown operator (Xzh & Wang Yanlin)
Close #2520
* [CALCITE-3935] Enhance Join-Materialization, support to pull-up filters under join of left or right (xurenhe)
* [CALCITE-4784] Ensure Correlate#requiredColumns is subset of columns in left relation
Enforce the implicit class invariant (see Correlate#getRequiredColumns)
via assertion to prevent construction of invalid plans and fail fast.
* Remove obsolete/misleading comments in RelOptUtil#classifyFilters
The claim that filters are not pushed in the NULL generating side is not
true (at least not anymore).
Consider the following query:
SELECT *
FROM emp e
LEFT JOIN dept d
ON d.deptno > 20
In this case the filter in the ON clause can be pushed in the right
which is the NULL generating side. This is already done by the
FilterJoinRule generating a plan similar to the query below.
SELECT *
FROM emp e
LEFT JOIN (SELECT * FROM dept d WHERE d.deptno > 20) ON true
* Site: Update jhyde's organization and add pronouns
* Site: Add michaelmior's pronouns
* Site: Add zabetak's pronouns
* Site: HTTP to HTTPS redirection is not working
Restore .htaccess file which was removed accidentally by commit
c1bf421c0816909fe53d557eb5ee012f779bfa11 after introducing an explicit
`include` directive. Add the .htaccess file in the `include` directive
to ensure that it is not lost when generating the website.
* [CALCITE-4486] UNIQUE sub-query (NobiGo)
Rewrite UNIQUE to EXISTS.
Close apache/calcite#2495
* [CALCITE-4789] Build is broken on Guava versions < 21
Fix build for Guava 19.0.
In Travis, test oldest and newest supported Guava versions.
(Previously we were only testing against the default Guava
version, 29.0-jre, and that's why we didn't notice the build
breakage.) Enable ErrorProne only on the newest Guava
version (because sometimes we use APIs that are marked Beta
in early versions and the Beta is removed in later versions).
Allow Guava version up to 30.1.1-jre. Lowest Guava version
is still 19.0. Default version is still 29.0-jre.
* [CALCITE-4796] Travis links in README.md should point to 'app.travis-ci.com' instead of 'travis-ci.org' (Alessandro Solimando)
* [CALCITE-4772] PushProjector should retain alias when handling RexCall (YuKong)
Close apache/calcite#2516
* [CALCITE-4544] Deprecate Metadata API backed by Java Reflection (James Starr)
Deprecating:
* RelOptPlanner.registerMetadataProviders - Used to support custom nodes
in reflection based rel metadata. The generated code based rel
metadata has tight couple so this is not needed.
* RelOptPlanner.getRelMetadataTimestamp - Used for cache invalidation in
reflection based rel metadata. The generated code based rel
metadata has tight couple so this is not needed.
* RelOptCluster.metadataFactory - Exposing an api for reflection based
rel metadata.
* RelNode.metadata - An api for accessing reflection based rel metadata.
* CachingRelMetadataProvider - Implements caching for reflection based
rel metadata. Generated code base rel metadata use a Table in
RelMetadataQuery and tight coupling for caching.
* RelMetadataProvider.apply - The primary entry point reflection based
rel metadata.
* MetadataFactory - An api for reflection based rel metadata.
* MockRelOptPlanner.setRelMetadataTimestamp - Used in testing reflection
based rel metadata.
* ReflectiveRelMetadataProvider.{map, metadataClass0 - Used in the
implementation reflection based rel metadata.
* VolcanoRelMetadataProvider - Supports custom volcanno rels in
reflection based rel metadata.
* HepRelMetadataProvider - Supports custom hep rels in reflection based
rel metadata.
Close apache/calcite#2475
* Replace deprecated com.google.common.io.Files.createTempDir() with java.nio.file.Files.createTempDirectory() in ElasticSearch tests
* [CALCITE-4793] CassandraAdapterDataTypesTest.testCollectionsInnerValues fails depending on the user timezone (Alessandro Solimando)
* [CALCITE-4773] RelDecorrelator's RemoveSingleAggregateRule can produce result with wrong row type
* [CALCITE-4656] Broken CI links on develop web page
* Site: Remove contributors name from commit summary
See discussion in dev@ list[1].
[1] https://lists.apache.org/thread.html/r0e198225f7912fa3d6eb910b9c271b2856cf156e87caadca5bf4cd8e%40%3Cdev.calcite.apache.org%3E
* Site: Add upcoming talk about Morel and update past talks section
* Site: Remove vote email from release instructions, and minor improvements
A draft email with the same content is printed by Gradle `prepareVote`
task so we can remove it from the site to avoid maintaining both
versions.
* [CALCITE-3409] Add a method in RelOptMaterializations to allow registering UnifyRule (xzh)
Close apache/calcite#2094
* [CALCITE-4532] Correct code generated for primitive-object ConstantExpression
* Site: Update fengzhu's organization and add pronouns
* [CALCITE-4779] GroupByList contains constant literal, materialized view recognition failed (xzh)
* [CALCITE-4790] Make Gradle pass the 'user.timezone' property to the test JVM (Alessandro Solimando)
* [CALCITE-4805] Calcite should convert a small IN-list as if the user had written OR, even if the IN-list contains NULL
Since [CALCITE-373], Calcite has converted "x IN (1, 2)" to
"x = 1 OR x = 2" but it still converts "x IN (1, NULL)" to
"x IN (VALUES 1, NULL)". Now that some bugs have been fixed,
there's no longer any reason to treat lists with NULL any
differently.
Close apache/calcite#2545
* [CALCITE-4787] Replace ImmutableBeans with Immutables in core module
- Move core rules to use Immutables instead of ImmutableBeans
- Update ImmutableBeans to ignore default property return values used
by Immutables
- Adjust EnumerableProjectToCalcRule so it doesn't try to use proxy
behavior with non-empty config
- Update MaterializedViewProjectAggregateRule.DEFAULT so it sets
fastBailout config property
- Update SqlValidator, SqlWriter, SqlParser, SqlToRelConverter,
RelDecorrelator, Hoist and RelBuilder to use Immutables for
configuration
- Switch gradle to use G1 Garbage collector to improve GC performance
- Update ConverterRule to have default properties since properties
were declared required but had no default values.
- Update gradle build to automatically generate annotation source on
Intellij and Eclipse sync operations
- Add travis_wait to jdk11 checkerframework build to avoid premature
termination due to core compile time.
- Make ExchangeRemoveConstantKeysRule and ValuesReduceRule consistent
with SubqueryRemoveRule by making MatchHandler type parameter concrete.
Co-authored-by: Vladimir Sitnikov <sitnikov.vladimir@gmail.com>
* Bump JUnit5 to 5.8.1
* Skip EqualsHashCode verification in ErrorProne: it is already verified with Checkstyle
* [CALCITE-4823] Suppress warnings for java.security.AccessController deprecation
JEP 411: Deprecate the Security Manager for Removal
see https://openjdk.java.net/jeps/411
* [CALCITE-4821] Move utility test classes into calcite-testkit and unpublish -test.jar
* Add missing @Override annotations
* Fix ErrorProne violations in testkit
* [CALCITE-4829] Bump Gradle to 7.2 and test with Java 17 at GitHub Actions
* Fix string reference to HrSchema in MaterializationTest with HrSchema.class.getName()
* Use jdk16 instead of jdk17 since jdk17 is not yet available at AppVeyor
* [CALCITE-4825] Move remaining core/main off of ImmutableBeans
This is a follow up to CALCITE-4787 to move the rest of the core/main off ImmutableBeans
* [CALCITE-4546] Change metadata dispatch to avoid registration of all RelNode subtypes (James Starr)
Close apache/calcite#2538
* [CALCITE-4819] SemiJoin operator is not skipped in materialized view-based rewriting algorithm
Close apache/calcite#2557
* [CALCITE-4817] Expand SubstitutionVisitor of Aggregate with max/min, which column is the group by list of target (xurenhe)
Close #2556
* [CALCITE-4798] Gradle build fails due to deprecated metadata APIs
Using fully qualified path for deprecated class instead of import to
fix the build. Apparently the problem appears only in certain JDKs
most likely those without:
https://bugs.openjdk.java.net/browse/JDK-8032211
Close apache/calcite#2540
* Bump JDK from 15 to 17 in seed build cache CI jobs
* [CALCITE-4834] JaninoRelMetadataProvider uses hardcoded class name
* [CALCITE-4830] Remove remaining uses of ImmutableBeans and deprecate
- Supersede and deprecate two-level nested Config classes to ensure all immutables are hidden
and carry consistene style.
- Remove the use of @Value.Style in non-test code to ensure consistent styling.
- Add annotation processing to all modules implementing new RelRule.Config subclasses
- Deprecate public methods and interfaces of ImmutableBeans
- Deprecate RelRule.Config.EMPTY
- Update javadocs for ImmutableBeans, RelRule, RelRule.CONFIG
- Add annotation processing to the core/test module
- Add compilation dependency on jsr305 to work around an Immutables bug when compiling against old Guava versions.
Note that during the development of this patch, kapt was evaluated extensively and ultimately ruled
out for use. The reasons included:
- Kapt is not compatible with JDK17: https://youtrack.jetbrains.com/issue/KT-45545
- Kapt causes deadlocks: https://youtrack.jetbrains.com/issue/KT-47853
Workarounds to those issues could not be made reliable.
* [CALCITE-4836] Upgrade protobuf-java 3.6.1 -> 3.17.1
* Revert "[CALCITE-4817] Expand SubstitutionVisitor of Aggregate with max/min, which column is the group by list of target (xurenhe)"
This reverts commit 74e97780add051cb71a122075e5bcbceb40e889c.
* Site: Add Zhaohui Xu as committer
* [CALCITE-3745] CompileException in UnitCompiler when using multiple class loaders
Janino is not able to determine the correct compiler for loading a
given class thus it fails with CompileException.
1. Use the class loader of the calling class instead of the default.
2. Upgrade Janino to 3.1.6 version which contains the fix for
https://github.com/janino-compiler/janino/pull/150 necessary for
passing correctly the class loader to the underlying methods.
3. Add/Update CheckerFramework .astub files for setParentClassLoader
method based on the changes in the Janino APIs. Methods have moved in
the new version so it is necessary to reflect the changes in the stub
files.
4. Enforce class loader is not null before calling
CompilerFactoryFactory#getDefaultCompilerFactory method. The parameter
does not have a nullability annotation so by default CheckerFramework
considers it as not-null. Indeed, if null is given as parameter a NPE
will be raised from within that method.
Close apache/calcite#2449
* [CALCITE-3583] Exchange operator deserialize fail when the RexInput has no RelCollcation (Xzh)
* [CALCITE-4811] Check if input type is a struct in RelDataTypeFactoryImpl#leastRestrictiveStructuredType
Close apache/calcite#2553
* [CALCITE-3338] Error with executeBatch and preparedStatement when using RemoteMeta (Xzh & Jin Xing)
* [CALCITE-4762] Upgrade Calcite to Avatica 1.19
(Following this commit are a few commits that add test cases
in Calcite for functionality that was fixed in Avatica 1.19.)
Close apache/calcite#2578
* [CALCITE-4600] ClassCastException retrieving from an ARRAY that has DATE, TIME or TIMESTAMP elements
Now that we have upgraded Calcite to Avatica 1.19, which
fixes [CALCITE-4600], we can add this test case to Calcite.
Close apache/calcite#2548
* [CALCITE-4602] ClassCastException retrieving from ARRAY that has mixed INTEGER and DECIMAL elements
Now that we have upgraded Calcite to Avatica 1.19, which
fixes [CALCITE-4602], we can add this test case to Calcite.
The fix is not perfect: the ClassCastException has gone, but
the resulting BigDecimal value is wrong; [CALCITE-4850] is
logged for that.
Close apache/calcite#2547
* [CALCITE-4757] In Avatica, support columns of type "NULL" in query results
The bug was fixed in Avatica, and here we add tests to
Calcite now that we have upgraded to Avatica 1.19.
Close apache/calcite#2507
* [CALCITE-4628] If SqlImplementor fails, include the RelNode in the exception
Add a test case for a bug that was fixed by "[CALCITE-4524]
Make some fields non-nullable (SqlSelect.selectList,
DataContext.getTypeFactory)".
* [CALCITE-4795] In class SqlBasicCall, make the "operands" field private (part 1)
In class SqlBasicCall, the operands field is currently a
public array. This gives too much freedom for clients to do
crazy things, and creates unexpected behavior because the
array is not copied when a SqlBasicCall is cloned. We would
like to make the first private, but that would be a breaking
change, so we must do it over two releases.
This is part 1: deprecate the "operands" field; after 1.28,
part 2 will make the "operands" field private, and also
change it from an array to an immutable List. You will still
be able to call SqlCall.setOperand, but the whole new List
will be created.
* [CALCITE-4847] Parse SQL with BigQuery-style quoted identifiers and character literals
In BigQuery, identifiers are quoted with backticks and an
included backtick is escaped with a backslash; character
literals are quoted with single-quotes or double-quotes and
an included quote is escaped with a backslash.
The parser enters this mode if you specify 'dialect=BIG_QUERY',
'lex=BIG_QUERY', or 'quoting=BACK_TICK_BACKSLASH' in the
connect string.
The connect string parameter 'quoting' previously allowed
values 'DOUBLE_QUOTE', 'BACK_TICK' and 'BRACKET', and now
also allows 'BACK_TICK_BACKSLASH'.
* [CALCITE-4616] AggregateUnionTransposeRule causes row type mismatch when some inputs have unique grouping key
Close apache/calcite#2437
* Refactor: Introduce field SqlUtil.GENERATED_EXPR_ALIAS_PREFIX
* [CALCITE-4740] JDBC adapter generates incorrect HAVING clause in BigQuery dialect
Close apache/calcite#2504
* [CALCITE-2736] ReduceExpressionsRule never reduces dynamic expressions but this should be configurable (Yingyu Wang)
1. Add new config option treatDynamicCallsAsConstant.
2. Update ReducibleExprLocator.analyzeCall() method to also rely on
treatDynamicCallsAsnConstant option when determining whether to
reduce a dynamic function.
Fixup following "[CALCITE-4830] Immutables" (Julian Hyde).
Close apache/calcite#2502
* [CALCITE-4550] Simplify JaninoRelMetadataProvider API for binding methods (James Starr)
Decoupling MetadataDef and JaninoMetadataProvider.
Reworking binding RelMdPercentageOriginalRows to handlers. Adding
DescriptiveCacheKey to replace the method in metadata cache list.
Adding a handlers method to RelMetadataProvider keys on the class
of MetadataHandler. Reworking RelMetadataQuery to not use defs.
Using MetadataHandler methods instead of Metadata methods in code
generation.
Close apache/calcite#2570
* [CALCITE-4848] Adding a HAVING condition to a query with a dynamic parameter makes the result always empty
* [CALCITE-3524] In RelBuilder, add methods for creating various kinds of sub-query
The methods are `in`, `exists`, `some`, `all`, `scalarQuery`, `unique`.
The methods all create and return a `RexSubQuery` object.
`RexSubQuery` is a scalar expression (a sub-class of `RexNode`)
and can therefore be used in a `Filter`, `Project`, etc.
Decorrelating or expanding the sub-query can be deferred,
and handled later by `SubQueryRemoveRule`.
Close apache/calcite#1690
Co-authored-by: Julian Hyde <jhyde@apache.org>
* [CALCITE-4719] Add variants of RexSubQuery that collect sub-queries into MULTISET, ARRAY and MAP collections
In RexSubQuery, add methods `array`, `map`, `multiset`.
In RelBuilder, add methods `arrayQuery`, `mapQuery`,
`multisetQuery`.
Close apache/calcite#2582
* [CALCITE-4735] Enhance Aggregate-Materialization, when aggcall of query could be expressed by target's grouping (xurenhe)
* [CALCITE-4853] Gradle could not determine the dependencies of task ':syncPreviewSiteRepo'
Upgrade vlsi-release-plugins to version 1.74 which has a workaround
for this error that comes from the upgrade to Gradle 7.0
* Site: Pronouns, talks
* Bump Guava maximum version up to 31.0.1-jre
* [CALCITE-4835] Release Calcite 1.28.0
* Add news item for release 1.28.0
Jacques' description of breaking changes is rather too long
for the release notes, so I added it to the news item, and
referenced it from the release notes.
Make other fixes to the release notes, such as the poorly
escaped '<>' in one commit message.
Add permanent anchors (qualified by release name) to the
release notes, so that #fixes for one release doesn't become
that.
Prepare for next development iteration by changing version
in gradle.properties.
Close apache/calcite#2584
* [CALCITE-4639] Broken links on download page
Close apache/calcite#2583
* [CALCITE-4856] Gradle prepareVote fails with 'not authorized'
Solve by documenting the `Pasf.git.pushRepositoryProvider=GITBOX`
flag. Without the flag, the `prepareVote` task pulls from
`github.com`, which no longer allows password-based authentication.
Add some sample git reports reports, run using `sqlsh`.
* [CALCITE-4839] Remove ImmutableBeans implementation and usage
Also remove superseded rule configuration interfaces
* [CALCITE-4795] In class SqlBasicCall, make the "operands" field private (part 2)
This commit is part 2 of 2, after release 1.28, and removes
APIs marked 'deprecated to be removed before 1.29'.
The 'operands' field is now an immutable list (that may
contain null values). If a user calls setOperand, a new list
is created. Therefore operands do not need to be cloned when
the call is cloned.
The 'expanded' field has been removed, because it is typically
false. There is a sub-class where isExpanded() returns true.
* Site: Add Xiong Duan as committer
* [CALCITE-4858] Use Log4j2 instead of unsupported Log4j (1.x) in tests
1. Replace slf4j-log4j12 dependency (using Log4j 1.x) with
log4j-slf4j-impl (using Log4j 2.x) when possible. Cannot remove
Log4j 1.x from Pig/Piglet modules due to compilation dependency of the
latter directly to log4j.
2. Use XML syntax instead of property syntax for Log4j configuration. A
Log4j configuration is hierarchical by nature so formats with natural
support for nesting (such as XML) are easier to use.
3. Exclude slf4j-log4j12 when it comes transitively from other
dependencies. It gets in conflict with log4j-slf4j-impl and it may also
appear transitively in projects using Calcite.
4. Add log4j-slf4j-impl in modules (kafka, mongodb, plus, example) to
silence error messages due to missing logger implementation.
5. Always use testRuntimeOnly annotation for dependencies on SLF4J
logger implementation. Calcite shouldn't enforce a specific logger
implementation so they should never appear as compile or runtime
dependencies. A concrete implementation is only needed when we want to
log messages for testing purposes.
6. Bump Cassandra version to 3.11.3 to use Log4j2 in tests. Previous
versions did not allow the use of any custom logger implementation
(only logback). This was resolved CASSANDRA-13396 so now we can use
Log4j2 as the rest of the tests.
7. Unify display pattern in elasticsearch log4j2.xml with the rest of
tests.
8. Add/Modify Log4j2 configuration (log4j2-test.xml) to silence errors
and not display logging messages in tests. Instead of globally turning
off logs, specific logger entries were set in the configuration files:
(i) to avoid hiding by accident important problems;
(ii) to faciliate switching log levels during debugging in the future.
Close apache/calcite#2587
* [CALCITE-4818] AggregateExpandDistinctAggregatesRule must infer correct data type for top aggregate calls (Taras Ledkov)
Close apache/calcite#2560
* Fix typo in reference.md
Close apache/calcite#2589
* [CALCITE-4822] Add functions ARRAY_CONCAT, ARRAY_REVERSE, ARRAY_LENGTH for BigQuery dialect
* [CALCITE-4551] Reusing Immutable metadata cache keys
* Changing RelMetadataQueryBase.map generic from Table<Object, List, Object>
to Table<Object, Object, Object> to support more efficient cache keys.
* Adding regression test generated RelMetdata code.
* Adding api status internal to DescriptiveCacheKey.
* Splitting code generation out of JaninoRelMetadataHandler to
RelMetadataHandlerGenerator.
* Moving DispatchGenerator to be packaged scoped.
* In DispatchGenerator, sorting RelNode class by name so the dispatch orders
is repeatable.
Close apache/calcite#2585
* [CALCITE-4768] Upgrade DataStax Driver for Apache Cassandra® version to latest 4.x
* Updated Cassandra from 3.11.2 to 4.0.1
* Updated Datastax Driver from 3.6.0 to 4.13.0
* Updated cassandra-unit from 3.5.0.1 to 4.3.1.0
* Cassandra tests are:
- now compatible with Guava >= 25 but are not anymore compatible with Guava < 20
- still incompatible with Eclipse OpenJ9 (due to Cassandra)
- still incompatible with JDK11+ (due to cassandra-unit)
* [CALCITE-4131] remove the XmlFunctions exception handled by System.out
* Bump com.github.vlsi.vlsi-release-plugins to 1.76
It resolves some of Gradle 7 warnings, and it hides some stacktraces (e.g. code style violations, compilation failure)
* Update Gradle to 7.3
Gradle 7.3 is the first release with the official support of Java 17
See https://docs.gradle.org/7.3/release-notes.html
* [CALCITE-4875] Preserve Operand Nullability in NVL rewrite
* [CALCITE-4877] Ignore test for invalid foreign schema class
* [CALCITE-4844] IN-list that references columns is wrongly converted to Values, and gives incorrect results
* [CALCITE-4877] Enable schema.iq test, with variants depending on Avatica version
Close apache/calcite#2608
* [CALCITE-4846] IN-list that includes NULL converted to Values throws exception
* [CALCITE-4884] Provide a new constructor for RelJsonWriter to allow customized JsonBuilder
* [CALCITE-4841] Support decimal column type in CSV file format
- Introduce a new `decimal` type in the CSV file format for
more accurate representation of decimal data.
- Implement the `decimal` SQL type using the BigDecimal JAVA type.
* [CALCITE-4876] Converting RelNode to SQL with CalciteSqlDialect gets wrong result while EnumerableIntersect is followed by EnumerableLimit
* Exclude kotlin-stdlib from :core runtime dependencies
* Clarify why squash commits option in GitHub PR merge is disabled
Unfortunately, the button loses committer info as follows (it uses GitHub noreply for committer):
commit a8e50734e0460e506f1762fbe0f628bcb444b8f5
Author: Denys Kuzmenko <dk...@cloudera.com>
AuthorDate: Tue Nov 30 10:09:06 2021 +0200
Commit: GitHub <no...@github.com>
CommitDate: Tue Nov 30 09:09:06 2021 +0100
* [CALCITE-4883] The traitset of Exchange operator losses the distribution when externalizing Exchange operator from JSON
* [CALCITE-4783] RelFieldTrimmer incorrectly drops filter condition
* [CALCITE-4851] Build gives lots of 'Execution optimizations have been disabled' warnings
* Use GitHub Action concurrency feature to cancel stale CI executions
See https://stackoverflow.com/a/68422069/253468
* Keep backslash when autoformatting <<...\n" +>>
We do not have multiline string literals in Java yet, so
the code convention is that \n should be at the end of the line,
in other words, the newline inside the literal should be paired
with a newline outside of the literal.
Java's Matcher#replaceAll treats \ in the replacement a special way, so
literal \ in the replacement string should be coded as \\\\ ¯\_(ツ)_/¯
* Log plan after physical tweaks in new line
* [CALCITE-4927] Remove deprecated RelBuilder#groupKey(ImmutableBitSet, ImmutableList) clashing with new replacement API
Keeping the deprecated API creates ambiguity and requires everybody to
upcast to Iterable if they want to use the new API.
Removing it on the other hand does not seem to affect much backward
compatibility (new API does not allow nulls) since callers will
automatically use the new API with no changes required in their code.
Close apache/calcite#2635
* [CALCITE-4930] Update log4j2 version to 2.15.0
* [CALCITE-4925] AggregateReduceFunctionsRule should accept arbitrary predicates
Allow selective application of the reduce-functions rule with arbitrary specificity.
Close apache/calcite#2634
* [CALCITE-4928] Decouple JaninoRelMetadataProvider from RelMetadataQuery
- Introduce a new interface: MetadataHandlerProvider.
- Move RelMetadataQuery to rely on MetadataHandlerProvider instead of Janino directly
- Update JaninoRelMetaDataProvider to implement MetadataHandlerProvider
- Deprecate several public (likely unused) surfaces related to the janino coupling
Co-authored-by: James Starr <jamesstarr@gmail.com>
* [CALCITE-4932] Deprecate JdbcCalc and remove JdbcCalcRule
Close apache/calcite#2642
* [CALCITE-4894] MV rewriting fails for conjunctive top expressions in SELECT clause
1. Remove conjuction decomposition in getExpressionLineage causing the
problem in the first place. Based on other unit tests in
RelMetadataTest the new behavior is the correct one.
2. Turn assertions into exceptions.
3. Improve error message for (possibly remaining) unhandled cases
where a single expression can have multiple lineages.
4. Add unit tests covering "complex" expressions in both views and
queries (highlighting the limitation).
Close apache/calcite#2613
* [CALCITE-4929] Implement default getDef() methods in MetadataHandler interfaces
- Add default getDef() methods.
- Mark all function metadata handler interfaces as FunctionalInterface where possible.
- Update MetdataDef and RelMetdataHandlerGeneratorUtil to ignore getDef() methods.
* [CALCITE-4937] Upgrade Calcite to Avatica 1.20
Fixes CVE-2021-44228.
* [CALCITE-4938] Upgrade SQLLine to 1.12.0
* [CALCITE-4939] Upgrade log4j2 version to 2.16.0
The 2.16.0 release has two additional fixes related to CVE-2021-4422 for
hardening security.
* Improve debug message in IterativeRuleDriver
Remove PLANNER logging from the message and better formulate at which
point during the rule matching the cost is printed.
The original message is misleading cause it is not the planner that is
printed but the queue. Moreover, IterativeRuleDriver#toString just
prints the class name and hashcode of the instance which is useless.
Class name might be useful but can be obtained by changing the logger
configuration (pattern layout).
* Remove duplicate entries from RelOptRules.CALC_RULES
It makes no difference for the planner since duplicate rules are
ignored.
* [CALCITE-4906] Wrong result for scalar subquery from empty input
* [CALCITE-4941] SemiJoinRule loses hints
* [CALCITE-4895] MAP type in UDF cannot be externalized from json correctly
Close apache/calcite#2614
* [CALCITE-4898] Upgrading Elasticsearch version from 7.0.1 to 7.15.2
* [CALCITE-4737] Add RelOptPlanner visualizer for debugging (Zuozhi Wang, Thomas Rebele)
* [CALCITE-4948] Revert Elasticsearch to 7.10.2
Remove extra maven repository.
* [CALCITE-4946] Add method RelBuilder.size()
Implement RelJson.toJson(Object) for an object of type
RelDataType.
Continuing the work started in [CALCITE-4719], implement
RexSubQuery.array and array query constructor, e.g.
SELECT ARRAY (SELECT ... FROM ...) AS lineItems
FROM Orders
Generalize Collect to be able to produce columns of type
ARRAY and MAP, not just MULTISET.
Add couple of test cases for USING.
* Finishing up [CALCITE-4937], remove workarounds for [CALCITE-4877]
* [CALCITE-4700] AggregateUnionTransposeRule produces wrong group sets for the top Aggregate (Vladimir Ozerov)
* [CALCITE-4704] Log plan on rule application using explain formatting
1. Add RuleEventLogger for logging events around rule execution.
2. Centralize logs around rule application and production in
RuleEventLogger and unify output for volcanoPlanner and HepPlanner
3. Use only id and operator name for displaying rule arguments; remove
redundant/not helpful information.
4. Add FULL_PLAN SLF4J marker to conrol display explain/tree logging on
produced expressions.
5. Add FULL_PLAN marker template config in tests.
Close apache/calcite#2469
* [CALCITE-4950] Upgrade log4j2 version to 2.17.0
The 2.17.0 release fixes a new security vulnerability CVE-2021-45105.
* Set timeout for running Druid tests in GitHub CI
When tests run successfully they don't take more than a few minutes.
Sometimes though they get stuck and run forever (till the max GitHub
timeout of 6h) so set a timeout to 10minutes to avoid wasting resources
and fail-fast.
* [CALCITE-4917] Add test for 'a IS NOT NULL AND a = b' simplification
* [CALCITE-4955] Release Calcite 1.29.0.
* Add 1.29.0 release announcement
* Prepare for next development iteration
* Revise release notes
Fix SHA in release news item and downloads page.
Add "[CALCITE-4547] Support Java 16 and 17" to list of bugs fixed in 1.29.
* [CALCITE-4547] Support Java 16 and 17
Close apache/calcite#2656
* Site: Use openjdk-17 to generate javadoc with docker
Use fixed docker image to avoid spurious changes during the website
generaion.
Close apache/calcite#2663
* [CALCITE-4952] Introduce a simplistic RelMetadataQuery option
- Introduce new ProxyingMetadataHandlerProvider that avoids code compilation
- Update RelMetadataTest to factor out metadata handling from tests
- Introduce tests for both Janino and the proxying/simplistic path.
- A simple microbenchmark for metadata
- Minor version bump to JMH
- Enable clearing of static cache within JaninoRelMetadataProvider
- Update RelMetadataTest to be parameterized
- Add test to confirm cyclic exception
* Site: Change the javadoc title to Apache Calcite API
* [CALCITE-4968] Use TOP N for MsSQL instead of FETCH without OFFSET
* [CALCITE-4963] Make it easier to implement interface SqlDialectFactory
To create a SqlDialect, you need an instance of
SqlDialect.Context, but the code to create this was until
now locked in class SqlDialectFactoryImpl. This change moves
that code into a public method
SqlDialects.createContext(DatabaseMetaData). It is now
easier to sub-class SqlDialectFactoryImpl or to directly
implement SqlDialectFactory.
Close apache/calcite#2658
* Remove unused package-private RelNullShuttle class
Close apache/calcite#2654
* [CALCITE-4960] Enable unit tests in Elasticsearch Adapter
1. Increase SocketTimeout and ConnectionTimeout for RestClient to avoid
CI failures.
2. Disable/Skip test cases with appropriate JIRA references.
Close apache/calcite#2659
* [CALCITE-4973] Upgrade log4j2 version to 2.17.1
The 2.17.1 contains a fix for CVE-2021-44832 (security vulnerability)
and other minor improvements.
* [CALCITE-3627] Incorrect null semantic for ROW function
* Update javacc official website
* Site: Improve HTML tables display & update CSV tutorial
1. Allow code pre-wrap in tables.
2. Display horizontal scrollbar in tables when content is too large
and cannot be wrapped.
3. Update CSV tutorial example based on current code.
Close apache/calcite#2632
* Site: Add syntax highlighting to SQL statements
Close apache/calcite#2632
* Site: For tables that display results, center the content horizontally
Close apache/calcite#2632
* [CALCITE-4953] Deprecate TableAccessMap class
The TableAccessMap class is not used anywhere in the project, it is
untested, and hasn't received any real update since its addition in the
project in 2012.
Close apache/calcite#2675
* Site: Fix typo in howto.md
* [CALCITE-4901] JDBC adapter incorrectly adds ORDER BY columns to the SELECT list of generated SQL query
Close apache/calcite#2665
* Site: Add Alessandro Solimando as committer
* [CALCITE-4977] Support Snapshot in RelMdColumnOrigins
* [CALCITE-4965] IS NOT NULL failed in Elasticsearch Adapter
* [CALCITE-4967] Support SQL hints for temporal table join
close apache/calcite#2664
* Corrected json stream property name
* [CALCITE-4986] Make HepProgram thread-safe
Before this change, HepProgram used mutable fields for its
working state and was therefore not thread-safe. If two
threads were planning queries simultaneously and are using
the same HepProgram instances they might conflict.
This change moves the mutable state out of HepProgram, so
that HepProgram is fully immutable and therefore thread-safe.
We make other fields final and not-nullable where possible.
Before using a HepProgram, you now need to call the new
HepProgram.prepare() method to create an instance of class
HepState (which contains the state for any program
instruction, including the whole program), then call the
method HepState.execute().
Close apache/calcite#2691
* [CALCITE-4991] Improve RuleEventLogger to also print input rels in FULL_PLAN mode
* [CALCITE-4885] Fluent test fixtures so that dependent projects can write parser, validator and rules tests
The goal is to be able to write tests from both inside and outside of
Calcite, and without deriving from a particular test class.
We achieve that goal by creating *fixture* objects for each kind of
test. Test configuration (e.g. what SQL query to test) belongs in that
fixture, and is set using wither methods.
Inside the fixture is a *factory* that creates the objects necessary
to run the test (parser, validator, and so forth). The factory
necssarily contains the config for the parser, validator, etc. Tests
configure the factory by calling methods in their fixture.
Also inside a fixture is a *tester* that orchestrates the lifecycle
(parse, validate, SQL-to-rel, check results). The tester is stateless
(necessary state is provided by parameters to its methods, which often
include a `SqlTestFactory`). There is one main implementation of the
tester, but it can be substituted with an radically different
implementations (say one that no-ops, or dumps all SQL expressions to
a file).
Tests that compare actual results with expected results may have a
`DiffRepository` attached to their fixture.
There follows descriptions of other refactorings included in this
change.
The code was deprecated in [CALCITE-4591], [CALCITE-4593],
[CALCITE-4446] to be removed before release 1.28.
Move classes `DiffRepository`, `MockRelOptPlanner`,
`SqlToRelTestBase`, `RelOptTestBase`, `SqlRuntimeTester`,
`SqlOperatorTest` (renaming to `CoreSqlOperatorTest`),
`SqlOperatorBaseTest` (renaming to `SqlOperatorTest`).
Class `Fixtures` is a single place to obtain fixtures to tests of the
parser, validator, operators, planner rules, and more. These fixtures
can be used within Calcite but also by any project that uses Calcite
(the project just needs to use the `testkit` module).
`class FixtureTest` tests `Fixtures` and contains examples.
Add `fixture()` methods, to create fixtures without SQL, to many test
classes including `RelToSqlConverterTest`, `InterpreterTest`,
`SqlParserTest`, `SqlValidatorTest`, `DruidAdapter2IT`,
`DruidAdapterIT`. (Previously many tests would write `sql("?")` to
create tests with dummy SQL.
In `class CalciteAssert`, move method `withRel` from `AssertQuery` to
`AssertThat`, so that tests can provide a `RelNode` without first
providing dummy SQL. Also make the list of hooks (used heavily by
`AssertQuery`) immutable.
In `SqlTester`, remove methods `checkFieldOrigin`, `checkResultType`,
`checkCollation`, `checkCharset`, `checkIntervalConv`,
`checkMonotonic`; all can now be implemented in terms of a new method
`validateAndThen`, and similar methods `validateAndApply` and
`forEachQueryValidateAndThen`.
Obsolete `interface SqlToRelTestBase.Tester`; `SqlToRelFixture` now
uses `SqlTester`.
Move inner `class MockViewExpander` from `SqlToRelTestBase` to
`SqlTestFactory`.
Method `SqlToRelTestBase.assertValid` becomes `Matchers.relIsValid`.
Refactor/rename a few methods, and change 'query' field to 'expression'.
State is now in the fixture (`class SqlParserFixture`) or
`SqlTestFactory`.
Create a fixture (`class SqlParserListFixture`) for list-based parser
tests.
Remove fixture's `transform` field; config is now transformed
immediately, not deferred.
Remove field `LINUXIFY` (thread-local state).
Fields `SqlToRelFixture.expression` and `SqlValidatorFixture.expression`
were each previously called `query` and had the opposite sense;
both are now consistent with `SqlParserFixture.expression`.
Rename method `sql` to `withSql`, `config` to `withConfig`, etc.
Rename method `withDecorrelation` to `withDecorrelate`,
`withLateDecorrelation` to `withLateDecorrelate`
`with(RelOptPlanner)` to `withPlanner`,
`with(HepProgram)` to `withProgram`.
`MaterializedViewTester` is now a utility class for materialized view
tests but is no longer required to be a base class.
Move `interface AbstractMaterializedViewTest.Sql` to top-level
`class MaterializedViewFixture`.
In `class MaterializedViewSubstitutionVisitorTest`, create a fixture
for satisfiability tests.
`RelSupplier` was previously an inner class of `class RelOptTestBase`.
It is used in metadata tests (`RelMetadataFixture`) and planner rule
tests (`RelOptFixture`) but could be used in other tests too.
Add `class RelSuppliers` with utilities and implementations for
`RelSupplier`.
Move `assertXxx` methods into a fixture, `class
SqlAdvisorTest.Fixture`.
Remove inner `class LexConfiguration`; it relied on a mutable tester
in each `SqlValidatorTest` instance, which is no longer there.
Implement methods `toString`, `equals`, `hashCode`.
Move inner `class Sql` to top-level `class SqlPrettyWriterFixture`.
Add `class ConnectionFactories`, utilities for `ConnectionFactory` and
`CalciteAssert.ConnectionPostProcessor`.
Enumerates all possible 'xxx' in JDBC `ResultSet.getXxx` and
`PreparedStatement.setXxx` methods.
Utilities for `interface SqlTester.ResultChecker`.
Move `interface SqlOperatorTest.Fixture` to `SqlOperatorFixture`.
In `class SqlOperatorFixture`, remove `double delta` arguments from
methods that test floating-point values; instead use a `ResultChecker`
created using a `ResultCheckers` method such as `isWithin` or
`isExactly`.
Remove method `withLibrary2`.
Move `class SqlToRelConverterTest.RelValidityChecker` to top-level.
In class `SqlLimitsTest`, move method `checkTypes` to
`class SqlTests`.
In a few cases we would built complex transforms (instances of
`UnaryOperator`) that were applied just before the test ran. Now we
try to apply transforms early, storing a transformed config rather
than a chained transform. It's easier to understand and debug.
Remove method `RelDataTypeSystemImpl.allowExtendedTrim`, which was
added in [CALCITE-2571] but was never used.
Remove some uses of `assert` in `TypeCoercionTest`. (Never use
`assert` in tests!)
Use `Assertions.assertEquals`, `Objects.requireNonNull` via static import.
In `class SqlValidator.Config`, rename method `sqlConformance()` to
`conformance()`, to be consistent with conformance properties
elsewhere.
In `class TryThreadLocal`, add methods `letIn(T, Supplier)` and
`letIn(T, Runnable)`.
Close apache/calcite#2685
* [CALCITE-4980] Babel parser support MySQL NULL-safe equal operator '<=>' (xurenhe&&DuanXiong)
* Site: Add "calcite-clj - Use Calcite with Clojure" in talks section
Close apache/calcite#2704
* Site: Add external resources section in the community page
Close apache/calcite#2703
* Site: update PMC Chair
* [CALCITE-4994] SQL-to-RelNode conversion is slow if table contains hundreds of fields
If a table contains hundreds or thousands of fields,
SQL-to-RelNode conversion is slow because the operation to
lookup a field within a record type is O(n) in the number of
fields and is called O(n) times. This manifests in the method
SqlToRelConverter.Blackboard.lookupExp.
The solution we adopted is to add a map in RelRecordType
from field names (case-sensitive) to fields (Julian Hyde).
We hope that this map will improve performance in other
parts of the planning process besides SQL-to-RelNode (e.g.
validation and rewrite rules).
If the record has 20 (RelRecordType.THRESHOLD) or fewer
fields, the map is not populated. We believe that this saves
memory and effort for the common case, small to medium-sized
records.
In SqlToRelConverter.Blackboard, change the contract of
method lookupExp. It would previously return null if a field
was not found, whereupon the caller would throw; now the
method throws, and is declared not-nullable. The method
previously returned a Map from field names to field ordinals,
and now returns a Function that can convert a field name to
an expression accessing that field; the new contract is
easier to implement efficiently with available knowledge.
Add a benchmark, RelNodeConversionBenchmark, that
demonstrates improvement for tables with 100 and 1,000
fields (Jay Narale).
Close apache/calcite#2701
Co-authored-by: Julian Hyde <jhyde@apache.org>
* [CALCITE-4702] Error when executing query with GROUP BY constant via JDBC adapter
Add new method in SqlDialect controlling whether GROUP BY using
literals is supported. Note that the whole Postgres family returns
false by precaution; some literals may be supported by Postgres or some
derivation of it. We agreed the extra complexity needed handle those
special cases was not worth it so we decided to return false for all
kinds of literals.
Introduce a new rule to rewrite the GROUP BY using an inner join with a
dummy table for those dialects that do not support literals.
Add a rule based transformation step at the beginning of the rel to SQL
conversion and ensure callers are passing from there. This allows to
keep the aggregate constant transformation in a single place.
Add tests with GROUP BY and different types of literals.
Close apache/calcite#2482
* [CALCITE-4872] Add UNKNOWN value to enum SqlTypeName, distinct from the NULL type
Before this change, the UNKNOWN type would become the NULL type
when switching nullability.
Close apache/calcite#2595
* [CALCITE-4996] In RelJson, add a readExpression method that can convert JSON to a RexNode expression
Previously `RelJson` was only able to deserialize `RexNode`
expressions that are part of a `RelNode` such as `Project`,
`Filter` or `Join`; references to input fields would always
be converted to a `RexInputRef`, referencing a field of the
input `RelNode`'s output row type.
But if the expression is not part of a `RelNode`, there is no
input `RelNode`. So this change adds `interface
InputTranslator` to specify how references to input fields
are to be translated.
In RuleMatchVisualizer, make method static, to appease lint.
Close apache/calcite#2709
* [CALCITE-4997] Keep APPROX_COUNT_DISTINCT in some SqlDialects
Some engines(Hive,Spark,BigQuery,Oracle,Snowflake) support APPROX_COUNT_DISTINCT function, while others do not.
So we can use the parameter SqlDialect#supportsApproxCountDistinct to control
whether to use APPROX_COUNT_DISTINCT(It is the same as APPROX_DISTINCT for Presto).
* [CALCITE-5006] Gradle tasks for launching JDBC integration tests are not working
1. Resolve the problem by removing the custom set of testClassesDirs
2. Set correctly the classpath to include JDBC drivers from a custom
configuration.
3. Add a bit of documentation around the JDBC tasks and configs.
Close apache/calcite#2716
* [CALCITE-5007] Upgrade H2 database version to 2.1.210
The 1.4.197 version suffers from multiple CVEs such as:
https://nvd.nist.gov/vuln/detail/CVE-2018-14335
https://nvd.nist.gov/vuln/detail/CVE-2018-10054
https://nvd.nist.gov/vuln/detail/CVE-2021-42392
https://nvd.nist.gov/vuln/detail/CVE-2022-23221
In the project, we use H2 only for testing purposes thus the H2
binaries are not present in the runtime classpath thus these CVEs do
not pose a problem for Calcite or its users.
Close apache/calcite#2717
* [CALCITE-4988] ((A IS NOT NULL OR B) AND A IS NOT NULL) can't be simplify to (A IS NOT NULL) When A is deterministic
* [CALCITE-4995] AssertionError caused by RelFieldTrimmer on SEMI/ANTI join
* Fix typo in filterable-model.yaml
* [CALCITE-4323] If a view definition has an ORDER BY clause, retain the sort if the view is used in a query at top level
The goal is for simple queries to retain the sorting. For example, given
CREATE VIEW v AS
SELECT *
FROM emp
WHERE job = 'CLERK'
ORDER BY deptno
the query
SELECT *
FROM v
WHERE ename LIKE 'M%'
will be expanded as if the user had written
SELECT *
FROM emp
WHERE ename LIKE 'M%'
AND job = 'CLERK'
ORDER BY deptno
But the ORDER BY will be removed from more complex queries
(e.g. those involving GROUP BY, UNION, sub-query) where it
would have no effect.
If the ORDER BY also includes LIMIT or OFFSET, it affects the
rows returned, not just their sort order; such an ORDER BY is
never removed.
Close apache/calcite#2679
* [CALCITE-4683] IN-list converted to JOIN throws type mismatch exception
* [CALCITE-4912] Confusing javadoc of RexSimplify.simplify
* [CALCITE-5011] CassandraAdapterDataTypesTest fails with initialization error
Cache Cassandra sessions based on "hostname, port, keyspace, username, password" information.
* [CALCITE-3673] ListTransientTable should not leave tables in the schema
[CALCITE-4054] RepeatUnion containing a Correlate with a transientScan on its RHS causes NPE
* [CALCITE-5008] Ignore synthetic and static methods in MetadataDef
According to https://www.jacoco.org/jacoco/trunk/doc/faq.html , Jacoco
adds a synthetic static method called $jacocoInit() during
instrumentation.
MetadataDef constructor breaks if it encounters such method on
the class given to it because it does not expect it. But it looks like
it simply should not consider synthetic methods (as well as static
methods) at all.
- extract code for finding handler methods
- reuse it in both places
- add tests concerning synthetic methods
* [CALCITE-5025] Upgrade commons-io version from 2.4 to 2.11.0
commons-io versions before 2.7 suffer from CVE-2021-29425 which allows
to traverse into the parent directory.
Update to a more recent version to avoid the afforementioned security
vulnerability and benefit from the other improvements in this library.
Close apache/calcite#2734
* [CALCITE-1794] Expressions with numeric comparisons are not simplified when CAST is present
* Site: Reorganise website update process in README & howto
1. Highlight the role & existence of two repos at the beginning of
README.
2. Provide a high-level overview of the update procedure early on.
3. Move RM related instructions from README to the appropriate howto
sections.
4. Remove git reset/rebase related commands for syncing master & site
branches from the beginning of the release process cause it is too
early to rebase then. Mention rebase later towards the end of the
process.
Close apache/calcite#2708
* [CALCITE-5030] Upgrade jsonpath version from 2.4.0 to 2.7.0
* Following [CALCITE-1794], update DruidDateTimeUtils and plans in Druid adapter
* [CALCITE-5019] Avoid multiple scans when table is ProjectableFilterableTable and projections and filters act on different columns
* [CALCITE-5040] SqlTypeFactoryTest.testUnknownCreateWithNullabilityTypeConsistency fails
[CALCITE-4872] added an UNKNOWN type that was implemented in some
places by class SqlTypeFactoryImpl.UnknownSqlType, but in others
by BasicSqlType. This would cause
SqlTypeFactoryTest.testUnknownCreateWithNullabilityTypeConsistency
to succeed or fail non-deterministically, depending on which of the
above had made it into the map of canonical type instances.
This commit solves the problem by removing class
SqlTypeFactoryImpl.UnknownSqlType and always using BasicSqlType.
* [CALCITE-4976] Release Calcite 1.30.0
* miniscule change
* removed removed function 'with' and created appropriate methods. Still having many import errors, but need to commit to flip branches
* removed old functions
* Fixed named parameter tests
* minor changes
* resolving various compilation errors
* updated our tests to work with the new testing infrastructure. Running into some rule inconsistency with autostyle checker?
* issues with .withNamedParameters() removing previous table data.
* gradle build passes
* possibly this fixes it?
* I'm dumb, wasn't in correct conda env
* removed redeclarations from named params, beacuse one of the calcite github tests found it to be errorprone. Also changed two function calls to static.
* changed some comments as they were not proper html, and change a non implememnted function to returned a non null
* updating ci files. Not sure why these weren't updated durring the merge
* updated to use the right branch name
* fixed style errors created
* fixed an error where named parameters where not typing properly when used within a case stmt
* changes to comments to be html compatible
Co-authored-by: jacky <jacky.yin@eoitek.com>
Co-authored-by: shengjiasen <angelzouxin@gmail.com>
Co-authored-by: Rafay <rafay@dremio.com>
Co-authored-by: Ulrich Kramer <u.kramer@sap.com>
Co-authored-by: NobiGo <nobigogle@gmail.com>
Co-authored-by: Stamatis Zampetakis <zabetak@gmail.com>
Co-authored-by: Michael Mior <mmior@cs.rit.edu>
Co-authored-by: TJ Banghart <tjbanghart@google.com>
Co-authored-by: Rafay <qureshi.rafay@gmail.com>
Co-authored-by: Vladimir Ozerov <ppozerov@gmail.com>
Co-authored-by: James Starr <jamesstarr@gmail.com>
Co-authored-by: rubenada <rubenql@gmail.com>
Co-authored-by: Alon Eldar <alon.eldar@neuroblade.ai>
Co-authored-by: liyafan82 <fan_li_ya@foxmail.com>
Co-authored-by: Narayanan Venkateswaran <vnhive@gmail.com>
Co-authored-by: Julian Hyde <jhyde@apache.org>
Co-authored-by: Nicholas J Riasanovsky <nick@bodo.ai>
Co-authored-by: zstan <stanilovsky@gmail.com>
Co-authored-by: Jesus Camacho Rodriguez <jesusca@microsoft.com>
Co-authored-by: tledkov <tledkov@gridgain.com>
Co-authored-by: Liu Enze <liuenze6516@gmail.com>
Co-authored-by: xurenhe <xurenhe19910131@gmail.com>
Co-authored-by: yanlin-Lynn <1989yanlinwang@163.com>
Co-authored-by: Will Noble <wnoble@google.com>
Co-authored-by: dongjianhui03 <dongjianhui03@meituan.com>
Co-authored-by: dz <953396112@qq.com>
Co-authored-by: Jack Scott <jackscott@google.com>
Co-authored-by: Alessandro Solimando <alessandro.solimando@gmail.com>
Co-authored-by: qianshen <xiongping.xp@alipay.com>
Co-authored-by: Grzegorz Gierlach <grzegorz.gierlach@hazelcast.com>
Co-authored-by: fenzhu <fenzhu@ebay.com>
Co-authored-by: Jacques Nadeau <jacques@apache.org>
Co-authored-by: Vladimir Sitnikov <sitnikov.vladimir@gmail.com>
Co-authored-by: Haisheng Yuan <h.yuan@alibaba-inc.com>
Co-authored-by: Bruce Irschick <brucei@bitquilltech.com>
Co-authored-by: snuyanzin <snuyanzin@gmail.com>
Co-authored-by: hannerwang <hannerwang@tencent.com>
Co-authored-by: yingyuwang <yingyu.w@gmail.com>
Co-authored-by: Thomas Rebele <thomas.rebele@gmail.com>
Co-authored-by: amir gajst <amir.gajst@sisense.com>
Co-authored-by: Jay <jay.narale@dremio.com>
Co-authored-by: chunwei.lcw <chunwei.lcw@alibaba-inc.com>
Co-authored-by: xwkuang5 <louisconfussing@gmail.com>
Co-authored-by: huzhe <iluffzhe@163.com>
Co-authored-by: liuyanze <yanze.lyz@antgroup.com>
Co-authored-by: Ada Wong <rsl4@foxmail.com>
Co-authored-by: Francesco Gini <francesco.gini@gmail.com>
Co-authored-by: Aleksey Plekhanov <plehanov.alex@gmail.com>
Co-authored-by: Rui Wang <amaliujia@apache.org>
Co-authored-by: Volodymyr Vysotskyi <vvovyk@gmail.com>
Co-authored-by: Marco Jorge <marco.jorge@outsystems.com>
Co-authored-by: Talat Uyarer <tuyarer@paloaltonetworks.com>
Co-authored-by: Guiyanakuang <guiyanakuang@gmail.com>
Co-authored-by: mans2singh <mans2singh@yahoo.com>
Co-authored-by: lmagic233 <mirai.senshi@outlook.com>
Co-authored-by: Jing Zhang <beyond1920@gmail.com>
Co-authored-by: Eugen Stan <eugen.stan@netdava.com>
Co-authored-by: Soumyakanti Das <soumyakanti.das.08@gmail.com>
Co-authored-by: Marieke Gueye <mariekes@google.com>
Co-authored-by: xiejiajun <jiajunbernoulli@foxmail.com>
Co-authored-by: 吴伟杰 <wuweijie@apache.org>
Co-authored-by: Roman Puchkovskiy <roman.puchkovskiy@gmail.com>
Co-authored-by: Scott Reynolds <sreynolds@twilio.com>
Co-authored-by: IceMimosa <chk19940609@gmail.com>
Co-authored-by: Keaton Drebes <keatontdrebes@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.