[CALCITE-4677] Fix different physical conventions conversion on new physical node producing#2452
Closed
chenzl25 wants to merge 51 commits intoapache:mainfrom
chenzl25:master
Closed
[CALCITE-4677] Fix different physical conventions conversion on new physical node producing#2452chenzl25 wants to merge 51 commits intoapache:mainfrom chenzl25:master
chenzl25 wants to merge 51 commits intoapache:mainfrom
chenzl25:master
Conversation
chenzl25
changed the title
Contributor
Author
done |
1. Allow code pre-wrap in tables. 2. Display horizontal scrollbar in tables when content is too large and cannot be wrapped. 3. Update CSV tutorial example based on current code. Close #2632
Use fixed docker image to avoid spurious changes during the website generaion. Close #2663
- Introduce new ProxyingMetadataHandlerProvider that avoids code compilation - Update RelMetadataTest to factor out metadata handling from tests - Introduce tests for both Janino and the proxying/simplistic path. - A simple microbenchmark for metadata - Minor version bump to JMH - Enable clearing of static cache within JaninoRelMetadataProvider - Update RelMetadataTest to be parameterized - Add test to confirm cyclic exception
To create a SqlDialect, you need an instance of SqlDialect.Context, but the code to create this was until now locked in class SqlDialectFactoryImpl. This change moves that code into a public method SqlDialects.createContext(DatabaseMetaData). It is now easier to sub-class SqlDialectFactoryImpl or to directly implement SqlDialectFactory. Close #2658
1. Increase SocketTimeout and ConnectionTimeout for RestClient to avoid CI failures. 2. Disable/Skip test cases with appropriate JIRA references. Close #2659
The 2.17.1 contains a fix for CVE-2021-44832 (security vulnerability) and other minor improvements.
The TableAccessMap class is not used anywhere in the project, it is untested, and hasn't received any real update since its addition in the project in 2012. Close #2675
…SELECT list of generated SQL query Close #2665
Before this change, HepProgram used mutable fields for its working state and was therefore not thread-safe. If two threads were planning queries simultaneously and are using the same HepProgram instances they might conflict. This change moves the mutable state out of HepProgram, so that HepProgram is fully immutable and therefore thread-safe. We make other fields final and not-nullable where possible. Before using a HepProgram, you now need to call the new HepProgram.prepare() method to create an instance of class HepState (which contains the state for any program instruction, including the whole program), then call the method HepState.execute(). Close #2691
…ite parser, validator and rules tests The goal is to be able to write tests from both inside and outside of Calcite, and without deriving from a particular test class. We achieve that goal by creating *fixture* objects for each kind of test. Test configuration (e.g. what SQL query to test) belongs in that fixture, and is set using wither methods. Inside the fixture is a *factory* that creates the objects necessary to run the test (parser, validator, and so forth). The factory necssarily contains the config for the parser, validator, etc. Tests configure the factory by calling methods in their fixture. Also inside a fixture is a *tester* that orchestrates the lifecycle (parse, validate, SQL-to-rel, check results). The tester is stateless (necessary state is provided by parameters to its methods, which often include a `SqlTestFactory`). There is one main implementation of the tester, but it can be substituted with an radically different implementations (say one that no-ops, or dumps all SQL expressions to a file). Tests that compare actual results with expected results may have a `DiffRepository` attached to their fixture. There follows descriptions of other refactorings included in this change. The code was deprecated in [CALCITE-4591], [CALCITE-4593], [CALCITE-4446] to be removed before release 1.28. Move classes `DiffRepository`, `MockRelOptPlanner`, `SqlToRelTestBase`, `RelOptTestBase`, `SqlRuntimeTester`, `SqlOperatorTest` (renaming to `CoreSqlOperatorTest`), `SqlOperatorBaseTest` (renaming to `SqlOperatorTest`). Class `Fixtures` is a single place to obtain fixtures to tests of the parser, validator, operators, planner rules, and more. These fixtures can be used within Calcite but also by any project that uses Calcite (the project just needs to use the `testkit` module). `class FixtureTest` tests `Fixtures` and contains examples. Add `fixture()` methods, to create fixtures without SQL, to many test classes including `RelToSqlConverterTest`, `InterpreterTest`, `SqlParserTest`, `SqlValidatorTest`, `DruidAdapter2IT`, `DruidAdapterIT`. (Previously many tests would write `sql("?")` to create tests with dummy SQL. In `class CalciteAssert`, move method `withRel` from `AssertQuery` to `AssertThat`, so that tests can provide a `RelNode` without first providing dummy SQL. Also make the list of hooks (used heavily by `AssertQuery`) immutable. In `SqlTester`, remove methods `checkFieldOrigin`, `checkResultType`, `checkCollation`, `checkCharset`, `checkIntervalConv`, `checkMonotonic`; all can now be implemented in terms of a new method `validateAndThen`, and similar methods `validateAndApply` and `forEachQueryValidateAndThen`. Obsolete `interface SqlToRelTestBase.Tester`; `SqlToRelFixture` now uses `SqlTester`. Move inner `class MockViewExpander` from `SqlToRelTestBase` to `SqlTestFactory`. Method `SqlToRelTestBase.assertValid` becomes `Matchers.relIsValid`. Refactor/rename a few methods, and change 'query' field to 'expression'. State is now in the fixture (`class SqlParserFixture`) or `SqlTestFactory`. Create a fixture (`class SqlParserListFixture`) for list-based parser tests. Remove fixture's `transform` field; config is now transformed immediately, not deferred. Remove field `LINUXIFY` (thread-local state). Fields `SqlToRelFixture.expression` and `SqlValidatorFixture.expression` were each previously called `query` and had the opposite sense; both are now consistent with `SqlParserFixture.expression`. Rename method `sql` to `withSql`, `config` to `withConfig`, etc. Rename method `withDecorrelation` to `withDecorrelate`, `withLateDecorrelation` to `withLateDecorrelate` `with(RelOptPlanner)` to `withPlanner`, `with(HepProgram)` to `withProgram`. `MaterializedViewTester` is now a utility class for materialized view tests but is no longer required to be a base class. Move `interface AbstractMaterializedViewTest.Sql` to top-level `class MaterializedViewFixture`. In `class MaterializedViewSubstitutionVisitorTest`, create a fixture for satisfiability tests. `RelSupplier` was previously an inner class of `class RelOptTestBase`. It is used in metadata tests (`RelMetadataFixture`) and planner rule tests (`RelOptFixture`) but could be used in other tests too. Add `class RelSuppliers` with utilities and implementations for `RelSupplier`. Move `assertXxx` methods into a fixture, `class SqlAdvisorTest.Fixture`. Remove inner `class LexConfiguration`; it relied on a mutable tester in each `SqlValidatorTest` instance, which is no longer there. Implement methods `toString`, `equals`, `hashCode`. Move inner `class Sql` to top-level `class SqlPrettyWriterFixture`. Add `class ConnectionFactories`, utilities for `ConnectionFactory` and `CalciteAssert.ConnectionPostProcessor`. Enumerates all possible 'xxx' in JDBC `ResultSet.getXxx` and `PreparedStatement.setXxx` methods. Utilities for `interface SqlTester.ResultChecker`. Move `interface SqlOperatorTest.Fixture` to `SqlOperatorFixture`. In `class SqlOperatorFixture`, remove `double delta` arguments from methods that test floating-point values; instead use a `ResultChecker` created using a `ResultCheckers` method such as `isWithin` or `isExactly`. Remove method `withLibrary2`. Move `class SqlToRelConverterTest.RelValidityChecker` to top-level. In class `SqlLimitsTest`, move method `checkTypes` to `class SqlTests`. In a few cases we would built complex transforms (instances of `UnaryOperator`) that were applied just before the test ran. Now we try to apply transforms early, storing a transformed config rather than a chained transform. It's easier to understand and debug. Remove method `RelDataTypeSystemImpl.allowExtendedTrim`, which was added in [CALCITE-2571] but was never used. Remove some uses of `assert` in `TypeCoercionTest`. (Never use `assert` in tests!) Use `Assertions.assertEquals`, `Objects.requireNonNull` via static import. In `class SqlValidator.Config`, rename method `sqlConformance()` to `conformance()`, to be consistent with conformance properties elsewhere. In `class TryThreadLocal`, add methods `letIn(T, Supplier)` and `letIn(T, Runnable)`. Close #2685
Some engines(Hive,Spark,BigQuery,Oracle,Snowflake) support APPROX_COUNT_DISTINCT function, while others do not. So we can use the parameter SqlDialect#supportsApproxCountDistinct to control whether to use APPROX_COUNT_DISTINCT(It is the same as APPROX_DISTINCT for Presto).
…not working 1. Resolve the problem by removing the custom set of testClassesDirs 2. Set correctly the classpath to include JDBC drivers from a custom configuration. 3. Add a bit of documentation around the JDBC tasks and configs. Close #2716
The 1.4.197 version suffers from multiple CVEs such as: https://nvd.nist.gov/vuln/detail/CVE-2018-14335 https://nvd.nist.gov/vuln/detail/CVE-2018-10054 https://nvd.nist.gov/vuln/detail/CVE-2021-42392 https://nvd.nist.gov/vuln/detail/CVE-2022-23221 In the project, we use H2 only for testing purposes thus the H2 binaries are not present in the runtime classpath thus these CVEs do not pose a problem for Calcite or its users. Close #2717
…lify to (A IS NOT NULL) When A is deterministic
…e sort if the view is used in a query at top level
The goal is for simple queries to retain the sorting. For example, given
CREATE VIEW v AS
SELECT *
FROM emp
WHERE job = 'CLERK'
ORDER BY deptno
the query
SELECT *
FROM v
WHERE ename LIKE 'M%'
will be expanded as if the user had written
SELECT *
FROM emp
WHERE ename LIKE 'M%'
AND job = 'CLERK'
ORDER BY deptno
But the ORDER BY will be removed from more complex queries
(e.g. those involving GROUP BY, UNION, sub-query) where it
would have no effect.
If the ORDER BY also includes LIMIT or OFFSET, it affects the
rows returned, not just their sort order; such an ORDER BY is
never removed.
Close #2679
…n error Cache Cassandra sessions based on "hostname, port, keyspace, username, password" information.
[CALCITE-4054] RepeatUnion containing a Correlate with a transientScan on its RHS causes NPE
According to https://www.jacoco.org/jacoco/trunk/doc/faq.html , Jacoco adds a synthetic static method called $jacocoInit() during instrumentation. MetadataDef constructor breaks if it encounters such method on the class given to it because it does not expect it. But it looks like it simply should not consider synthetic methods (as well as static methods) at all. - extract code for finding handler methods - reuse it in both places - add tests concerning synthetic methods
commons-io versions before 2.7 suffer from CVE-2021-29425 which allows to traverse into the parent directory. Update to a more recent version to avoid the afforementioned security vulnerability and benefit from the other improvements in this library. Close #2734
…d when CAST is present
1. Highlight the role & existence of two repos at the beginning of README. 2. Provide a high-level overview of the update procedure early on. 3. Move RM related instructions from README to the appropriate howto sections. 4. Remove git reset/rebase related commands for syncing master & site branches from the beginning of the release process cause it is too early to rebase then. Mention rebase later towards the end of the process. Close #2708
…leTable and projections and filters act on different columns
julianhyde
force-pushed
the
main
branch
2 times, most recently
from
8a5cf83 to
cf7f71b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…roducing