Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IGNITE-4523 #1858

Closed
wants to merge 50 commits into from
Closed

IGNITE-4523 #1858

wants to merge 50 commits into from

Conversation

ascherbakoff
Copy link
Contributor

Allow distributed SQL query execution over explicit set of partitions.

ascherbakoff and others added 30 commits April 1, 2017 22:08
      added missed licenses
      renamed packages
      fixed wrong ml profile activation
(cherry picked from commit d78e071)
…-4523

Conflicts:
	modules/core/src/main/java/org/apache/ignite/cache/query/SqlFieldsQuery.java
	modules/core/src/main/java/org/apache/ignite/cache/query/SqlQuery.java
	modules/core/src/main/java/org/apache/ignite/internal/processors/cache/IgniteCacheProxy.java
	modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/IgniteH2Indexing.java
	modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/twostep/GridReduceQueryExecutor.java
	modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/twostep/msg/GridH2QueryRequest.java
…-4523

Conflicts:
	modules/core/src/main/java/org/apache/ignite/cache/query/SqlFieldsQuery.java
	modules/core/src/main/java/org/apache/ignite/cache/query/SqlQuery.java
	modules/core/src/main/java/org/apache/ignite/internal/processors/cache/IgniteCacheProxy.java
	modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/IgniteH2Indexing.java
	modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/twostep/GridReduceQueryExecutor.java
	modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/twostep/msg/GridH2QueryRequest.java
…nite-4523

# Conflicts:
#	modules/core/src/test/java/org/apache/ignite/internal/processors/service/GridServiceProcessorMultiNodeConfigSelfTest.java
#	modules/core/src/test/java/org/apache/ignite/internal/processors/service/GridServiceProcessorMultiNodeSelfTest.java
#	modules/ml/src/main/java/org/apache/ignite/ml/math/ValueMapper.java
#	pom.xml
…nite-4523

# Conflicts:
#	modules/core/src/test/java/org/apache/ignite/internal/processors/service/GridServiceProcessorMultiNodeConfigSelfTest.java
#	modules/core/src/test/java/org/apache/ignite/internal/processors/service/GridServiceProcessorMultiNodeSelfTest.java
#	modules/ml/src/main/java/org/apache/ignite/ml/math/ValueMapper.java
#	pom.xml
@@ -31,7 +31,7 @@
</parent>

<artifactId>ignite-indexing</artifactId>
<version>2.0.0-SNAPSHOT</version>
<version>2.1.0-SNAPSHOT</version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we have some extra commits from master here.

@@ -780,6 +780,13 @@ private ClusterGroup projection(boolean loc) {

final SqlQuery p = (SqlQuery)qry;

if (ctx.isReplicated() && p.getPartitions() != null)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not look at ctx.isReplicated() here anymore because we are allowed to run any type of query from any cache (partitioned or replicated). This check probably must be somewhere else.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added eager check for isReplicatedOnly mode and another when all caches are prepared and determined to be replicated only. In such case setting explicit partitions will throw an exception.

@@ -794,6 +801,13 @@ private ClusterGroup projection(boolean loc) {

SqlFieldsQuery p = (SqlFieldsQuery)qry;

if (ctx.isReplicated() && p.getPartitions() != null)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same as above.

if (!F.isEmpty(dataNodes(cctx.name(), NONE)))
// Handle special case: no mapping is configured for a partition.
if (F.isEmpty(cctx.affinity().assignment(NONE).get(p))) {
partLocs[p] = Collections.emptySet(); // Mark unmapped partition.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets add a constant UNMAPPED_PARTS for Collections.emptySet() and use it everywhere.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done for all cases.

@@ -1088,6 +1166,9 @@ private GridThreadLocalTable fakeTable(Connection c, int idx) {
for (int p = 0, parts = extraCctx.affinity().partitions(); p < parts; p++) {
List<ClusterNode> owners = extraCctx.topology().owners(p);

if (partLocs[p] == Collections.<ClusterNode>emptySet())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use a constant instead, see above.

@@ -1119,6 +1200,9 @@ private GridThreadLocalTable fakeTable(Connection c, int idx) {
return null; // Retry.

for (Set<ClusterNode> partLoc : partLocs) {
if (partLoc == Collections.<ClusterNode>emptySet())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use a constant instead, see above.

@@ -1134,6 +1218,10 @@ private GridThreadLocalTable fakeTable(Connection c, int idx) {
for (int p = 0; p < partLocs.length; p++) {
Set<ClusterNode> pl = partLocs[p];

// Skip unmapped partitions.
if (pl == Collections.<ClusterNode>emptySet())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use a constant instead, see above.

* Added the constant for unmapped partition marking.
* Improved validation for query with replicated only caches and not empty explicit partitions.
@asfgit asfgit closed this in 5ef610c Apr 26, 2017
SomeFire added a commit to SomeFire/ignite that referenced this pull request May 24, 2017
* Removed duplicate package-info.java for "org.apache.ignite.hadoop" package.

* IGNITE-3386 Reverted: "Reentrant lock is lost when lock owner leaves topology"

Signed-off-by: nikolay_tikhonov <ntikhonov@gridgain.com>

* Replaced partsToEvict from queue to map to do not add the same partition several times.

* IGNITE-5023 .NET: IgniteConfiguration.MemoryConfiguration

* IGNITE-5043 .NET: CacheConfiguration.WriteBehindCoalescing

* Minor doc fix in DataPageEvictionMOde

* ignite-2.0 Fixed IGNITE_MODULES search path.

* IGNITE-4997: DDL: Fixed schema state replay on client reconnect. This closes apache#1845.

* IgniteCacheAtomicProtocolTest fix - Fixes apache#1839.

Signed-off-by: Alexey Goncharuk <alexey.goncharuk@gmail.com>

* IGNITE-4952 - Delete swap-related functionality - Fixes apache#1794.

Signed-off-by: Alexey Goncharuk <alexey.goncharuk@gmail.com>

* IGNITE-4699: Added custom executors for compute tasls. This closes apache#1718.

* ignite-2.0 - Properly handle DhtInvalidPartitionException in GridDhtGetFuture

* IGNITE-4949 - Corrected JdbcPojoStore for keepBinary flag

* GridCachePartitionedTxSalvageSelfTest fix (remove unnecessary topology change wait) - Fixes apache#1853.

Signed-off-by: Alexey Goncharuk <alexey.goncharuk@gmail.com>

* Spring Data version fix

* Minor codestyle fix

* IGNITE-4594 - SQL Index hints - Fixes apache#1832.

Signed-off-by: Sergi Vladykin <sergi.vladykin@gmail.com>

* IGNITE-5031 .NET: Added Partition Loss APIs

ICache.GetLostPartitions, ICache.WithPartitionRecover, IIgnite.ResetLostPartitions, CacheConfiguration.PartitionLossPolicy

* Added test.

* ignite-4984 Fixed potential double processing of node failed event. Removed unused 'tryPut' operation.

* IGNITE-5039 .NET: Improve error message for failed ignite.jni.dll load

* ignite-2.0 - Fixed NPE

* Fixed CacheExchangeMessageDuplicatedStateTest.

* ignite-2893 Restored explicit tx usages to avoid issues when there are no classes on servers

* IGNITE-4988 Fixed merge result.

* ignite-2.0 - H2 upgraded to 1.4.195

* ignite-2.0 - Do not call primaryByKey() in cache entry synchronized block - Fixes apache#1824

* ignite-2.0 - minor fixes

* IGNITE-2578 .NET: Un-ignore AtomicReference tests

* IGNITE-5058: Fixed QueryIndex validation when QueryEntity.valueType is not set yet. This closes apache#1861.

* IGNITE-5018 review and improve javadocs in ML module

* IGNITE-5018 review and improve javadocs in ML module (minor fixes)

* IgniteCacheClientMultiNodeUpdateTopologyLockTest minor.

* IGNITE-5066 .NET: Add continuous query test to verify that the problem from 1.9 no longer reproduces

* IGNITE-3521: IGFS: Removed "maxSpace" property. This closes apache#1854.

* IGNITE-5057 .NET: Fix build.ps1 to handle Any CPU config on PowerShell 4.0 and lower

* ignite-2.0 - SQL: Geo tests fixed + distributed join on replicated cache

* Removed unused swap-related code.

* Try avoid timeout in IgniteCacheGetRestartTest.

* IGNITE-1439: Implemented Futures for C++.

* Fixed AtomicSequence test

* GG-12132 Added magic header to protect from unexpected input.

* ignite-2.0 Update classnames.properties to run code from IDE.

* Fixed failures in IgniteCacheJoinQueryWithAffinityKeyTest.

* Fixed failures in IgniteCrosscacheJoinsQueryTest.

* IGNITE-5069: Fixed failure in QueryWords example.

* IGNITE-5025 - Do not allow null names for memory policies

* CacheConfiguration code style fixes.

* IGNITE-5064 Removed obsolete EventTypes + minor cleanup.

* IGNITE-3521: IGFS: removed "maxSpaceSize" remnants from XML files.

* IGNITE-4523 Allow distributed SQL query execution over explicit set of partitions - Fixes apache#1858.

Signed-off-by: Sergi Vladykin <sergi.vladykin@gmail.com>

* IGNITE-5036: @QuerySqlField and @QueryTextField can no longer be applied to methods. This closes apache#1863.

* Formatting

* Formatting

* ignite-2.0 - Javadoc fix + code style

* ignite-2.0 - Fixing tests

* ignite-1794 Refactored hibernate modules, switched to hibernate 5.1

* IGNITE-4539 - Added RocketMQ integration

* IGNITE-4774 Redesign table.

* ignite-4799 TcpDiscoverySpi: removed missedHeartbeats properties, heartbeatFrequency (instead use IgiteConfiguration.metricsUpdateFrequency). Added IgiteConfiguration.clientFailureDetectionTimeout.

* ignite-2.0 - Fixed offheap RWLock tests

* IGNITE-3487: hidden _key and _val columns - Fixes apache#1865.

Signed-off-by: Sergi Vladykin <sergi.vladykin@gmail.com>

* IGNITE-5088: Fixed OSGI test build instructions.

* IGNITE-5088: Fixed OSGI test build instructions.

* IGNITE-3488: Prohibited null as cache name. This closes apache#1790.

* ignite-5041 NPE in deadlock detection fixed

* IGNITE-4799 Fix broken .NET merge

* IGNITE-4799 Missing change after merge.

* Set cache name to DEFAULT_CACHE_NAME for RocketMQ tests.

* IGNITE-4988 Rework Visor task arguments. Code cleanup for ignite-2.0.

* IGNITE-4988 Fixed VisorRunningQueriesCollectorTask.

* IGNITE-5096 Fixed RAT.

* IGNITE-5095 NULL strings in REST-HTTP should be serialized as null.

* Fixed IGFS and Memcached examples.

* ignite-2.0 - Added missing license file for RocketMQ module

* Fixed IGFS test causing overtimes.

* Quick fix for cache activate/deactivate.

* Fixed default test timeouts.

* IGNITE-5094 - Fix data page eviction for near-enabled caches

* TcpDiscoverySpi: removed unneeded const.

* Fixed failure in TxDeadlockDetectionMessageMarshallingTest.

* Fixed IgniteClientReconnectCacheTest.

* IGNITE-5090 - Get rid of startSize configuration property

* ignite-2.0 - Added onheap caches and extended memory size

* IGNITE-5040: ML examples: restored missed changes. This closes apache#1881.

* GridCacheAbstractFailoverSelfTest: fixed configuration.

* ignite-2.0 - Added logging for binary metadata processor

* IGNITE-4052 Add ability to set up users for MESOS.

Signed-off-by: nikolay_tikhonov <ntikhonov@gridgain.com>

* IGNITE-4052 Add ability to set up users for MESOS.

Signed-off-by: nikolay_tikhonov <ntikhonov@gridgain.com>

* ignite-2.0 - Do not wait for topology in restart tests

* IGNITE-5072 - Updated memory metrics to comply with other metrics

* Improved MemoryMetrics documentation.

* fixed memory policies example

* Disable striped pool for exchange messages.

* IGNITE-4988 Code cleanup for ignite-2.0.

* master Fixed version.

* ignite-2.0 - Fixed javadoc

* IGNITE-4988 Code cleanup for ignite-2.0.

* IGNITE-5077 - Support service security permissions

* IGNITE-5024 code of two tests was fixed to pass new validation rules

* IGNITE-4082 Fixed tests runner.

* IGNITE-4082 Define filter names in app.js instead of filter definitions.

* .NET: Cleanup unused SqlOnheapRowCacheSize mentions

* IGNITE-4988 Fixed metadata import from RDBMS.

* Moved hibernate-core to lgpl profile.

* Release notes for 2.0.

* IGNITE-4995 Minor code cleanup.

* IGNITE-5120 Fixed LGPL build.

* IGNITE-5120: Fixed build of Hibernate module.

* IGNITE-5122: Fixed http-rest-example.php.

* IGNITE-5080 - Fixes in SecurityBasicPermissionSet

* IGNITE-5081 - Removed redundant duplication of permissions in SecurityPermissionSetBuilder

* IGNITE-5127 Fixed validation of url fields. Refactored mixins.

* IGNITE-5134: Fixed ClassCastException in IgniteCacheDatabaseSharedManager. This closes apache#1895.

* IGNITE-5143 Fixed agent watch logic.

* Hibernate module deploy fix

(cherry picked from commit ff08c8e)

* IGNITE-4989 Assertion error in BPlusTree#finishTail - Fixes apache#1894.

Signed-off-by: Sergi Vladykin <sergi.vladykin@gmail.com>

* IGNITE-5132: Fixed PutGet example.

* IGNITE-5135 .NET: Improve remote error propagation

This closes apache#1899

* IGNITE-4995 Fixed javadoc.

* IGNITE-5109 Refactoring for SparseDistributedMatrix

* IGNITE-5012 Implement ordinary least squares (OLS) linear regression.

* IGNITE-5147 Fixed generation of project structure view.

* IGNITE-5146 Simplified generation for user properties.

* topologyVersion was not initialized for GridDhtAtomicSingleUpdateRequest.

* Moved JDBC driver property info to top-level.

* IGNITE-5162: ODBC: removed unused version-dependent code. simplified version enum.

* IGNITE-5164: ODBC: renamed classes which is obviously common with JDBC.

* IGNITE-5162: ODBC: minor removal of a dead code.

* Added docker files for 2.0.0 version

Signed-off-by: nikolay_tikhonov <ntikhonov@gridgain.com>

* IGNITE-5167: ODBC: extracted handler/parser interfaces.

* IGNITE-5118 Added prefix to pojo class if have same name with java built-in classes.

* IGNTIE-4220: Support statements for JDBC and Cassandra store. This closes apache#1898.

* IGNITE-5071 Added table name property for query entity.

* IGNITE-518: Unmuted tests that was fixed in ignite-4036. This closes apache#1636.

(cherry picked from commit c4de164)

* Fixed JavaDocs in MemoryMetricsImpl.

* IGNITE-5116: Fixed stale DML plan caching in DmlStatementsProcessor.

* IGNITE-5169: ODBC: reworked handshake semantics to support JDBC driver integration. This closes apache#1908.

* IGNITE-5172 .NET: Fix type name parser and resolver to handle arrays and simple names

This closes apache#1909

* ignite-4760 Added test

* Make TestMemcacheClient#cacheMetrics runnable with Java 8.

* IGNITE-5119 Append extra message to exception when unable to determine affinity

* master - fixed SELECT (SELECT COUNT(1)) FROM

* master - minor fix for subqueries with aggregates

* master - Fixed typo.

* Optional optimization to all client caches on client join.

* GridCacheEntryInfo: removed old code.

* IGNITE-5054: Preliminar refactoring for decoupled SQL schema. This closes apache#1923.

* .NET: Remove unused imports

* IGNITE-5194 .NET: Fix TryGetIgnite to return an instance with any name when there is only one

* IGNITE-5190 - ArrayIndexOutOfBoundsException in GridMergeIndexSorted

* Removed condition, simplified logic

http://ci.ignite.apache.org/viewLog.html?buildId=606030&tab=buildResultsDiv&buildTypeId=IgniteTests_IgniteDataStrucutures
http://ci.ignite.apache.org/viewLog.html?buildId=606029&tab=buildResultsDiv&buildTypeId=IgniteTests_IgniteBinaryObjectsDataStrucutures

* ignite-4932 When possible for cache 'get' read directly from offheap without entry creation.

* IGNITE-5082 Fixed UI for header and footer.

* IGNITE-5082 Fixed UI for footer.

* IGNITE-5082 Fixed RAT.

* IGNITE-5207 .NET: Support non-Int32 enums

This closes apache#1933

* IGNITE-5165 Add classes for new design buttons and links.

* IGNITE-5086 Changed design for modals.

* Removed unused IgniteInternalCache methods.

* IGNITE-5213 .NET: Fix collection handling in reflective serializer

* IGNITE-5112 Create OLS example:
  Created example & tests refactoring.

* GridCacheConcurrentMap: renamed size -> internalSize.

* Added IO latency test + made it available from MBean

* Fixed thread pools incorrect shutdown.

* IGNITE-5225: Fix NPE caused by changes in IGNITE-4577.

(cherry picked from commit d463840)

* DirectByteBufferStreamImpl: converted asserts into exceptions.

(cherry picked from commit 560ef60)

* IGNITE-5231 Web Console: Add support for Ignite 2.0 cluster on Queries screen.

* IGNITE-5191 .NET: BinaryEnum.ToString

* Added discovery ring latency test + made it available from MBean

* IGNITE-5050 .NET: IIgnite.GetMemoryMetrics

* IGNITE-5082 Change footer links color in order to improve readability.

* Web Console: bind to all interfaces in dev mode.

* IGNITE-5131: Hadoop: updated org.ow2.asm version from 4.2 to 5.0.3. This closes apache#1946.

* IGNITE-5208: Fixed sigfault on concurrent map access

* IGNITE-5193: Hadoop: better handling of empty paths in environment variables. This closes apache#1928.

* IGNITE-4597: CPP: Add methods to reset arguments in sql query.

* IGNITE-5050 .NET: IIgnite.GetMemoryMetrics - fix test stability

* Results printout for IO latency test

* IGNITE-5104: fixing code style issues
Reviewed and merged by Denis Magda.

* Quick fix for regression in tracer utility.

* IGNITE-5082 Fixed layout issues in IE11.

* IGNITE-5165 Fixed button issues in IE11.

* IGNITE-5230 Load SVG icons with svg-sprite-loader, add icons JS entry point.

* IGNITE-4406 .NET: Control DateTime serialization via attribute

* IGNITE-4644 Value from IgniteQueue in atomic mode could be lost

(cherry picked from commit b214211)

* IGNITE-4406 .NET: Control DateTime serialization via attribute - seal the class

* Moved logic related to caches discovery data handling to ClusterCachesInfo.
Start of statically configured caches in the same way as dynamic ones: from GridDhtPartitionsExchangeFuture.

* Fixed condition in GridDiscoveryManager.reconnectSupported

* Removed some unused code.

* IGNITE-4842 Now containsKey() respects isReadFromBackup() flag.

* master - Improve internal javadoc

* IGNITE-5230 Fixed dropdown trigger.

* Added Affinity topology version and Pending exchanges to Visor data collector task.

* Fixed CacheGetFromJobTest failure (wait for correct exchange version for cache start operation).

* Fixed validator suite name.

* Changed RendezvousAffinityFunction Externalizable->Serializable, also affinityBackupFilter was not serialized.

* IGNITE-5244 Added web agent download button.

* IGNITE-5238: Resolved dependency problems of Ignite Spark and Ignite Rest modules. - Fixes apache#1973.

Signed-off-by: shroman <rshtykh@yahoo.com>

* Results printout for IO latency test

(cherry picked from commit 096404d)

* IGNITE-5054: SQL: Simplified query descriptor, partially removed dependencies on 1-to-1 cache-schema dependency. This closes apache#1962.

* Fixed "IGNITE-4205 CassandraCacheStore should start IgniteThread threads in loadCache() method"

Signed-off-by: nikolay_tikhonov <ntikhonov@gridgain.com>

* ignite-4763 doSetRollbackOnly method to be implemented in the SpringTransactionManager

* IGNITE-5257 .NET: SQL query timeouts

This closes apache#1985

* IGNITE-4947 - Move to Ignite 2.0 test suites - Fixes apache#1986.

Signed-off-by: Alexey Goncharuk <alexey.goncharuk@gmail.com>

* IGNITE-5175: Performance degradation using evictions in near-enabled caches - Fixes apache#1931.

Signed-off-by: Alexey Goncharuk <alexey.goncharuk@gmail.com>

* IGNITE-5248 - Detect a 32-bit JVM using too large init page size. Closes apache#1976

* IGNITE-5264: Improved schema-based handling for CREATE INDEX and DROP INDEX operations. This closes apache#1990.

* IGNITE-5054: Minor renames in GridQueryProcessor.

* IGNITE-5215 - Allow user to configure memory policy with maxSize smaller than default initialSize. Closes apache#1938

* IGNITE-5150 - Corrected allocated memory pages count. Closes apache#1915

* IGNITE-5252: Expose getFieldName method to SqlFieldsQuery result. - Fixes apache#1982.

Signed-off-by: Sergi Vladykin <sergi.vladykin@gmail.com>

* Fixed segmented indices snapshots. - Fixes apache#1936.

Signed-off-by: Sergi Vladykin <sergi.vladykin@gmail.com>

* ignite-5212 Allow custom affinity function for data structures cache

* IGNITE-5163: Implemented infrastructure for the new JDBC driver. This closes apache#1912.

* Results printout for IO latency test and new metrics

(cherry picked)

* IGNITE-5281: Indexing: changed "space" to "cacheName". No more "spaces". This closes apache#1992.

:quit
:quit
mdules/web-console/frontend/app/modules/states/configuration/summary/summary-zipper.service.js~HEAD

* Fixed a misprint in the non collocated join example

* IGNITE-5282: SQL: Correct top-bottom propagation of "keepBinary" flag. This closes apache#1993.

* master - BPlusTree: compare with lvl

* Minor - corrected TODO

* Minor - corrected TODO

* Typo

* IGNITE-5121: REST API call with empty cache name to call "default." - Fixes apache#1920.

Signed-off-by: shroman <rshtykh@yahoo.com>

* IGNITE-5284: Splitted IgniteH2Indexing into several classes. This closes apache#1999.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants