* adding commandline arg to kill on OOM

* checking if system is 64 bit to apply right libgemfirexd<32/64>.so

* adding agent in case of mac/linux

* enable copy in case of mac also

* implementing review suggestions

* checking if agent load would be successful else continue without agent

* implementing review suggestions

* undo-ing wrong changes to gradle.properties

* syncing store

Reset the pool at the end of collect to avoid spillover of lowlatency pool setting to later operations that
may not use the CachedDataFrame execution paths.

For cases like MacOSX that ships with bash 3.x by default

…directory structure in order to avoid the confusion with PID of hydra JVM Vs PID of snappy node

- Correcting table names in configuration file required for dmlOps tests

Testing Done: Verified the changes by running few tests from HA and non-HA bts

- avoid unncessary re-evaluation of cluster properties target
- redirect error of spark-shell in testSparkShell to output string for checks too
- update store and spark links
- remove distZip from default assemble target

…#1194)

* ignoring failing tests

* removing wrongly created test suite

* formatting changes

…the server.

Because of the previous order there were chances that this function came for
execution on this node even before it is registered.

…ere was an issue with running snappy job with --packages option ouside the snappydata build directory.

…#1195)

Key-based aggregations (GROUP BY) already handle copying of incoming value
but was missing in non-key flat aggregations.

## Changes proposed in this pull request

- refactored value copy code in ObjectHashMapAccessor and string clone code (only if required)
- use above for non-key aggregations too
- renamed io.snappydata.implicits to io.snappydata.sql.implicits
- handle null values during clone/copy of non-primitive aggregate results

Also corrected the copyright headers to be SnappyData ones.

Changed the year from 2017 to 2018 in license headers.

…alCatalog

- for paths from SnappyExternalCatalog (e.g. "CREATE VIEW"), the catalog cache needs
  to be cleared too; added unit test for this too
- fixing some occasional failures due to test issues
- renamed SnappyTableStatsProviderService.suspendCacheInvalidation to TEST_SUSPEND_CACHE_INVALIDATION
  to indicate clearly it is meant only for tests

When using SnappySession, the default, the temporary hive configuration passed
is not just used by HiveServer2, but also overrides the internal hive
configuration used by SnappyStoreHiveCatalog causing problems.

Now using SnappySession hive configuration after adding the "hive.server2"
configuration read from the temporary "executionHive" client (that in turn
    will set it up using hive-site.xml etc that are ignored by SnappySession's
    hive meta-store client)

Also reduced logging during the temporary hive client initialization.

- new "snappydata.sql.hiveCompatible" to turn some SQL output to be more hive compatible;
  currently this includes "show tables ..." variants that have only one "name" column in output
- added unit tests for above property and "show tables ..." variants

.jvmkill*.log files. .hprof can be pulled by passing an option '-m'
or '--hprofdump' to the script.

Hot Fix Changes
Updates to backlog doc items
New Spark Extension API Guide

Jar published for snappydata-jdbc is shadow one that includes all dependencies

…ze" in Tables. (TIBCOSoftware#1185)

* Changes for SNAP-2602:
 - HTML and CSS changes for displaying tables overflown size to disk as Spill-To-Disk Size on UI.
 - TableSummary is updated to hold tables spillover size to disk.
 - SnappyTableStatsProviderDUnitTest is updated accordingly.
 - and relevant code changes at other places.
 - Named column as Spill-To-Disk Size for tables spillover size to disk.
 - Renaming column Memory Size to In-Memory Size.

- Cleanup and formatting changes in CDC tests.

…TIBCOSoftware#1198)

ParamLiterals are not constants, so common-subexpression elimination phases of optimizer
may fail when values inside them are actually equal.

## Changes proposed in this pull request

This adds a new RefParamLiteral that points to a ParamLiteral which is used when a constant value is
found that is same as one found before.
The hashCode mirrors that of original ParamLiteral while equals is reference equality with original.
This enables a ParamLiteral to be equal to a RefParamLiteral in a different position.

When considering two different plans for equality, the RefParamLiterals have to be in exactly same
positions to be evaluated as equal since RefParamLiterals are never equal to other ParamLiterals.
For two RefParamLiterals, they are equal iff their original ParamLiterals are.

This allows expressions like "a = 4 and b = 4" and "a = 6 and b = 6" to be equal after tokenization
in plan caching, but not "a = 4 and b = 4" and "a = 6 and b = 5" which will have different plans.

Also added a test for above including check for filter push down.

- Updating variable into ExternalTableSummary to hold tables fully qualified name.
- Set tables schema and fully qualified names while creating SnappyExternalTableStats objects.

…lExecution (TIBCOSoftware#1201)

As part of these changes, we have removed Snappy's DefaultPlanner and instead of passing Snappydata specific strategies as part of `experimentalMethods.extraStrategies` of `SnappySessionState`.

Also, SnappyAggregationStrategy is disabled for stream queries as it is clashing with StatefulAggregationStrategy which is added by Spark as part of `IncrementalExecution`. This is achieved by setting snappydata.sql.hashAggregateSize to -1 in SnappySession's configuration.
This implies that Snappydata aggregation optimization will be turned off for any usage of that particular SnappySession even for non-streaming queries.

…om hive meta-store client

- add support for SHOW VIEWS
- use "schemaName" for the column instead of Spark's "database" in SHOW TABLES
- show CHAR/VARCHAR types instead of STRING for those types of columns in meta-data queries

…1202)

* Disabled plan caching for LocalTableScanExec and RDDScanExec. Fixes SNAP-2712.

Added following tests cases to test  cluster recovery in the following cases:
1)When server/lead as a new node is started first in the cluster.
2)When new node is added and rebalance along with abrupt server kill is called in parallel.
3)When any of the nodes in the cluster crashes or gets OOM/Low Memory exception, along 
   with the 2nd test case.
4)When the cluster is restarted with min memory.
5)When all locators are down and a new node tries to join the cluster along with rebalance.
5)When collocated tables are dropped in wrong order/right order and servers are killed abruptly.

Minor fixes.

…derby for validation. (TIBCOSoftware#1178)

* - Increasing wait time after lead HA.
* - Test changes to send tableName as argument to snappy job.
* - Adding support for having the same operation as in streaming in the derby to be performed parallelly. This is for data validation.
* - Changing modifier for derby methods.
- Rearranging the CLOSETASK to get proper execution sequence.
* - Adding a method in kafka producer to call it from hydra.
* - removing eventType from row as it is getting added to derby as column.
* - Modify the streaming_sink test to use derby validation.
* - Adding test for conflation testing - with small set of keys with repeated operartions on same key.
* - Adding test for generic column names
* - Making changes to the Message producer to run multi threaded.
* - Changing the number of events to send from kafka producer.
* - Renaming the replicated row table test bt.

- Adding some more compatibility tests for structured streaming
- Enabling few passing tests which were ignored earlier

)

* [SNAP-2381] global lock to serialize concurrent puts

- global lock in putInto execution to serialize concurrent puts
- added lock to session context and clear in putInto plan's executeCollect

* Avoid region lock in case of smart connector mode

* Added links in the release note corresponding to the items that have supporting documentation. 
* Incorporated changes to Spark Extension API Guide 
* Added How-to topic for connecting tableau to SnappyData. 
* Edited the Setting up OBDC Driver and Tableau chapter and moved the Tableau content to the how-to topic.
* Incorporated review comments. 
* Add description for server-auth-provider.
* Added a section on Handling OOM Error with jvm kill agent in the best practices section. 
* Added security based syntax and examples for Command Line Utilities. 
* Added syntax and corresponding examples for PUT INTO in column tables and row tables. Added a warning in column tables for the appropriate usage of syntax.

…oftware#1193)

* Changes for SNAP-2661 : Provide Snappy UI User a control over Auto Update
 - Adding HTML code changes for Auto Update ON/OFF Switch on Snappy UI (Dashboard and Member Details page).

…IBCOSoftware#1203)

- Removed framework defaults for eviction and critical heap percentage, instead tests will use product defaults in case not provided
- Fixed an issue with installJar bt run. The bt started failing after recent code factoring changes
- Added hydra test coverage to simulate a memory leak issue reported by the user with concurrent put into and select operations.

…ays (TIBCOSoftware#1210)

* - Automated hydra test to calculate load time for tables in different ways(csv/parquet,dataFrame/externalTable).

@lizygeogy please review formatting.

- check for members with offheap settings persist the default CACHE  TABLE result in offheap
- instead of onDemand, use already stored stats

- switched all intermediate arrays in decoders/encoders to use BufferAllocator so that it can
  be accounted propertly (only in off-heap config for now)
- added proper close methods everywhere for both encoders and decoders for above and invoke
  the close in ColumnTableScan generated code
- force a repartition on the implicit BATCHID column in column update/delete when there is
  no explicit partitioning in the table so that all changes for a batch are together and get
  applied as a single delta rather than spread across all partitions
- use KEY_COLUMNS as partitioning columns if none have been specified expliciitly
- use SnappyCacheTableCommand for putInto intermediate cache to show the plan of cache properly
- moved task interrupt checks to both column and row iterators so that task execution can
  stop quickly; removed the check from ColumnTableScan generated code
- drop temp table when no caching is done
- reduce default spark.memory.fraction to 0.85
- project on only the table columns (and implicit batchId columns) before caching to reduce cache size
- use the cached intermediate result for both insert and update parts of putInto (instead of just former)

Fix for InternalGemFireError seen with putIntos

Reason was that ColumnDelta.apply could return original value without changing
anything for stats rows, but it was returning the result of getValueRetain call
(which could be a temporary copy) instead of original region value. This could
lead to the temporary value being released completely after writing to Oplog
so that the value put in region will have incorrect reference count leading
to a cascading set of problems. The specific InternalGemFireError issue is due
to the fact that the internal buffer of this released value was changed during
released to an empty one without doing any accounting for the same.

Fix for SNAP-2756. In some rare cases a value might get completely released
before a thread reaches the second sync block in ColumnFormatValue.decompressValue.
For such cases return null as the result value (caller will read from disk if required).

Some cleanups in ColumnFormatEntry after code reviews.

Log orignal DDL string (constructed) for column table creates

Test fixes

- fixed a failure in SplitSnappyClusterDUnitTest.testRowTableCreation due to "ANONYMOUS" schema
  created in previous embedded thrift server test
- removed obsolete ResolveAggregationExpressions which is now handled by RefParamLiteral
- moved snappy-core dependency above for kafka tests to work from IDEA due to conflicting
  class KafkaTestUtils getting picked from spark instead
- fixing failing test: a column table must have at-least one extra column apart from partition column to
  perform put into. With recent code changes, we are making key columns as partition columns if
  partition columns are not explicitly specified. Hence the test started failing as it was having only one
  column and the same column is declared as key column as key columns are now treated as partition
  columns.

- pass the query string along to SnappyCacheTableCommand to display in GUI
- removed the extra DS.count() and instead pass back the count of cached batches from cache query
- put into will pass "CACHE FOR (sql)" string if SQL is available else use "PUT INTO <plan>"
  when using putInto API
- also changed CACHE TABLE <table> without query to use off-heap route if enabled,
  and display the full plan in GUI as done for the query case
- set SQL text as Spark job description to show in the jobs/stages tabs in the UI; set and clear the
  SparkContext job description in pairs else it can get carried over to next execution (e.g. API
   execution with smart connector or zeppelin that does not use SnappySession.sqlPlan)
- avoid showing two plans for CTAS and CACHE QUERY statements
  - skip the empty second plan for ExecutedCommandExec and only the insertion plan is shown

…tion packed between two inserts results into extraneous records

 - if event type of the last event for a key is insert and there are more than one events for the same key, then convert inserts to put into
 - skipping caching(and materializing) of incoming dataframe when target tables don't contain key_columns/primary_keys as all events will be treated as inserts in this case.

- Modified the schema in the existing putInto long-running test to include 30 columns including most of the string columns
- In this test, 20 columns will be having valid values and 10 columns will be having null values
- Key_columns is specified by a composite key including two string columns
- Added insert ops in the test with using data from JSON files at regular intervals.
- Added a job to generate the required number of JSON data files containing unique values for key_columns for a user-specified schema
- Framework changes to remove parameters added explicitly for taking heap dump on OOME
- Reduced default SleepTimeSecsForJobStatus and SleepTimeSecsForMemberStatus to 5 secs from 120 secs and 30 secs respectively in order to minimize the test execution time

…teral (TIBCOSoftware#1212)

RefParamLiteral of RefParamLiteral was being created so referenceEquals was returning false.
Correct the search to ensure that RefParamLiterals are skipped and also added an assertion that
RefParamLiteral should never wrap another RefParamLiteral.

Added a test with views to check above.

…COSoftware#1219)

This replaces the current session catalog implementations (SnappyStoreHiveCatalog,
SnappyConnectorExternalCatalog) with a single new implementation SnappySessionCatalog that
is common for both embedded and connector modes. All real work has now been shifted to
ExternalCatalog implementations SnappyHiveExternalCatalog and ConnectorExternalCatalog
respectively. Former now extends HiveExternalCatalog and has minimal overrides to behave as
close to Spark as possible.

Smart connector catalog has been re-written to use two common procedures to get and update
catalog meta-data passing generic arguments encapsulated in thrift classes. The older separate
procedures for DDL executions are now all removed. It no longer uses a hive client for queries either.
These changes allow the smart connector to execute all kinds of catalog updates available in
the Spark interface including thing like altering buckets/partitions for Spark parquet/ORC tables.

## Other changes proposed in this pull request

- enabled a bunch of compatibility tests; this will closed multiple open issues that will be done
  separately when this branch merges
- added implicit retry for catalog stale exception in queries
- invalidate entire cache of connector for a create/drop/alter since the version
  stored for other relations in RelationInfo will also certainly be stale

Replace koloboke maps with eclipse collections:
- The koloboke project has been dead and unmaintained for a couple of years now
  so replaced with eclipse collections though latter are a bit slower for some operations
  and also add significant bulk (~10M).

- change tokenize to be session property rather than global
- fixed issue with update sub-query due to alias removal
- clear global view catalog explicitly in close
- allow for absence of baseTable in external catalog table drop since it can be a temporary table
- add gemfire to the providers for which dbtable is added implicitly
- add special path for "gemfire" data source:
  Allow for "gemfire" data source to make a catalog entry during create table execution
  in its createRelation itself. It needs the creation to add new parameters to the options bag.
- fixed dependent handling to avoid duplicates

…IBCOSoftware#1211)

* * Changes from @sumwale to set the credentials from thrift layer into session conf.
* Also, adding a property to trigger authentication when set.
* Unit test.

* * Moved properties to enable hive server start from QueryRoutingDUnitSecurityTest to
  ClusterManagerLDAPTestBase so that the test does not fail when run in a suite.
* Replaced null for username, password with empty strings.

Broadcast/exchange reuse was not happening for column/row tables because the scans
had RDD and BaseRelation objects in their case classes so the equality match against
similar scan but having different objects for those two failed.

  Changes proposed in this pull request

- added overrides for sameResult to both column and table scans that depends only on the table name
  and schema of the output projection
- added a unit test for checking reuse of exchange/broadcast in QueryTest
- corrected the code to avoid two execution display in SQL tab for CTAS (was missing
    the check against CreateTableUsingCommand)
- enhanced the data path in ExternalTableMetadata to include locationUri's for file based stores,
  URL for JDBC sources masking the password portion
- refactored catalog classes to split out into public and impl classes cleanly so that connector v2
  implementation can use it

- call "pruneSchema" for key_columns that checks for presence of columns
- fail if key_columns is specified for a row table
- change Seq.empty to Nil

- updated build files for gradle 5.0
- updated dependency versions to compatible recent ones where possible
- changed all tests to avoid using println/show as far as possible and use logInfo instead;
  this is to avoid polluting the standard output with all kinds of information
- updated NOTICE file with the current dependency jar versions

… tables other than of type policy.

…tware#1227)

- previous changes set SparkContext.SPARK_JOB_DESCRIPTION property to the query string
  for SnappySession.sql executions but this can exceed 32K and property will fail in
  serialization, so trim it to 100 size with "..." continuation like done in SQL tab display
- added large view test to ViewTest and enhanced it to accept a generic "String => DataFrame"
  closure so that the same can be used for scala tests for SnappySession as well as dunits
  for JDBC Statement.execute; added the same tests to DDLRoutingDUnitTest using this

Note: the above test is unable the reproduce the original issue with CREATE VIEW but it does
reproduce for a large query string

- disallow CREATE INDEX creation on column tables without experimental-features property
- clear catalog cache in shutdown to avoid its accidental use by subsequent tests

also change javax.servlet-api version to 3.0.1

…incoming batch doesn't contain any update or delete events (TIBCOSoftware#1229)

- Create table with one of the column as ARRAY TYPE and run the queries.
- Create table with multiple MAP TYPE columns and run the queries.
- Create table with one of the column as STRUCT TYPE and run the queries.
- Create table with one of the column as ARRAY Of STRUCT and run the queries.
- Create table with one column as ARRAY TYPE, second column as MAP TYPE and third column 
   as STRUCT TYPE.
- Create table with one of the column as MAP TYPE[String as Key, Array as Value] and
   run the queries.
- Create the complexType SQL scripts to test the above cases with Snappy-Shell. Will add the
   Hydra test later on. Currently park the Snappy-Shell (JDBC Hydra Test) on low priority.

- Separating out the encoders and decoders classes from core a separate encoders module. This will help in accessing encoders from different connectors (e.g. V2 Connector in stock spark case)
- A separate jar is available of the encoders which will work across the spark versions.

…TIBCOSoftware#1230)

Changes for Auto-Refresh button on SnappyData pulse and misc changes.
Edits for grammar and language and consistency
Changes in the properties section.
Section created: Important Settings > SnappyData Smart Connector Mode and Local Mode Settings for Handling Out-of-Memory Error in SnappyData Cluster.

Added content in Accessing SnappyData Tables from any Spark (2.1+) Cluster topic in the Programming guide section.

* Updates done for SNAP-2651 misc documentation changes.
* Changed "class SnappySampleJob extends SnappySQLJob to "class SnappySampleJob extends JavaSnappySQLJob as suggested by Vatsal.
* Add -critical-off-heap-percentage and -eviction-off-heap-percentage properties to Configuring Servers section. Plus edits for language, formats, and consistency.
* edits to content for handling large size tableau extract results and minor edits to list of properties.
* Add topic in the Programming section > Using SnappyData for any Spark Distribution, Adding a topic in How to > How to use SnappyData for any Spark Distribution. Add corresponding links in the programming guide. Add a corresponding entry in mkdocs.yml.
* Edited content in Important Settings > SnappyData Smart Connector Mode and Local Mode Settings > Handling Out-of-Memory Error in SnappyData Cluster to include details about the timeout properties for jvmkill and heap dump.
* edited eviction-heap-percentage and log-size file properties in both the topics.
*  Updates to include Auto-Refresh switch on the SnappyData Pulse page. Some minor language and grammar edits.
* Add Spark jobserver properties and other properties as specified in SNAP-2538.
* Updates to Supported Datatypes guide and a new "how-to store retrieve complex datatypes" topic  based on inputs from Shirish.
* Edits and changes to supported data types guide and to the related How-to
* Changed default buckets to 8 from 128.
* Add content for topic Access SnappyData Tables from any Spark (2.1+) cluster.

…COSoftware#1238)

- add schema name to BASETABLE if it is not fully qualified
- explicitly add weightage column at catalog level if not present
- delete existing temporary hive directory on restart

…are#1232)

* - Fixing path for store test jar.
* - Adding code to check and ignore the .01 decimal mismatch in case rows in validation do not match.
* - Fixing decimal precision issue for columns as in ct tables aggregate query values were returned as 'NULL'.
* - Fixing issue when snappy job could not be resubmitted after leadHA, as the old lead details were used.
* - Removing commented query
* - Rounding off the decimal difference before comparison.

… to compare CachedKey (TIBCOSoftware#1243)

…IBCOSoftware#1247)

Fix to enforce unique alias names to packages - deploy package command

…issue (TIBCOSoftware#1247)"

This reverts commit b47944b.

…le in input (TIBCOSoftware#1240)

- skipping _eventType column handling from conflation logic when input
dataframe doesn't contain _eventType column
- throwing AnalysisException when target table doesn't contain
key columns or primary key

Changed SnappyData version number.

but in Cast node.

…issue via test

…ming queries and applying plan caching only for JDBC queries (TIBCOSoftware#1205)

- SNAP-2719 - falling back to uncached SQL flow for streaming queries

- SNAP-2719  renaming sqlCached method to sqlInternal as sqlCached is misleading considering the fact that disabling plan caching can still make that method behave in the uncached manor. Also making the method private.

- [SNAP-2457] Apply plan caching only for JDBC queries (TIBCOSoftware#1119)

- updating tests failing with plan caching disabled

- changes in test frameworks to run tests with a random value of plan caching

- removing planCachingAll property as its behavior was misleading and not required after keeping plan caching default value to false.