Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SNAP-2366] row buffer fault-in, forced rollover, merge small batches #1046

Open
wants to merge 32 commits into
base: master
Choose a base branch
from

Commits on May 30, 2018

  1. Update store link

    Sumedh Wale committed May 30, 2018
    Configuration menu
    Copy the full SHA
    2972c4d View commit details
    Browse the repository at this point in the history
  2. fix for SNAP-2365

    Trilok Khairnar committed May 30, 2018
    Configuration menu
    Copy the full SHA
    ff3cca0 View commit details
    Browse the repository at this point in the history

Commits on May 31, 2018

  1. [SNAP-2366] row buffer fault-in, forced rollover, merge small batches

    - add check for the two cases in table stats service:
      - "large enough" row buffer (currently "large enough" is anything more than maxDeltaRows/8)
        that has not seen any updates/deletes since the last check; in this case schedule
        a task to force rollover the row buffer in column table
      - also check if a bucket of column table has multiple small batches (non-transactional check);
        if so then submit a task to merge those after checking for transactional snapshot;
        merge is done by locally created ColumnTableScan->ColumnInsertExec plan where the scan
        uses an iterator only on the small batches
    - added a ColumnFormatStatsIterator that can take a bunch of stats rows and create an iterator
      over just those (like required for batch merge)
    - added new scan metrics for disk reads: a) disk rows from row buffer,
      b) partial column batches on disk, c) full column batches on disk
    - extended SQLMetrics types with a new SPLIT_SUM_METRIC that allows displaying multiple metrics
      against a common name; ColumnTableScan now uses this to combine some metrics else it becomes
      too large in display (especially for the newly added disk read metrics)
    - use hive-metadata (ExternalTableMetaData) to get number of rows instead of getting
      from row buffer table (that is subject to changes in future)
    Sumedh Wale committed May 31, 2018
    Configuration menu
    Copy the full SHA
    608bf3d View commit details
    Browse the repository at this point in the history
  2. update store link

    Sumedh Wale committed May 31, 2018
    Configuration menu
    Copy the full SHA
    cd21c4d View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2018

  1. Some optimizations and fixed few issues

    - fixed disk metrics collection added previously; set the metric correctly
      for both row buffer iterator (ResultSetTraversal) and ColumnFormatIterator
    - added a metric for remote batch fetch
    - fixed multiple ColumnTableScans causing split metrics to add up into one
      ColumnTableScan; now use a unique ID for split metrics for each ColumnTableScan instance
    - fix an NPE in SnappyTableStatsProviderService while filling up
      result map from members since CHM cannot hold null values
    - use a common entry map in ColumnFormatIterator disk iteration instead of creating
      separate for every column batch
    - added implementation of PURGE_CODEGEN_CACHES as StoreCallbacksImpl.clearCodegenCaches
    - limit to one task per table for background rolloverRowBuffer and mergeSmallBatches tasks
    - replaced a few usage of Map.put with justPut for koloboke maps
    Sumedh Wale committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    00f6cda View commit details
    Browse the repository at this point in the history
  2. minor change

    Sumedh Wale committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    a1dadd9 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2018

  1. Fixing precheckin failure in SNAP-2365 and and adding similar fix for…

    … replicated table
    Trilok Khairnar committed Jun 5, 2018
    Configuration menu
    Copy the full SHA
    cf1eb84 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2018

  1. Merge remote-tracking branch 'origin/master' into SNAP-2366

    Conflicts:
    	core/src/main/scala/io/snappydata/SnappyTableStatsProviderService.scala
    	core/src/main/scala/io/snappydata/TableStatsProviderService.scala
    	store
    Sumedh Wale committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    215c6e3 View commit details
    Browse the repository at this point in the history
  2. Merge remote-tracking branch 'origin/SNAP-2365' into SNAP-2366

    Conflicts:
    	cluster/src/test/scala/io/snappydata/benchmark/TPCHColumnPartitionedTable.scala
    Sumedh Wale committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    7f69db0 View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2018

  1. Merge remote-tracking branch 'origin/master' into SNAP-2366

    Conflicts:
    	core/src/main/scala/org/apache/spark/sql/store/CompressionCodecId.scala
    Sumedh Wale committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    9595f20 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2018

  1. Configuration menu
    Copy the full SHA
    88045eb View commit details
    Browse the repository at this point in the history
  2. coarse container locking to fix rollover/merge when running in parall…

    …el with update/delete
    Sumedh Wale committed Jun 28, 2018
    Configuration menu
    Copy the full SHA
    c858273 View commit details
    Browse the repository at this point in the history
  3. temp

    Sumedh Wale committed Jun 28, 2018
    Configuration menu
    Copy the full SHA
    0ea9eb9 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2018

  1. Configuration menu
    Copy the full SHA
    dbdbcb7 View commit details
    Browse the repository at this point in the history
  2. minor formatting change

    Sumedh Wale committed Jul 10, 2018
    Configuration menu
    Copy the full SHA
    c824cd6 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2018

  1. fixes

    Sumedh Wale committed Aug 2, 2018
    Configuration menu
    Copy the full SHA
    bf26063 View commit details
    Browse the repository at this point in the history
  2. fix a ClassCast

    Sumedh Wale committed Aug 2, 2018
    Configuration menu
    Copy the full SHA
    f423108 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    59588c0 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2018

  1. fixing putInto which clears context prematurely

    added an OperationContext in SnappySession that can be used to persist context across
    multiple plan executions (e.g. caching for putInto, then actual execution)
    Sumedh Wale committed Aug 3, 2018
    Configuration menu
    Copy the full SHA
    6ec25d0 View commit details
    Browse the repository at this point in the history
  2. fix COMMIT call

    Sumedh Wale committed Aug 3, 2018
    Configuration menu
    Copy the full SHA
    4905afc View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    fa3f639 View commit details
    Browse the repository at this point in the history
  4. update store link

    Sumedh Wale committed Aug 3, 2018
    Configuration menu
    Copy the full SHA
    778bb4b View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2018

  1. updates and fixes

    Sumedh Wale committed Aug 6, 2018
    Configuration menu
    Copy the full SHA
    647d243 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2018

  1. Configuration menu
    Copy the full SHA
    86f23d9 View commit details
    Browse the repository at this point in the history
  2. fix build issues after master merge

    Sumedh Wale committed Aug 10, 2018
    Configuration menu
    Copy the full SHA
    94c45d6 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2018

  1. Configuration menu
    Copy the full SHA
    4fb08bf View commit details
    Browse the repository at this point in the history
  2. minor cleanups

    Sumedh Wale committed Aug 31, 2018
    Configuration menu
    Copy the full SHA
    e7960a9 View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2018

  1. Configuration menu
    Copy the full SHA
    f2be757 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    872bd8b View commit details
    Browse the repository at this point in the history
  3. minor updates to tests

    Sumedh Wale committed Nov 2, 2018
    Configuration menu
    Copy the full SHA
    ce7fac9 View commit details
    Browse the repository at this point in the history

Commits on Dec 22, 2018

  1. Configuration menu
    Copy the full SHA
    08a6956 View commit details
    Browse the repository at this point in the history

Commits on Dec 29, 2018

  1. Configuration menu
    Copy the full SHA
    772c4af View commit details
    Browse the repository at this point in the history