[CARBONDATA-3718] Support SegmentLevel MinMax for better Pruning and less driver memory usage for cache #3584

Indhumathi27 · 2020-01-17T13:46:29Z

Why is this PR needed?

In Cloud scenarios, index is too big to store in SparkDriver, since VM may not have
so much memory. Currently in Carbon, we will load all indexes to cache for first time query.
Since Carbon LRU Cache does not support time-based expiration, indexes will be removed from
cache based on LeastRecentlyUsed mechanism, when the carbon lru cache is full.

In some scenarios, where user's table has more segments and if user queries only very
few segments often, we no need to load all indexes to cache. For filter queries,
if we prune and load only matched segments to cache,
then driver's memory will be saved.

What changes were proposed in this PR?

Added all block minmax with column-id and sort_column info to segment metadata file
and prune segment based on segment files and load index only for matched segment. Added a configurable carbon property 'carbon.load.all.index.to.cache'
to allow user to load all indexes to cache if needed. BY default, value will
be true, which loads all indexes to cache.

Does this PR introduce any user interface change?

Yes.

Is any new testcase added?

Yes

CarbonDataQA1 · 2020-01-17T14:56:03Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1681/

CarbonDataQA1 · 2020-01-19T02:53:44Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1683/

core/src/main/java/org/apache/carbondata/core/util/SegmentMinMax.java

CarbonDataQA1 · 2020-01-22T10:14:36Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1735/

CarbonDataQA1 · 2020-01-23T10:05:24Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1756/

CarbonDataQA1 · 2020-01-28T17:20:25Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1783/

CarbonDataQA1 · 2020-01-29T07:18:36Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1786/

CarbonDataQA1 · 2020-02-03T06:49:15Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/6/

CarbonDataQA1 · 2020-02-14T10:49:44Z

Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/288/

CarbonDataQA1 · 2020-02-14T10:50:22Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1992/

CarbonDataQA1 · 2020-02-14T11:12:44Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1990/

akashrn5 · 2020-02-14T06:09:33Z

core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java

+
+  public static final String CARBON_LOAD_ALL_INDEX_TO_CACHE = "carbon.load.all.indexes.to.cache";
+
+  public static final String CARBON_LOAD_ALL_INDEX_TO_CACHE_DEFAULT = "true";


add comment when to set false

akashrn5 · 2020-02-14T06:09:40Z

core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java

@@ -2333,4 +2333,9 @@ private CarbonCommonConstants() {
   * Default first day of week
   */
  public static final String CARBON_TIMESERIES_FIRST_DAY_OF_WEEK_DEFAULT = "SUNDAY";
+


add comment

akashrn5 · 2020-02-14T06:10:04Z

core/src/main/java/org/apache/carbondata/core/datamap/Segment.java

@@ -85,6 +86,8 @@
   */
  private transient Map<String, String> options;

+  private List<SegmentMinMax> segmentMinMax;


add comment what it stores

akashrn5 · 2020-02-14T08:35:26Z

core/src/main/java/org/apache/carbondata/core/indexstore/SegmentBlockInfo.java

+
+import org.apache.carbondata.core.util.SegmentMinMax;
+
+public class SegmentBlockInfo {


give class level and variable level comment

akashrn5 · 2020-02-14T09:06:12Z

core/src/main/java/org/apache/carbondata/core/util/SegmentMinMaxStats.java

+  private SegmentMinMaxStats() {
+  }
+
+  public static SegmentMinMaxStats getInstance() {


in getInstance, create new object and map, only once and then reuse for each load, just clear the map entries once filled in accumulator.

akashrn5 · 2020-02-14T11:16:48Z

core/src/main/java/org/apache/carbondata/core/util/SegmentMinMaxStats.java

+    }
+  }
+
+  public void clear() {


this method can be removed

akashrn5 · 2020-02-14T11:17:54Z

core/src/main/java/org/apache/carbondata/core/util/SegmentMinMaxStats.java

+      segmentMinMaxList.add(new SegmentMinMax(minValues, maxValues));
+      this.segmentMinMaxMap.put(segmentId, segmentMinMaxList);
+    } else {
+      this.segmentMinMaxMap.get(segmentId)


in if check already this.segmentMinMaxMap.get(segmentId) is null, so here put the segment id in map, nullpointer can come

akashrn5 · 2020-02-14T11:18:21Z

...ark-common/src/main/scala/org/apache/carbondata/spark/rdd/InsertTaskCompletionListener.scala

+      // add segment level minMax to accumulator
+      accumulator.add(SegmentMinMaxStats.getInstance().getSegmentMinMaxMap.
+        asScala.mapValues(_.asScala.toList).toMap)
+      SegmentMinMaxStats.getInstance().clear()


can just be map.clear

akashrn5 · 2020-02-14T11:18:54Z

integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/UpdateDataLoad.scala

@@ -58,6 +61,9 @@ object UpdateDataLoad {
      loadMetadataDetails.setSegmentStatus(SegmentStatus.SUCCESS)
      val executor = new DataLoadExecutor
      TaskContext.get().addTaskCompletionListener { context =>
+        accumulator.add(SegmentMinMaxStats.getInstance().getSegmentMinMaxMap.
+          asScala.mapValues(_.asScala.toList).toMap)
+        SegmentMinMaxStats.getInstance().clear()


same as above

akashrn5 · 2020-02-14T11:19:33Z

integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala

@@ -315,7 +316,10 @@ object CarbonDataRDDFactory {
    val carbonTable = carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable
    var status: Array[(String, (LoadMetadataDetails, ExecutionErrors))] = null
    var res: Array[List[(String, (LoadMetadataDetails, ExecutionErrors))]] = null
-
+    // accumulator to collect segment minmax
+    val minMaxAccumulator = sqlContext


rename to segmentMinMaxAccumulator

CarbonDataQA1 · 2020-02-14T13:59:18Z

Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/292/

CarbonDataQA1 · 2020-02-14T14:00:08Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1996/

CarbonDataQA1 · 2020-02-14T16:02:55Z

Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/295/

CarbonDataQA1 · 2020-02-14T17:18:12Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1999/

CarbonDataQA1 · 2020-02-14T17:52:13Z

Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/297/

CarbonDataQA1 · 2020-02-14T19:13:09Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2001/

CarbonDataQA1 · 2020-02-15T10:44:35Z

Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/307/

CarbonDataQA1 · 2020-03-14T07:00:39Z

Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/759/

CarbonDataQA1 · 2020-03-14T07:37:37Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2467/

CarbonDataQA1 · 2020-03-16T12:04:36Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2479/

CarbonDataQA1 · 2020-03-16T12:15:15Z

Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/771/

ajantha-bhat · 2020-03-17T04:45:46Z

core/src/main/java/org/apache/carbondata/core/datamap/TableDataMap.java

@@ -135,7 +136,7 @@ public CarbonTable getTable() {
    int datamapsCount = 0;
    // In case if filter has matched partitions, then update the segments with datamap's
    // segment list, as getDataMaps will return segments that matches the partition.
-    if (null != partitions && !partitions.isEmpty()) {
+    if (isFilterPresent) {


This reset of segments has to happen only when the partition and filter both are present.

Now in case of non-partition + just the filter also we are resetting.

Now, we are loading cache for only the matched segments for filter (partition/non-partition). For non-partition table, based on segment level min-max. Hence we have to reset the segments.

akashrn5 · 2020-03-18T12:09:04Z

core/src/main/java/org/apache/carbondata/core/datamap/TableDataMap.java

@@ -135,7 +136,7 @@ public CarbonTable getTable() {
    int datamapsCount = 0;
    // In case if filter has matched partitions, then update the segments with datamap's


modify the comment according to new solution.

akashrn5 · 2020-03-18T12:36:44Z

...rc/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMapFactory.java

+              columnSchema.getColumnProperties().get(CarbonCommonConstants.COLUMN_DRIFT);
+          if (null != isSortColumn) {
+            if (isSortColumn.equalsIgnoreCase("true") && !isSortColumnInBlock) {
+              modifyColumnSchemaForSortColumn(columnSchema, isColumnDriftInBlock, isColumnDrift);


please send sort info also modify method. Both set and unset can be done from that method

akashrn5 · 2020-03-18T12:40:25Z

...rc/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMapFactory.java

+      ColumnSchema columnSchema = tableColumnSchemas.get(columnMetaData.getKey());
+      if (null != columnSchema) {
+        // get segment sort column and column drift info
+        boolean isSortColumnInBlock = columnMetaData.getValue().isSortColumn();


please rename

akashrn5 · 2020-03-18T12:53:16Z

core/src/main/java/org/apache/carbondata/core/metadata/schema/table/column/ColumnSchema.java

@@ -600,4 +604,20 @@ public boolean isIndexColumn() {
  public void setIndexColumn(boolean indexColumn) {
    this.indexColumn = indexColumn;
  }
+
+  public ColumnSchema clone() {
+    try {


you can use try with resource else stream will be open if write fails

akashrn5 · 2020-03-18T13:01:03Z

core/src/main/java/org/apache/carbondata/core/segmentmeta/SegmentMetaDataInfoStats.java

+      BlockColumnMetaDataInfo currentBlockColumnMetaInfo) {
+    // check if tableName is present in tableSegmentMetaDataInfoMap
+    if (!this.tableSegmentMetaDataInfoMap.isEmpty() && null != this.tableSegmentMetaDataInfoMap
+        .get(tableName) && !this.tableSegmentMetaDataInfoMap.get(tableName).isEmpty()


i thin !this.tableSegmentMetaDataInfoMap.get(tableName).isEmpty() this check is not needed, please check again and remove if not required

akashrn5 · 2020-03-18T13:04:04Z

...ion/spark/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessorStepOnSpark.scala

@@ -26,12 +26,13 @@ import org.apache.spark.sql.Row
 import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.expressions.GenericInternalRow
 import org.apache.spark.TaskContext
-import org.apache.spark.util.LongAccumulator
+import org.apache.spark.util.{CollectionAccumulator, LongAccumulator}


akashrn5 · 2020-03-18T13:12:18Z

integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala

            new Segment(seg.getLoadName, file)
          }.filter(_.getSegmentFileName != null).asJava
          segmentFilesForIUDCompact = new util.ArrayList[Segment](segmentFilesList)
        } else {
+          // get segmentMetadata info from accumulator
+          val segmentMetaDataInfo = CarbonDataRDDFactory.getSegmentMetaDataInfoFromAccumulator(


getSegmentMetaDataInfoFromAccumulator move this method to carbonLoaderUtil as its used in all load flows

akashrn5 · 2020-03-18T13:13:22Z

...tion/spark/src/main/scala/org/apache/carbondata/spark/rdd/InsertTaskCompletionListener.scala

  extends CarbonLoadTaskCompletionListener {
  override def onTaskCompletion(context: TaskContext): Unit = {
    try {
-      dataLoadExecutor.close()
+      // fill segment level minMax to accumulator
+      CarbonDataRDDFactory.fillSegmentMetaDataInfoToAccumulator(tableName,


fillSegmentMetaDataInfoToAccumulator same as above comment

akashrn5 · 2020-03-18T13:16:07Z

...ation/spark/src/main/scala/org/apache/spark/sql/secondaryindex/util/SecondaryIndexUtil.scala

@@ -203,6 +203,7 @@ object SecondaryIndexUtil {
              seg.getLoadName,
              segmentIdToLoadStartTimeMapping(seg.getLoadName).toString,
              carbonLoadModel.getFactTimeStamp.toString,
+              null,


please add a subtask in jira to support for SI

CarbonDataQA1 · 2020-03-19T09:48:18Z

Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/804/

CarbonDataQA1 · 2020-03-19T09:55:58Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2509/

akashrn5 · 2020-03-19T17:16:24Z

LGTM

ajantha-bhat · 2020-03-21T12:23:45Z

@Indhumathi27 : make that carbon property deafult value to false
and In query flow, If the table is transactional table and segment minmax is not set. Throw runtime exception. So, that CI can catch all the missed scenarios.

is it verified ?

ajantha-bhat · 2020-03-22T01:39:38Z

docs/configuration-parameters.md

@@ -146,6 +146,7 @@ This section provides the details of all the configurations required for the Car
 | carbon.query.prefetch.enable | true | By default this property is true, so prefetch is used in query to read next blocklet asynchronously in other thread while processing current blocklet in main thread. This can help to reduce CPU idle time. Setting this property false will disable this prefetch feature in query. |
 | carbon.query.stage.input.enable | false | Stage input files are data files written by external applications (such as Flink), but have not been loaded into carbon table. Enabling this configuration makes query to include these files, thus makes query on latest data. However, since these files are not indexed, query maybe slower as full scan is required for these files. |
 | carbon.driver.pruning.multi.thread.enable.files.count | 100000 | To prune in multi-thread when total number of segment files for a query increases beyond the configured value. |
+| carbon.load.all.indexes.to.cache | true | Setting this configuration to false, will prune and load only matched segment indexes to cache using segment metadata information such as columnid and it's minmax values, which decreases the usage of driver memory.  |


This is renamed now right ? please update

… less driver memory usage

CarbonDataQA1 · 2020-03-22T10:25:26Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2536/

CarbonDataQA1 · 2020-03-22T11:18:16Z

Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/829/

Indhumathi27 · 2020-03-22T11:24:58Z

retest this please

CarbonDataQA1 · 2020-03-22T13:28:56Z

Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/830/

CarbonDataQA1 · 2020-03-22T13:52:22Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2537/

ajantha-bhat · 2020-03-23T02:59:20Z

...rc/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMapFactory.java

+        getTableBlockUniqueIdentifierWrappers(partitionsToPrune,
+            tableBlockIndexUniqueIdentifierWrappers, identifiers);
+      } else {
+        SegmentMetaDataInfo segmentMetaDataInfo = segment.getSegmentMetaDataInfo();


tableBlockIndexUniqueIdentifierWrappers is just a wrapper around TableBlockIndexUniqueIdentifier.
So,we should avoid creating TableBlockIndexUniqueIdentifier itself If that filter doesn't match segment min max. [line 153, should not be called if isScanRequired is false]

If we keep SegmentFileStore inside Segment class and read the segment before calling readCommittedScope.committedIndexFiles. we can avoid creating the TableBlockIndexUniqueIdentifier object. May be more changes now. Can do in other PR.

ok. Will take up in future PR

ajantha-bhat · 2020-03-23T05:45:08Z

LGTM

Indhumathi27 force-pushed the cache branch from d6d443f to fc23d62 Compare January 19, 2020 01:40

xuchuanyin reviewed Jan 19, 2020

View reviewed changes

core/src/main/java/org/apache/carbondata/core/util/SegmentMinMax.java Outdated Show resolved Hide resolved

Indhumathi27 force-pushed the cache branch from fc23d62 to 091b3b3 Compare January 22, 2020 08:59

Indhumathi27 force-pushed the cache branch from 091b3b3 to b9ef7d2 Compare January 23, 2020 08:45

Indhumathi27 force-pushed the cache branch from b9ef7d2 to 95c4389 Compare January 28, 2020 15:59

Indhumathi27 force-pushed the cache branch from 95c4389 to 897645c Compare January 29, 2020 06:05

Indhumathi27 force-pushed the cache branch 2 times, most recently from f34d7d2 to 46d59d0 Compare February 14, 2020 10:44

akashrn5 reviewed Feb 14, 2020

View reviewed changes

Indhumathi27 force-pushed the cache branch from 46d59d0 to d108110 Compare February 14, 2020 13:56

Indhumathi27 force-pushed the cache branch from d108110 to 3ad16f5 Compare February 14, 2020 15:40

Indhumathi27 force-pushed the cache branch from 3ad16f5 to 47ad303 Compare February 14, 2020 17:30

Indhumathi27 force-pushed the cache branch from 47ad303 to 48187b1 Compare February 15, 2020 10:38

Indhumathi27 force-pushed the cache branch from 09e6aeb to 8c08a55 Compare March 14, 2020 04:53

Indhumathi27 changed the title ~~[WIP][CARBONDATA-3718] Support SegmentLevel MinMax for better Pruning and less driver memory usage for cache~~ [CARBONDATA-3718] Support SegmentLevel MinMax for better Pruning and less driver memory usage for cache Mar 14, 2020

Indhumathi27 force-pushed the cache branch from 8c08a55 to b4ec963 Compare March 16, 2020 09:50

ajantha-bhat reviewed Mar 17, 2020

View reviewed changes

akashrn5 reviewed Mar 18, 2020

View reviewed changes

Indhumathi27 force-pushed the cache branch 2 times, most recently from 537ecff to cefb94e Compare March 19, 2020 07:39

ajantha-bhat reviewed Mar 22, 2020

View reviewed changes

Indhumathi27 added 5 commits March 22, 2020 13:45

[CARBONDATA-3718] Support segment level minmax for better pruning and…

838958d

… less driver memory usage

fix comments

d9866ea

new changes

76ccae5

fix failures

17aff4b

fix comments and failures

6a331f3

Indhumathi27 force-pushed the cache branch from cefb94e to 6a331f3 Compare March 22, 2020 08:16

ajantha-bhat reviewed Mar 23, 2020

View reviewed changes

asfgit closed this in d3204f7 Mar 23, 2020


		public static final String CARBON_LOAD_ALL_INDEX_TO_CACHE = "carbon.load.all.indexes.to.cache";

		public static final String CARBON_LOAD_ALL_INDEX_TO_CACHE_DEFAULT = "true";


		import org.apache.carbondata.core.util.SegmentMinMax;

		public class SegmentBlockInfo {

		@@ -135,7 +136,7 @@ public CarbonTable getTable() {
		int datamapsCount = 0;
		// In case if filter has matched partitions, then update the segments with datamap's

[CARBONDATA-3718] Support SegmentLevel MinMax for better Pruning and less driver memory usage for cache #3584

[CARBONDATA-3718] Support SegmentLevel MinMax for better Pruning and less driver memory usage for cache #3584

Conversation

Indhumathi27 commented Jan 17, 2020 • edited

Why is this PR needed?

What changes were proposed in this PR?

Does this PR introduce any user interface change?

Is any new testcase added?

CarbonDataQA1 commented Jan 17, 2020

CarbonDataQA1 commented Jan 19, 2020

CarbonDataQA1 commented Jan 22, 2020

CarbonDataQA1 commented Jan 23, 2020

CarbonDataQA1 commented Jan 28, 2020

CarbonDataQA1 commented Jan 29, 2020

CarbonDataQA1 commented Feb 3, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 15, 2020

CarbonDataQA1 commented Mar 14, 2020

CarbonDataQA1 commented Mar 14, 2020

CarbonDataQA1 commented Mar 16, 2020

CarbonDataQA1 commented Mar 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA1 commented Mar 19, 2020

CarbonDataQA1 commented Mar 19, 2020

akashrn5 commented Mar 19, 2020

ajantha-bhat commented Mar 21, 2020

ajantha-bhat Mar 22, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA1 commented Mar 22, 2020

CarbonDataQA1 commented Mar 22, 2020

Indhumathi27 commented Mar 22, 2020

CarbonDataQA1 commented Mar 22, 2020

CarbonDataQA1 commented Mar 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajantha-bhat commented Mar 23, 2020

Indhumathi27 commented Jan 17, 2020 •

edited

ajantha-bhat Mar 22, 2020 •

edited