[CARBONDATA-2749][dataload] In HDFS Empty tablestatus file is written during datalaod, iud or compaction when disk is full. #2517

mohammadshahidkhan · 2018-07-17T11:38:05Z

Problem:
When a failure happens due to disk full during load, IUD or Compaction, then while updating the tablestatus file, the tablestaus.tmp file during atomic file operation remains empty, and in the finally block the empty tablestaus.tmp file is getting renamed to the actual file. This leads to empty tablestatus file. Once such problem happens the tablestatus file can not be retrieved and the already loaded data can be used.
Solution:
If the failures happens during write then the the schema rename in the finally
block must be avoided.

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

Any interfaces changed?
NA
Any backward compatibility impacted?
NA
Document update required?
NA
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
NA
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA

CarbonDataQA · 2018-07-17T18:39:27Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6024/

CarbonDataQA · 2018-07-17T19:47:12Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7260/

mohammadshahidkhan · 2018-07-18T05:15:29Z

retest SDV please

ravipesala · 2018-07-18T13:06:12Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5903/

mohammadshahidkhan · 2018-07-18T13:08:26Z

The failled test case is unrelated and random.
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5903/

KanakaKumar · 2018-07-18T14:54:38Z

core/src/main/java/org/apache/carbondata/core/fileoperations/AtomicFileOperationsImpl.java

    }

  }

+  @Override public void setFailed() {


setFailed needs to be raised on Exception cases in the below usages of AtomicFileOperations

org.apache.carbondata.core.datamap.status.DiskBasedDataMapStatusProvider#writeLoadDetailsIntoFile

org.apache.carbondata.core.metadata.SegmentFileStore#writeSegmentFile(org.apache.carbondata.core.metadata.SegmentFileStore.SegmentFile, java.lang.String)

org.apache.carbondata.core.statusmanager.SegmentStatusManager#writeLoadMetadata

Please also handle any other references I missed.

CarbonDataQA · 2018-07-20T09:00:43Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7349/

CarbonDataQA · 2018-07-20T09:39:23Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6113/

brijoobopanna · 2018-07-22T16:16:05Z

retest sdv please

ravipesala · 2018-07-22T17:35:45Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5955/

mohammadshahidkhan · 2018-07-23T08:42:17Z

retest sdv please

KanakaKumar · 2018-07-23T15:56:36Z

core/src/main/java/org/apache/carbondata/core/fileoperations/AtomicFileOperationsImpl.java

      }
+    } else {
+      LOGGER.warn("The temporary file renaming skipped due to I/O error, deleting file "


Here actually delete code is added. AtomicFileOperationsImpl is taking care of overwriting temp file even if it exists. So, please correct the message accordingly.

@KanakaKumar Thanks for the review.
Actually when the delete fails the temp file will persists and its better to delete it at the same time, to avoid problem in secure environment.

ravipesala · 2018-07-23T19:07:30Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5966/

KanakaKumar · 2018-07-24T10:31:59Z

LGTM

CarbonDataQA · 2018-07-24T12:33:16Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7450/

CarbonDataQA · 2018-07-24T14:28:17Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6207/

ravipesala · 2018-07-24T19:08:00Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5982/

CarbonDataQA · 2018-07-25T08:36:04Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7489/

CarbonDataQA · 2018-07-25T09:18:42Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6247/

CarbonDataQA · 2018-07-25T12:50:08Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7499/

CarbonDataQA · 2018-07-25T14:04:56Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7504/

CarbonDataQA · 2018-07-25T16:27:18Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6261/

brijoobopanna · 2018-07-25T17:05:05Z

retest sdv please

brijoobopanna · 2018-07-26T04:13:11Z

retest sdv please

brijoobopanna · 2018-07-26T08:38:50Z

retest sdv please

ravipesala · 2018-07-26T08:58:43Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6001/

brijoobopanna · 2018-07-26T12:24:24Z

retest sdv please

ravipesala · 2018-07-26T19:27:40Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6012/

mohammadshahidkhan · 2018-07-27T05:03:48Z

retest sdv please

manishgupta88 · 2018-07-27T11:48:49Z

core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java

@@ -724,6 +724,7 @@ public void writeLoadDetailsIntoFile(List<SegmentUpdateDetails> listOfSegmentUpd
      brWriter.write(metadataInstance);
    } catch (IOException ioe) {
      LOG.error("Error message: " + ioe.getLocalizedMessage());
+      fileWrite.setFailed();


At some places IO exception is thrown back after setting the flag and some places exception is not thrown..any specific reason for this?...can we keep the behavior same..better to throw back the exception at all places

@manishgupta88 Thanks for the review.
Agreed with your point eating exception is not a good idea, handled throwing back at all the places.

… during datalaod, iud or compaction when disk is full.

ravipesala · 2018-07-27T14:07:33Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6025/

CarbonDataQA · 2018-07-27T14:53:10Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7572/

CarbonDataQA · 2018-07-27T17:34:38Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6330/

ravipesala · 2018-07-28T09:29:15Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6038/

kunal642 · 2018-07-30T06:10:14Z

LGTM. The failures in 2.2.1 build are note related to this PR. Those are failing on master as well. Need to fix.

during datalaod, iud or compaction when disk is full. Problem: When a failure happens due to disk full during load, IUD or Compaction, then while updating the tablestatus file, the tablestaus.tmp file during atomic file operation remains empty, and in the finally block the empty tablestaus.tmp file is getting renamed to the actual file. This leads to empty tablestatus file. Once such problem happens the tablestatus file can not be retrieved and the already loaded data can be used. Solution: If the failures happens during write then the the schema rename in the finally block must be avoided. This closes #2517

during datalaod, iud or compaction when disk is full. Problem: When a failure happens due to disk full during load, IUD or Compaction, then while updating the tablestatus file, the tablestaus.tmp file during atomic file operation remains empty, and in the finally block the empty tablestaus.tmp file is getting renamed to the actual file. This leads to empty tablestatus file. Once such problem happens the tablestatus file can not be retrieved and the already loaded data can be used. Solution: If the failures happens during write then the the schema rename in the finally block must be avoided. This closes apache#2517

KanakaKumar reviewed Jul 18, 2018

View reviewed changes

mohammadshahidkhan force-pushed the tablestatus_fix branch from 46b253e to d3d0484 Compare July 20, 2018 07:24

KanakaKumar reviewed Jul 23, 2018

View reviewed changes

mohammadshahidkhan force-pushed the tablestatus_fix branch from d3d0484 to 843fd9c Compare July 24, 2018 09:22

mohammadshahidkhan force-pushed the tablestatus_fix branch 2 times, most recently from 69c253f to cd03a27 Compare July 25, 2018 07:03

mohammadshahidkhan force-pushed the tablestatus_fix branch 2 times, most recently from 678f529 to e49c4ca Compare July 25, 2018 09:55

manishgupta88 reviewed Jul 27, 2018

View reviewed changes

[CARBONDATA-2749][dataload] In HDFS Empty tablestatus file is written…

af06cb7

… during datalaod, iud or compaction when disk is full.

mohammadshahidkhan force-pushed the tablestatus_fix branch from e49c4ca to af06cb7 Compare July 27, 2018 12:09

asfgit closed this in 7628571 Jul 30, 2018

[CARBONDATA-2749][dataload] In HDFS Empty tablestatus file is written during datalaod, iud or compaction when disk is full. #2517

[CARBONDATA-2749][dataload] In HDFS Empty tablestatus file is written during datalaod, iud or compaction when disk is full. #2517

Conversation

mohammadshahidkhan commented Jul 17, 2018 • edited

CarbonDataQA commented Jul 17, 2018

CarbonDataQA commented Jul 17, 2018

mohammadshahidkhan commented Jul 18, 2018

ravipesala commented Jul 18, 2018

mohammadshahidkhan commented Jul 18, 2018

KanakaKumar Jul 18, 2018

Choose a reason for hiding this comment

mohammadshahidkhan Jul 20, 2018

Choose a reason for hiding this comment

CarbonDataQA commented Jul 20, 2018

CarbonDataQA commented Jul 20, 2018

brijoobopanna commented Jul 22, 2018

ravipesala commented Jul 22, 2018

mohammadshahidkhan commented Jul 23, 2018

KanakaKumar Jul 23, 2018

Choose a reason for hiding this comment

mohammadshahidkhan Jul 24, 2018

Choose a reason for hiding this comment

KanakaKumar Jul 24, 2018

Choose a reason for hiding this comment

ravipesala commented Jul 23, 2018

KanakaKumar commented Jul 24, 2018

CarbonDataQA commented Jul 24, 2018

CarbonDataQA commented Jul 24, 2018

ravipesala commented Jul 24, 2018

CarbonDataQA commented Jul 25, 2018

CarbonDataQA commented Jul 25, 2018

CarbonDataQA commented Jul 25, 2018

CarbonDataQA commented Jul 25, 2018

CarbonDataQA commented Jul 25, 2018

brijoobopanna commented Jul 25, 2018

brijoobopanna commented Jul 26, 2018

brijoobopanna commented Jul 26, 2018

ravipesala commented Jul 26, 2018

brijoobopanna commented Jul 26, 2018

ravipesala commented Jul 26, 2018

mohammadshahidkhan commented Jul 27, 2018

manishgupta88 Jul 27, 2018

Choose a reason for hiding this comment

mohammadshahidkhan Jul 27, 2018

Choose a reason for hiding this comment

ravipesala commented Jul 27, 2018

CarbonDataQA commented Jul 27, 2018

CarbonDataQA commented Jul 27, 2018

ravipesala commented Jul 28, 2018

kunal642 commented Jul 30, 2018

mohammadshahidkhan commented Jul 17, 2018 •

edited