[SUPPORT] please rollback greater commits first

[tooptoop4]

hudi 0.5.3 is giving this error (first time using 0.5.3), all of the data before was written by hoodie 0.4.6.

2 Qs:

1. is this new error something introduced in new version? is there some config I can use to disable that error check?
2. I assume hive/presto.etc when querying this COW table does not see any data from the inflight files? can I safely delete the inflight files manually without risk of data loss/corruption (I'm talking about data that is currently queryable, not data that failed to be inserted in the past and was never queryable anyway)

2020-07-02 18:46:45,130 [main] INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants [[==>20190928011208__commit__INFLIGHT], [==>20190928034716__commit__INFLIGHT], [==>20190928200315__commit__INFLIGHT], [==>20190930151449__commit__INFLIGHT], [20191001132843__clean__COMPLETED], [20191001132843__commit__COMPLETED], [20191001132911__clean__COMPLETED], [20191001132911__commit__COMPLETED], [20191001133118__clean__COMPLETED], [20191001133118__commit__COMPLETED], [20191001133148__clean__COMPLETED], [20191001133148__commit__COMPLETED], [20191001133355__clean__COMPLETED], [20191001133355__commit__COMPLETED], [20191001133427__clean__COMPLETED], [20191001133427__commit__COMPLETED], [20191001133631__clean__COMPLETED], [20191001133631__commit__COMPLETED], [20191001133659__clean__COMPLETED], [20191001133659__commit__COMPLETED], [20191001133901__clean__COMPLETED], [20191001133901__commit__COMPLETED], [20191001133931__clean__COMPLETED], [20191001133931__commit__COMPLETED], [20191001134052__clean__COMPLETED], [20191001134052__commit__COMPLETED], [20191001134120__clean__COMPLETED], [20191001134120__commit__COMPLETED], [20191001134247__clean__COMPLETED], [20191001134247__commit__COMPLETED], [20191001134319__clean__COMPLETED], [20191001134319__commit__COMPLETED], [20191002120546__clean__COMPLETED], [20191002120546__commit__COMPLETED], [20191002120745__clean__COMPLETED], [20191002120745__commit__COMPLETED], [20191002122157__clean__COMPLETED], [20191002122157__commit__COMPLETED], [==>20191002122317__commit__INFLIGHT], [20191002122537__clean__COMPLETED], [20191002122537__commit__COMPLETED], [20191002122618__clean__COMPLETED], [20191002122618__commit__COMPLETED], [20191002123141__clean__COMPLETED], [20191002123141__commit__COMPLETED], [==>20191002123247__commit__INFLIGHT], [20191002132703__clean__COMPLETED], [20191002132703__commit__COMPLETED], [20191002132746__clean__COMPLETED], [20191002132746__commit__COMPLETED], [20191003112112__clean__COMPLETED], [20191003112112__commit__COMPLETED], [20191003112202__clean__COMPLETED], [20191003112202__commit__COMPLETED], [==>20191202224953__commit__INFLIGHT], [20191202225507__clean__COMPLETED], [20191202225507__commit__COMPLETED], [20191202230001__clean__COMPLETED], [20191202230001__commit__COMPLETED], [20191202231318__clean__COMPLETED], [20191202231318__commit__COMPLETED], [20191202231401__clean__COMPLETED], [20191202231401__commit__COMPLETED]]
Exception in thread "main" org.apache.hudi.exception.HoodieRollbackException: Found commits after time :20191202224953, please rollback greater commits first
at org.apache.hudi.client.AbstractHoodieWriteClient.doRollbackAndGetStats(AbstractHoodieWriteClient.java:358)
at org.apache.hudi.client.AbstractHoodieWriteClient.rollbackInternal(AbstractHoodieWriteClient.java:322)
at org.apache.hudi.client.HoodieWriteClient.rollback(HoodieWriteClient.java:716)
at org.apache.hudi.client.HoodieWriteClient.rollbackPendingCommits(HoodieWriteClient.java:1017)
at org.apache.hudi.client.HoodieWriteClient.startCommit(HoodieWriteClient.java:846)

[vinothchandar]

Actually this behavior of rolling back inflight before starting the best commit has been there forever in hudi..

Do you have any idea how you wrote these commits?

[==>20191202224953__commit__INFLIGHT], [20191202225507__clean__COMPLETED], [20191202225507__commit__COMPLETED], [20191202230001__clean__COMPLETED], [20191202230001__commit__COMPLETED], [20191202231318__clean__COMPLETED], [20191202231318__commit__COMPLETED], [20191202231401__clean__COMPLETED], [20191202231401__commit__COMPLETED]]

[vinothchandar]

As for upgrading, the data should not queryable as long as these commits were on the active timeline. So you should be able to do a manual rollback (I recommend using the cli to rollback to be safe)

[vinothchandar]

http://hudi.apache.org/releases.html Has all the changes for upgrading . Please read thru and We can take questions you may have around this

[tooptoop4]

@vinothchandar changing true to false on https://github.com/apache/hudi/blob/release-0.5.3/hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java#L195 fixes this for me. Any issues from this change?

[bhasudha]

@vinothchandar changing true to false on https://github.com/apache/hudi/blob/release-0.5.3/hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java#L195 fixes this for me. Any issues from this change?

@tooptoop4 Setting false to this ensures the ingestion path does not bother about rolling back inflight commits before starting a new commit. That will certainly get rid of the above exception. But, the inflight commits will still need to be cleaned up manually using the cli.

[tooptoop4]

@bhasudha what if i don't clean them up? ie if i just leave them there then what is the issue?

[bhasudha]

@bhasudha what if i don't clean them up? ie if i just leave them there then what is the issue?

It shouldn't have any impact on queries. since inflight commits are not visible to queries anyways.

[bvaradar]

@bhasudha what if i don't clean them up? ie if i just leave them there then what is the issue?

@tooptoop4 : You should be cleaning these files before we archive to prevent reading partially written parquet files which were written during the failed commit.

[bvaradar]

closing this ticket. @tooptoop4 : if you have any more questions, please open a new ticket or contact in slack.