[SUPPORT]The file being written is in an invalid state. Probably caused by an error thrown previously. Current state: COLUMN

[wwli05]

Describe the problem you faced

we are using flink compaction async.
sometimes, the same file group compact task will be distributed to different task manager even though the compaction plan distribute the compact task using keyby filegroupid(hudi-4397),

this will lead to the exception:
The file being written is in an invalid state. Probably caused by an error thrown previously. Current state: COLUMN

To Reproduce
Steps to reproduce the behavior:

1. job manager is killed and restart
   2.all task managers register again
   3.say task manager1 will be assigned different file group id compact task, which maybe distribute to task manager2 before task manager re-register

i noticed that, when task manager re-registered, the compact task will continue to run which started before this re-register happened

Environment Description

• Hudi version : 0.11.0 + hudi-4397

• flink version : 1.14.4

• Running on Docker? (yes/no) : yes

[nsivabalan]

@yuzhaojing @danny0405 : Can you folks follow up when you get a chance.

[danny0405]

We have figured some reason in: https://issues.apache.org/jira/browse/HUDI-4880, can you help to validate with the fix ?

[wwli05]

will validate it, maybe take long time to confirm, thank you ~