Flatbuffers dependency causing dirty git checkout

[suo]

For some reason, flatbuffer is not cleaning up after itself, and our assertion that at the end of each job the git checkout is clean is failing. This is the most significant issues currently affecting PyTorch CI flakiness.

See examples.

cc @ezyang @gchanan @zou3519 @bdhirsh @seemethere @malfet @pytorch/pytorch-dev-infra

[suo]

@gmagogsfm it looks like @qihqi is out. Can you find someone to take a look at this?

[malfet]

I though flatbuffer PR was reverted, wasn't it?

[gmagogsfm]

It was reverted (a few days ago actually), I checked master there is no reference to flatbuffer submodule. @suo, is this issue persistent now, or is it historical failure?

[ngimel]

The examples above show failures today and yesterday

[qihqi]

hi @suo Do you have a link with full log? Clicking any link on http://torchci.us-east-1.elasticbeanstalk.com/failure?capture=Build+left+local+git+repository+checkout+dirty gets back to a similar page...

[suo]

@qihqi sorry, my website was broken. You can click the links now

[suo]

Btw this is still ongoing and still the top cause of flaky job failures in CI. The fixed failures page shows samples from an hour ago.

[qihqi]

Investigation so far:

1. Looked at: https://github.com/pytorch/pytorch/runs/4595505484?check_suite_focus=true; the logs has phrases 2021-12-21T14:58:14.6109736Z Entering 'third_party/flatbuffers' meaning it has flatbuffers as a submodule.

2. It's corresponding PR is https://github.com/pytorch/pytorch/pull/67475/commits
   Looking at its corresponding repo state has no flatbuffers third_party/ nor in .gitmodules

3. Looking at logs see 2021-12-21T14:58:16.4262221Z [command]/usr/bin/git checkout --progress --force refs/remotes/pull/67475/merge 2021-12-21T14:58:16.5068645Z warning: unable to rmdir 'third_party/flatbuffers': Directory not empty 2021-12-21T14:58:16.7766848Z Previous HEAD position was 6fd69f10ce Merge . Meaning, when CI tried to checkout the diff to run test; CI attempted to delete third_party/flatbuffers because third_party/flatbuffers is not part of the diff. However, that deletion failed. then subsequence submodules init just crawled it leaving it in dirty status.

Based on above my deduction is that the machine that ran this CI has third_party/flatbuffers directory somehow from before. That before could because the machine ran some CIs of when flatbuffer change was in, and didn't clean properly.

I don't know how to fix it per-se. However, since we are rolling flatbuffer change out again: #70201 (and it has no CI failures on dirty change). So current course of action should be continuing with flatbuffer rollout then this problem will be fixed automatically.