RAT errors during release process

[andygrove]

Describe the bug

Running rat license checker on /home/andy/git/apache/arrow-datafusion/dev/dist/apache-arrow-datafusion-37.0.0-rc1/apache-arrow-datafusion-37.0.0.tar.gz
NOT APPROVED: .github/workflows/pr_benchmarks.yml (apache-arrow-datafusion-37.0.0/.github/workflows/pr_benchmarks.yml): false
NOT APPROVED: .github/workflows/pr_comment.yml (apache-arrow-datafusion-37.0.0/.github/workflows/pr_comment.yml): false
2 unapproved licences. Check rat report: rat.txt

To Reproduce

No response

Expected behavior

No response

Additional context

No response

[andygrove]

I plan on fixing this

[andygrove]

When I run the rat report locally, I see that these files do not have approved licenses, but somehow this is not detected in CI.

$ python dev/release/check-rat-report.py dev/release/rat_exclude_files.txt rat.txt 
NOT APPROVED: workflows/docs.yaml (.github/workflows/docs.yaml): false
NOT APPROVED: workflows/pr_benchmarks.yml (.github/workflows/pr_benchmarks.yml): false
NOT APPROVED: workflows/pr_comment.yml (.github/workflows/pr_comment.yml): false

[comphead]

ouch, thanks, but why it didn't complain before...

[andygrove]

ouch, thanks, but why it didn't complain before...

I'm not sure yet. It seems specific to all files in the .github directory.

[andygrove]

I think I understand the issue now. When running rat against a directory (as in CI) then it will exclude hidden directories such as .git and .github. When running against the release tarball, it does not ignore the .github directory.

The latest version of rat has a --scan-hidden-directories flag, but we really don't want to scan .git so we would need to find a way to exclude that. I will look at this some more tomorrow.