[Improvement]: Protect from out-of-memory error when optimizing a table with many files.

[zhoujinsong]

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

Many processes like optimizing planning will load file metadata of tables into memory, which may cause OOM errors if processing tables with many files.

When dealing with tables with many files, we should limit memory usage to avoid OOM issues.

How should we improve?

Limit memory usage when dealing with tables.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Subtasks

• [Subtask]: Remove IcebergContentFile and related codes #1839
• [Subtask]: Do not cache DataFile in memory when evaluating tables. #1840
• [Subtask]: Drop stat information from delete files. #1841
• [Subtask]: Perform optimizing plan only on a few partitions to reduce memory consumption #1842

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'