[ARCTIC-1057][AMS] Only load self-optimizing enabled tables into cache

[XBaith]

Why are the changes needed?

This PR was inspired by related issue #1057 , but was not created for implementation.

AMS can not handle large scale Iceberg tables which has a certain number of small files/delete files. Some of tables in iceberg catalog are not optimizing enabled, so these tables are not nessessary for AMS to scan or load into memory.
I assume that self-optimizing disabled tables allocate a mount of memory and compare the GC times after remove them:

• Load all valid tables:
  
• Only load self-optimizing tables:
  

The below sreenshot lists number of instances and retained heap per class:

I also dump it and find that ConcurrentHashMap in OptimizeService retained too much heap

• case 1: Too many snapshots entry
  
  

• case 2: Too many delete files
  

Brief change log

• Reduce AMS memory
• Avoid load/display useless tables for AMS optimize
• Clean the tasks when self-optimizing disabled

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before making a pull request

Documentation

• Does this pull request introduces a new feature? (no)
• If yes, how is the feature documented? (not documented)

[XBaith]

#1057

[wangtaohz]

Thanks for your contribution! I will help to review it ASAP.

[zhoujinsong]

#1057

Thanks a lot for your contribution!
But I have also noticed that the PR title and description do not match the Arctic community requirement.
You can find the Arctic community requirement here:
https://docs.google.com/document/d/1Pwv1LflW3z5yq34qDpEYwSxOB3LIE5qoKQ_5Sv0IV1Y/edit?usp=sharing

[XBaith]

Okay

[wangtaohz]

@XBaith I left some comments, please take a look.

[codecov]

Codecov Report

Patch coverage has no change and project coverage change: +23.41 🎉

Comparison is base (0e5140e) 29.27% compared to head (76231da) 52.68%.

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #1145       +/-   ##
=============================================
+ Coverage     29.27%   52.68%   +23.41%     
+ Complexity     5374      523     -4851     
=============================================
  Files           695       43      -652     
  Lines         70786     3705    -67081     
  Branches       8180      354     -7826     
=============================================
- Hits          20723     1952    -18771     
+ Misses        48058     1623    -46435     
+ Partials       2005      130     -1875     

Flag	Coverage Δ
core	?
trino	52.68% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 652 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[XBaith]

Hi @wangtaohz, change complete , please review if you have time.

[wangtaohz]

It's a bit difficult to understand disableSelfOptimizing and enableSelfOptimizing, since they are not doing what the method name describes.

I suggest it to be:

validTables(or cachedTables) contains two collections:

• optimizeTables
• unOptimizeTables

and the process refreshAndListTables contains 4 steps:

• add new tables: add tables not in validTables into optimizeTables or unOptimizeTables
• clear tables: remove tables from validTables (optimizeTables or unOptimizeTables)
• enable optimize: move tables from unOptimizeTables to optimizeTables
• disable optimize: move tables from optimizeTables to unOptimizeTables