[Improvement]: Eliminate AMS Full GC impact deriving from local terminal clean spark context

[nicochen]

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

We experienced and suffered from AMS full GC for a long period. we dig into this case and finally found out local spark session terminal triggers java.lang.System.gc() every 30 minutes to clean contexts. As arthas stack traces as shown below:

How should we improve?

For external kyuubi session terminal, there is no need to revises anything. The problem is only related to local session which is embedding into AMS itself. Simply add config 'spark.cleaner.referenceTracking' = 'false' to solve the problem.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Subtasks

No response

Code of Conduct

• I agree to follow this project's Code of Conduct