[SPARK-29550][SQL] - enhance session catalog locking

[choojoyq]

What changes were proposed in this pull request?

In my streaming applicationspark.streaming.concurrentJobs is set to 50 which is used as size for underlying thread pool. I perform several sql operations on dataframes and automatically create/alter tables/view in runtime. I order to do that i invoke create ... if not exists operations on driver on each batch invocation. Once i noticed that most of batch time is spent on driver but not on executors. I made a thread dump and figured out that most of the threads are blocked on SessionCatalog operation waiting for a lock.

Existing implementation of SessionCatalog uses a single lock which is used almost by all the methods to guard currentDb and tempViews variables. I propose to enhance locking behaviour of SessionCatalog by :

1. Employing ReadWriteLock which allows to execute read operations concurrently.
2. Replace synchronized with the corresponding read or write lock.

Also it's possible to go even further and strip locks for currentDb and tempViews but i'm not sure whether it's possible from the implementation point of view.
Probably someone will help me with this?

How was this patch tested?

Only via existing test suits.

[AmplabJenkins]

Can one of the admins verify this patch?

[choojoyq]

@andrewor14 could you please take a look ?

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!