fix: retry loop for exception caused by deadlock on badge node

[allisonsuarez]

Summary of Changes

Explanation from @dikshathakur3119 :
We index Hive tables using index_table_mateadata and to expedite indexing all the hive table we are leveraging multiprocessing.
When a relationship is created, its endpoint nodes are write-locked. If multiple threads attempt to create relationships involving the same set of endpoint nodes, deadlocks can occur.
Earlier this deadlock could not take place since we are creating threads on the basis of shema name, so never used to process the same table node from different threads.
But we have recently added a new model - Badge, and we create a single badge for unique badge names, whether it is table level badge or column level badge. E.g there will be only one node with badge name ‘deprecated’ or ‘partition column’. We have n:n relationship between badge-table and badge-column. Now when we try to index table metadata, our task try to access the same badge node from different threads which causes a deadlock.

Tests

N/A

Documentation

N/A

CheckList

Make sure you have checked all steps below to ensure a timely review.

• PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.
• PR includes a summary of changes.
• PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
• PR passes make test

[allisonsuarez]

I have some open questions on the code comments that I need help with so please leave some comments with your thoughts @dikshathakur3119 @feng-tao @jinhyukchang

[dikshathakur3119]

Mostly looks good to me, and it solves the issue we are facing currently when we try to index hive tables in multiple processes. Do you think we should add note in our badge documents to make sure people understand that why sometimes a task of ingesting nodes may fail even after retries if they are trying to add same Badge node in parallel tasks?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[feng-tao]

CI fails?