Fixing purview test issues and improve performance

[xiaoyongzhu]

Currently there are two issues:

1. Failed to connect to PurView caused CI tests failure #349 where connecting to Purview always yield this error:

E           requests.exceptions.ConnectionError: HTTPSConnectionPool(host='some-purview-name.catalog.purview.azure.com', port=443): Max retries exceeded with url: /api/atlas/v2/types/typedefs (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f90c3e7ea60>: Failed to establish a new connection: [Errno 111] Connection refused'))

2. Listing entities in purview is extremely slow.

The current way we get features/list features for a project isn't very scalable. What we are currently doing is to first get all the entities, and filter out the ones that we need on the client side, which sometimes causes the service to throttle. We are also calling purview repeatedly for entities that we already fetched.

This PR solves those issue by issuing a server side filtering query, as well as optimize the get_features_from_registry logic to avoid duplicated purview calls.

Call time of get_features_from_registry() can be reduced from ~20s to around 3s now.

[xiaoyongzhu]

@YihuiGuo this should also fix your earlier issue

[windoze]

Could you please add more tech details and investigations about the bug? I'm not sure what happened actually.
To confirm that the server side throttling is the root cause, you may need try to change server side parameters to relax the restriction and check if it fixes the problem, otherwise the root cause could be somewhere else.

[xiaoyongzhu]

Could you please add more tech details and investigations about the bug? I'm not sure what happened actually. To confirm that the server side throttling is the root cause, you may need try to change server side parameters to relax the restriction and check if it fixes the problem, otherwise the root cause could be somewhere else.

As described in the PR, the implementation of the search_entities API in the underlying pyapacheatlas package is here (https://github.com/wjohnson/pyapacheatlas/blob/master/pyapacheatlas/core/discovery/purview.py#L226) where it's using a while True statement to keep querying the query API, and might got throttled.

More details are available in this issue: wjohnson/pyapacheatlas#206 where I talked with Will offline, and he agrees to add some backoff in the search_entities API.

[xiaoyongzhu]

Also update the PR description to make it a bit more descriptive.

[blrchen]

[windoze]

The canonical way to retrieve entity/entities from PurView is to fetch by GUID, also we already stored related GUIDs in our data model.
Querying/Searching is not the correct approach to load a project and may cause unwanted result if this PurView contains data other than what we've created by Feathr, it only fits for-eyes-only scenarios but not cases require accuracy and consistency such as Feathr client.
Please re-think and re-design the whole solution.

[blrchen]

The canonical way to retrieve entity/entities from PurView is to fetch by GUID, also we already stored related GUIDs in our data model.

I agree, when CLI save data to purview, it does not specify any hits to partition data by project, that means search via startsWith might still experience perf issues when data volume grows. Since CLI already writes lineage relationship, for example, project contains feature/anchor/derived, using guid list to fetch registered features sounds more efficient and scalable.

[xiaoyongzhu]

The canonical way to retrieve entity/entities from PurView is to fetch by GUID, also we already stored related GUIDs in our data model. Querying/Searching is not the correct approach to load a project and may cause unwanted result if this PurView contains data other than what we've created by Feathr, it only fits for-eyes-only scenarios but not cases require accuracy and consistency such as Feathr client. Please re-think and re-design the whole solution.

Agree, but the goal of this PR is not to solve all those issues. I have a separate PR solving those issues and please take a look: #368