feat(ingest): powerbi # store powerbi entity descriptions

[looppi]

Store description fields of Datasets, Reports and Dashboards from PowerBI REST API to ingested data. Showing the possible description of the entity in Datahub improves visibility and adds value to Datahub as there's no need to jump to other system just to see the documentation.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[jjoyce0510]

(Rebased on latest master)

[siddiquebagwan]

Can we return None from here ? and make return type as optional[str]

[looppi]

The DashboardInfoClass expects description to be str and not Optional[str], if I change the return value to be optional, then all the descriptions change to description=formulate_description(....) or "" which doesn't look too good to my eye. I'd like to keep the behavior as is just because it looks cleaner, what do you think?

[siddiquebagwan]

If that is the case then use str() , it returns empty string. and this is the default value for description in DashboardInfoClass

[looppi]

I'm not sure whether I fixed it with my change, but I tried! :)

[jjoyce0510]

I still don't quite get why we are changing the description to include the display name like this?

Feels like we should only take the raw description, not the display name

At min, we should leave a comment over this method describing why this is even necessary

[jjoyce0510]

I like that we are correctly extracting descriptions - totally jive with that. But why are we inserting the display name at the top? Feels a bit odd to change the description at all prior to ingestion? Hoping to gain answer before we merge this

[jjoyce0510]

After syncing with @mohdsiddique , we agree that we should first remove any additional formatting from the description before merging this PR.

Thanks
John

[looppi]

Originally I kept the entity name in the description field just because it was there before my changes. Now I think the common consensus lies in the fact that there shouldn't be entity name in the description field. I think that's the only way to not use any formatting in the description field and keep the data readable. Certainly that makes a lot of sense. So, thinking from this point of view, would the PR be more mergeable if I just remove the formulate_description function and use only the description field content everywhere?