test(ingest/nifi): Configure nifi tests for arm64

[asikowitz]

Bumps the nifi version which changes the custom property set (now expanded).
Also checks a url every 10 seconds for when nifi is done executing, rather than sleeping for 120 seconds always

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[mayurinehate]

The wait is not only for the nifi to start up but also for nifi workflow to execute and process all events. The current check only verifies if nifi is up. To check whether workflow has run completely and it has generated necessary provenance events, we'll need to call provenance api here and probably check the count of events has reached expected count.

[asikowitz]

Ok, I made a change here, lmk what you think. I'm not exactly sure what the API is saying but this seems like what changes between when the api endpoint first opens and its final state. Would prefer to avoid the full 2 minute wait just to keep things as quick as possible, if possible.

[asikowitz]

I broke the commits down between (i) expanding MCPs and (ii) changes made by the updates. Everything remains the same except some properties change, and urn:li:dataJob:(urn:li:dataFlow:(nifi,80820b2f-017d-1000-85cf-05f56cde9185,PROD),c8c73d4c-ebdd-1bee-9b46-629672cd11a0) loses an inputDatajob (order also seems to change). That seems somewhat concerning to me and this difference persists even with the flat 120 second wait. @mayurinehate does this seem concerning to you?

[mayurinehate]

lgtm