sync: Bulk push metadata-models changes

[jplaisted]

metadata-models 80.0.0 -> 90.0.13:

   90.0.13: Roll forward: Fix the open source build by avoiding URN method that isn't part of the open source URN.
    90.0.2: Refactor listUrnsFromIndex method
    90.0.0: Start distinguishing between [] aspects vs null aspects input param
    89.0.4: Fix the open source build by avoiding URN method that isn't part of the open source URN.
    89.0.2: fix some test case name
    89.0.0: META-12686: Made the MXE_v5 topics become strictly ACL'ed to avoid the wildcard write ACL as "MetadataXEvent.+"
    88.0.6: change DAO to take Storage Config as input
    88.0.3: Add a comment on lack of avro generation for MXEv5 + add MXEv5 to the pegasus validation task.
   87.0.15: META-12651: Integrate the metadata-models-ext with metadata-models
   87.0.13: add StorageConfig to Local DAO
    87.0.3: Treat empty aspect vs optional aspect same until all clients are migrated
    87.0.2: Treat empty aspect vs optional aspect differently
    87.0.1: META-12533: Skip processing unregistered aspect specific MAE.
    83.0.6: action method to return list of urns from strong consistent index
    83.0.4: Change input param type for batch backfill
    83.0.3: Implement batch backfill
    83.0.1: Implement support for OR filter in browse query
   82.0.10: Throw UnsupportedOperationException for unsupported condition types in search filter
    82.0.6: Implement local secondary backfilling index as part of backfill method
    82.0.5: [strongly consistent index] implement getUrns method
    82.0.4: Add indexing urn fields to the local secondary index
    82.0.0: Render Delta fiels in the MCE_v5.
    81.0.1: Add pegasus to avro conversion for FMCE
    80.0.4: add get all support for BaseSingleAspectEntitySimpleKeyResource
    80.0.2: Add a BaseSearchWriterDAO with an ESBulkWriterDAO implementation.
    80.0.1: META-12254: Produce aspect specific MAE with always emit option
    80.0.0: Convert getNodesInTraversedPath to getSubgraph to return complete view of the subgraph (nodes+edges)

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable)

[mars-lan]

Tidy up the changelog?

[jplaisted]

Tidy up the changelog?

To? Do we have guidelines?

[mars-lan]

We should create those guidelines. Suffice to say that "empty commits" and internal version numbers are not very useful for the OSS community.

[keremsahin1]

We should create those guidelines. Suffice to say that "empty commits" and internal version numbers are not very useful for the OSS community.

I guess version numbers should be OK to keep because we already state this at the top metadata-models 80.0.0 -> 90.0.13. I would suggest to get rid of these lines as a guideline:

• Commits which only makes sense internally like 83.0.0
• Empty commits for trigger
• Auto-reverts (+ the duplicate commits after these auto-reverts)

Also make sure to amend the commit log as well not just the PR log :) Don't forget to have TAG lines at the end MP_VERSION=metadata-models:{MM_LAST_COMMIT_TAG}. You should have that if you auto-generate the commit.

[keremsahin1]

I reviewed the changes. Other than the commit log, all seems good to me. Now that we log time in JIRA, you might want to consider to log how much you spend for this in the ticket :)

[mars-lan]

I feel internal version number is kind of a distraction, especially if there's gaps due to empty or internal-only commits. What OSS community really wants to know is the highlight of changes, rather than nitty gritty details IMO.

[jplaisted]

Other than the commit log, all seems good to me. Now that we log time in JIRA, you might want to consider to log how much you spend for this in the ticket :)

At this point, I don't remember. The answer is too long :p

I feel internal version number is kind of a distraction, especially if there's gaps due to empty or internal-only commits. What OSS community really wants to know is the highlight of changes, rather than nitty gritty details IMO.

I think it is still important for our tracking purposes. If we want to know what we last pushed, just look through commit history. Edit so maybe we don't need to label every internal version, but just the start -> end.

[jplaisted]

Commits which only makes sense internally like 83.0.0

imo this is too much onerous on the engineer doing the push to analyze the log.

We could start having internal commits include a RELEASE_NOTES tag or something, which are then parsed and used to build this commit message. And/or actual release notes.

Or something else? Open to suggestions. But I don't think we should have one person sit here and crawl through every commit to determine what should be labeled as important :)

[keremsahin1]

Or something else? Open to suggestions. But I don't think we should have one person sit here and crawl through every commit to determine what should be labeled as important :)

Sure. If you're up for extending the plugin for more automation, I have a better idea which can take all the onus from the shoulders of OS czar :)

You can check the diff in every commit/version and see if any OSable (meaning it exists both internally/OS) source is being updated. If it's not, you can safely discard the log of that commit.

[mars-lan]

imo this is too much onerous on the engineer doing the push to analyze the log.

I don't it's that bad to comb through 86 lines. Hopefully we don't have to do this too often in the future once GMA core becomes its own project & OSS-first.

[jplaisted]

I don't it's that bad to comb through 86 lines.

Assume I spend 30 seconds per line (I don't have context for everything!). That's 45 minutes.

[jplaisted]

Maybe to help short circuit this, we can do better in the future? But for now let's just catch up?

[mars-lan]

Let's at least clean up the empty commits and internal model changes.

[jplaisted]

Already done - at least the ones to trigger a build. What actually ended up being empty in practice I am not sure. I'd have to go through every version we have, apply it to OS, check the diff, and repeat.

Edit: I'm more than happy to have the tooling do that in the future.

[mars-lan]

Did some initial clean up. @keremsahin1 wanna give it a crack too?

[keremsahin1]

Did some initial clean up. @keremsahin1 wanna give it a crack too?

I just dropped 1 line which doesn't apply to OS.

       87.0.12: Update metadata-models-plugin and ACLs.

[mars-lan]

Let's sync the git commit message with the PR description. Thanks.

[jplaisted]

Let's sync the git commit message with the PR description. Thanks.

done