APEXMALHAR-2517 - Shaded 3.7.0 release with legacy packages

[tweise]

The resulting artifact will contain the old com.datatorrent packages as of release 3.7 and will become a dependency of malhar-library with the upcoming release to provide these classes for backward compatibility (classes removed in #662).

To be merged after #662

[pramodin]

What would be the benefit of doing this? Won't the malhar version 3.7 that is on maven central already have the 3.7 com.datatorrent classes. Is it to simulatenously use the older classes alongside new versions? What I am missing is how the apps written against 3.7 will benefit from newer versions without code changes. I thought we were going to create an alternate shaded version of the operators relocated to com.datatorrent so existing apps would work with newer versions.

[tweise]

That question was already answered during the mailing list discussion: The operators from old packages are deprecated and frozen as of 3.7, they are no longer developed (same as if you put @deprecated on a class and never touch it again). The only reason and goal for doing this is backward compatibility.

To take advantage of fixes and new features, the user will switch to the new packages. This will be mentioned in the release notes.

[pramodin]

I don't think there was an agreement to this on the mailing list discussion from any of the folks involved in the discussion. I like the shading idea in general but would like to see it used to support the existing package names without things being frozen to a specific release or go the old fashioned approach of creating deprecated extensions. The old names can be dropped when we go to the next major version.

[sandeshh]

There was no consensus on the mailing list. Urgency of this big change was not explained. We do need to keep end user's convenience in mind. This change needs to be done in 4.0, not in 3.8.

[tweise]

@sandeshh I suggest you review the original email thread as that may address most of your concerns. Specifically I find it inappropriate to loop back on "urgency" when this is a two year old topic that was discussed many months before now finally action is taken. The other point that was already addressed is whether or not 4.0 is required to make this change. There are several options to satisfy backward compatibility (which this PR and related discussion actually is about.) I will appreciate your constructive input and suggestions that help move things forward.

[tweise]

@PramodSSImmaneni I think what you are asking for is not backward compatibility but forward compatibility :) However, both of the approaches you mention (extending new classes and offering new classes under old name) may not provide the same level of backward compatibility, as they would be intermingled and not isolated. I will look at it a bit more.

[sandeshh]

@tweise I am not looping back on urgency, I am saying this looking forward. We have already waited for 2 years, why not wait for few more months and do this change with 4.0? I am not against this change. What's the benefit to end users? To get simple bug fixes they have to change all the namespaces? rather than just changing the version number in POM.

FYI
I have also done similar(name) changes, like https://issues.apache.org/jira/browse/APEXCORE-475

[tweise]

The change is overdue like many others. There is the tendency to wait for the next great opportunity and that is why we are having this conversation after 2 years. When you look around you will find more examples where things are discussed and never taken up. Instead, you need to break things down in achievable units and tackle them on an ongoing basis. A major version change is not just a label on the package but signifies more than that. I don't see any reason to declare a new major version unless there is either real progress in operator maturity or other breaking changes are due.

IMO the discussion made clear that package cleanup can be done without major version change, now we need to decide which of the available options to pick for backward compatibility. To me it would be quite normal to change the import statements to pick up the change I'm interested in after upgrade (or do nothing if I don't care). If that's not agreeable then we have other options to move forward.

[amolhkekre]

Please see my comments on #662

[pramodin]

During the discussion, we talked about creating a release-3 branch. Looks like that was not done.

[tweise]

Just pushed: https://github.com/apache/apex-malhar/tree/release-3

[pramodin]

Sounds good