Split Framework

[gianarb]

@Ocramius and I talked about the future of this project and the split of framework is an idea to manage in the best way single component. In my opinion these are the hardest structural problems:

• hard dependencies
• difficult to decouple single component
  Considering that we think that the possibility to split the framework could help us to manage this criticality.

Goal: Split framework and its history in diferent indipendent repositories (component + your tests).

Positive

• Different maintainers for different components..
• ZF2 project is a simple list of dependencies and every components follow its time
• ...

Negative

• Difficulties in mantainance and orchestration
• History consistency
• ...

I'm working for this split but the way is long and I'm not a git ninja :) This is my first tests but there are more problems

• Merge is not a good flow.. The history is incomplete.
• Rebase is not a good choose because the history is useless.
• Subsplit works with only one prefix path an this is not a good solution
• Maybe this script too easy.. What is your opinion about this proposal and is there a git ninja?! :)

[Martin-P]

The idea itself sounds a lot like the Symfony approach.

Different maintainers for different components..

Isn't that already the case? For certain issues I see specific maintainers.

ZF2 project is a simple list of dependencies and every components follow its time

If the goal is to shorten the time between releases, that should already be possible. It is a choice not to release and wait for some issues to be fixed. I think you will need a release plan with specific release dates (see #6957). If issues do not get fixed in time, the issue can be marked for the next milestone. That way you keep things moving and I think that is the main problem with ZF2. IMO splitting up the framework won't fix that.

[gianarb]

Symfony split component in read only, the idea is only one and persistent split after it every component lives own life..

Yes diffrent projects hav different maintainers but the framework have too many components and this split is a good opportunities to make clean.

The split not resolve the staticity problem, in first because I don't know reasons of it :) But maybe simplify the structure could be help us..

[EMCP]

"Yes different projects have different maintainers but the framework have too many components and this split is a good opportunity to make clean."

Isn't a split going to further cripple people from sharing code across ZF2? Apigility is a perfect example of a project that cannot take advantage of every-all ZF2 modules, due to it's decision to utilize certain modules that block very popular modules.. ie zfc-user. Now when I search for modules, if they depend on zfc-user I must automatically strike them from the list of possibilities.

[Ocramius]

IMO splitting up the framework won't fix that.

This will actually fix that, as we will be able to develop the single components in insulation, allowing more research work to happen without the burden of framework BC lock.

Also, we will be able to transfer ownership of the single packages to the various maintainers, allowing them to tag/release and merge independently from the framework decisions.

cannot take advantage of every-all ZF2 modules, due to it's decision to utilize certain modules that block very popular modules..

There's a radical difference between a module and a library, as a library won't do anything when "plugged in", while a module actually applies runtime operations.

[Martin-P]

This will actually fix that, as we will be able to develop the single components in insulation, allowing more research work to happen without the burden of framework BC lock.

Is a BC lock the problem? If a class method is removed/altered it is a BC break. I don't see how splitting up the framework in different components changes that. If I look at the issues labeled as BC break, there are very few real issues. Most of them are ZF3 suggestions and some other minor suggestions.

Also, we will be able to transfer ownership of the single packages to the various maintainers, allowing them to tag/release and merge independently from the framework decisions.

What if a components has very few releases? I understand that specific component is the only component affected, but if your application needs the PR but there is no release you still got the same problem as nowadays.

I can imagine splitting up of the framework can make things easier, but does splitting up the framework actually solve the issues like BC break and releases? And are those issues really the problem?

[Ocramius]

I don't see how splitting up the framework in different components changes that.

The point is allowing those components to jump forward, making them 3.x, 4.x and so on. We'd just lock some of the versions and make them "LTS"

if your application needs the PR but there is no release you still got the same problem as nowadays.

This is a problem that currently exists and that will always exist, but at least we'd be able to fetch dev- versions of atomic dependencies rather than risking an upgrade of dev-master of the entire thing.

but does splitting up the framework actually solve the issues like BC break and releases?

Yes, we discussed this carefully with a few other community members at zendcon and the only issue we found is the need for someone to coordinate the process and supervise it continuously.

[bakura10]

+1 on this, this should have been done long time ago. As said, the idea is that zf2 becomes a simple composer.json file, with only 2.* dependencies (unfortunately, because of the package name being zf2, I think we should make sure that zf2 actually never upgrades any of its dependencies to 3.x). Otherwise, people will be encouraged to set their own dependencies list, with only the components they need.

As I've pointed it, the main issue is this is that we will likely face issues because dependency graph of current components is stupidly complex, and we should refactor each component one by one to move code, remove dependencies... for this to make sense.

Also, we have planned long ago to move some components into their own package (hydrator, router...). How will this been handled?

[Ocramius]

Also, we have planned long ago to move some components into their own package (hydrator, router...). How will this been handled?

@bakura10 that would be done after the first split, as an additional split that is up to the component maintainer.

[gianarb]

zf2 is only a repository name.. :) We can rename it in zf after split. Now is necessary a good script to manage this split..

[Martin-P]

and the only issue we found is the need for someone to coordinate the process and supervise it continuously

I am glad you mention that, because IMO that is the main issue. It is nice to split up the framework, but isn't that just a side issue here? Without someone to coordinate the process and supervise it, it won't work. BTW: who is doing this for ZF2 right now?

[Ocramius]

who is doing this for ZF2 right now?

@weierophinney is in charge. The point is that every component maintainer (assuming he's aware of semver) would have a "main" or "LTS" version to maintain for compatibility with the framework, whereas master or subsequent versions could freely upgrade/break BC.

It just assumes that the maintainers know semver very well, and the coordinator would just check breakages before upgrading the composer.json in the main framework.

[akrabat]

Related to what @Ocramius has said, I think that we'll have a "release master" for the main ZF2 repository that will liaise with the component maintainers to ensure that each component's LTS branch is maintained in a BC manner.

[bacinsky]

This will be a huge improvement to stability, because if something is f_ck_d up somewhere in one component e.g. BC break, whole framework version is broken.

[macnibblet]

My only concern is the complexity of tracking the changes between component versions and keep the documentation up to date!

[danizord]

I think it would be great if we could subsplit documentation as well (not sure how), so that each component will have its own docs and code in same repository (markdown plx).

[Ocramius]

subsplit documentation as well

I think this would make sense only once a component moves to its own internal release cycles (for example zendframework/zend-servicemanager:3.0.0|4.0.0). Moving individual docs can probably be done without an actual subsplit, as there is no real advantage in keeping track of changes in older versions (current).

markdown plx

Not gonna happen: RST takes little effort to edit, and gives us a lot more flexibility/templating ;-)

[gianarb]

I created a zf2-prototype repo for split tests..
In this moment I'm working to write a little script to split framework with --filter-branch strategy..
is this a good method? I don't know :)
If anyone have suggestions or ideas write me please 👷

[pauloelr]

@Ocramius, I and @danizord talked with @weierophinney at PHP Conference Brasil and he said he is in favor of keep the docs and code on the same repository after the split.

This way i think the maintainner could try to only merge PR with tests and docs representing the changes on code, this way all the things stay up-to-date, and is easy to keep track of changes.

[gianarb]

I'm trying this script, in my prototype ZF2 repo work, there is only one problem for the root commit, it is unchanged.. This script run from seven hours in the official repository 🎱 . Maybe there are different strategies :)

[jaapio]

I think you already know but this project https://github.com/zendframework/subsplit-ng was used by weierophinney to create the first tags.
I found some comment where he wrote that because of the size of this repo --filter-branch was a bad idea.

[Ocramius]

@jaapio the problem just affected him because he has to re-run the operations at every release. Once 2.5 is out, we won't need to do that again, as every component will be tagged independently and the subtree split will be ancient history

[gianarb]

@jaapio I don't know if this script manage split of ./tests directory

[jaapio]

No it won't, it will only split the library. But I modified it for personal use. so I could give it a try

[gianarb]

This is the first split there are more points of discussion for example:

• the tree of directories is correct?
• README and CONTRIBUTING are the original of parent repository
• phpunit.xml.dist is in /tests or not?
• Move composer.json to root directory and fix missing node (autoload in first)
• ...

[Ocramius]

If a couple of folks could help @gianarb checking the history of his split (seeing if it reflects actual framework history for both sources and tests) that would speed up things a lot, thanks!

[bakura10]

Hi,

I have checked quickly, it looks good. As I said by email, I think README.md, CONTRIBUTING.md and composer.json should be moved to the top folder, as well as phpunit.xml.dist.

The structure should be flattened too, by renaming library\Zend\Stdlib to src\Zend\Stdlib (or src\ if we use PSR-4).

Regarding the history, when comparing the tests folder, I have commits in the split that are not present in the tests folder of zf2 repo. For instance, there are commits here on 1st August, while they do not appear there.

[GeeH]

Can you repeat with more clarity what the intended goal of this split is please?

To clarify, you say how, and you gloss on positive/negatives, but I'd like to know more about the why.

[GeeH]

I'm not against this by the way, I'm very much in favour, I just believe that PPPPPP.

[gianarb]

@bakura10 thanks for your history check! 👍 I have not removed files into tests/* and the commits refer to this files.. I don't know if this stuff is correct or not 😕 But I think so because is part of history..

Script Update

See it

• L55 - List of root files won't delete
• L82 - it move composer.json in root and update it with required-dev and autoload (PSR-0 is correct? 😴 )

[jaapio]

The files that you exclude are mostly created for zf2 as complete framework. I think after the split we should create the new buildscripts. Some of there files could be extracted from the zf2 repository. But others need to be written.

Moving the composer.json makes sense since it will have some history. And contains the actual requirements for each component.

[gianarb]

I agree with you @jaapio I move composer.json and not exlude .travis .cs... 👍

[Pittiplatsch]

Ocramius wrote:

The point is that every component maintainer (assuming he's aware of semver) would have a "main" or "LTS" version to maintain for compatibility with the framework, whereas master or subsequent versions could freely upgrade/break BC.

It just assumes that the maintainers know semver very well, and the coordinator would just check breakages before upgrading the composer.json in the main framework.

What I don't see is:
On using LTS versions, when a new LTS of a central component (say ServiceManager 5.1 after previous LTS ServiceManager 4.2) is released, each depending component must be updated to use SM5.1 before the complete framework can be pushed up to SM5.1.
This leads to "badly" maintained components which aren't updated soon block the complete framework, as we cannot depend on two different versions of a single component.
On the other hand, this fact could turn out as advantage when there is an agreement that such "mucky pups" are locked until they support the new SM5.1

I think this needs considerable organisation and communication like:

ZF3.5 is released on 15th July 2015 with the following dependencies:

• SM5.1
• other components current LTS versions

Each component must be updated by that date to fulfill these dependencies; any components failing to fulfill these dependencies will be removed from ZF3.5, but may come back later as soon as they have upgraded theire dependencies.

However, we are chasing our own tail here, when (mentioned dependencies are exemplary!) component Locale must be updated to SM5.1, however compenents Form, Text, Paginator, and Navigation for their parts depend on Locale...
Should become quite difficult to get LTS versions for each component synchronously...

[Ocramius]

Should become quite difficult to get LTS versions for each component synchronously...

An LTS would only bump minor/patch versions of dependencies, so anything should still be compatible.

Major versions will be free to bump wildly as long as the minor versions used in LTS releases are still maintained.

[Pittiplatsch]

An LTS would only bump minor/patch versions of dependencies, so anything should still be compatible.

That's what I first thought as well - no problem here, I thought.
As I stated (a little bit lenghty), the problem comes in with components being dependent from each other, where different components rely on some common component, but in different LTS versions.

[Ocramius]

but in different LTS versions

That is purely a component maintenance issue, not really a problem for the LTS "main" package: since components will be handled separately, then the feedback loop in case of incompatibilities will be much shorter, allowing for making stricter or more relaxed dependency constraints.

[Pittiplatsch]

It's an organisational problem of the "main" package to have components keeping their dependencies synchronously, as it's simply not possible to release ZF3.7 with SM5.1 if any of ZF's "functional" components doesn't have its dependencies updated to SM5.1 but still relies on SM4.2. This one "trailing" component thus prevents "main" package's complete process. Am I wrong here?

Disclaimer: all component names and version numbers are fictive

[Ocramius]

Am I wrong here?

No, you are correct, and that's not a problem.

The main point is to stop focusing this much on the main package: main package is an LTS-ish, and therefore people shouldn't bother having the latest and greatest of everything, just a "stable" version is fine ;-)

[fabiocarneiro]

Hey guys, git has a subtree comand you can set all commits to a specific branch a pull the created branch from another repo. Is this script really necessary?

http://stackoverflow.com/questions/359424/detach-subdirectory-into-separate-git-repository/17864475#17864475

[gianarb]

Hi! The problem of this solution is a tests dir, git subtree supporto only
directory :(

2015-02-04 18:40 GMT+01:00 Fábio Carneiro notifications@github.com:

Hey guys, git has a subtree comand you can set all commits to a specific
branch a pull the created branch from another repo. Is this script really
necessary?

http://stackoverflow.com/questions/359424/detach-subdirectory-into-separate-git-repository/17864475#17864475

—
Reply to this email directly or view it on GitHub
#7072 (comment).

Gianluca Arbezzano
www.gianarb.it

[fabiocarneiro]

@gianarb https://github.com/fabiocarneiro/zend-form

subtree with source to a branch
pull this branch from a new repo
move files to src folder, commit
subtree with tests to another branch
pull this another branch from the same new repo
move tests to test folder, commit
add other settings

Sorry do insist, but am I missing something? If you open any file you'll see it has no history, but that can be resolved with filter-branch rewriting all the history as the files were created under src folder (or whatever)

[gianarb]

I have tried this solution but I have spoke of it with @Ocramius, maybe the merge between two different branch, code_branch and test_branch is not a good solution
for the history..

@fabiocarneiro no problem for me :D Your solution is more easier and faster :) If it resolve the problem perfect!!!

[fabiocarneiro]

@gianarb https://github.com/fabiocarneiro/zend-form

I updated it with the filter-branch. It also took a lot of time to move, but i'm not sure for how long it was running since i was sleeping. Zend\Form is one if not the one with most commits, so maybe the other ones take less time.

The subtree approach might be helpful since it can split all commits that modify that files instead of dealing with the full zf2 history, then when running the filter-branch we have less commits to rewrite.

The only downside i could notice, is that there will be a merge commit of the test branch, but i'm not sure that is that bad.

Maybe i can put all together and create a sh script but i would like some review if that history is ok
zf2-split.sh component-src-dir component-test-dir dest
(then you need only to create the remote, add it, and push)

[gianarb]

@fabiocarneiro have you see this script? https://gist.github.com/gianarb/da3291c8671529cca691#file-console-php
👍 thanks for your works

Speed is a problem :) in my old macbook pro are ~10h for component

[fabiocarneiro]

@gianarb yes i saw it. To be honest i didn't understand what you're exactly doing there (how you move the code to a new repo part). The approach is different anyway, you are erasing files from the history and i'm moving them. If my history is ok, maybe we could merge both works to make it better?

[gianarb]

Oh yes, this is the objective! 👍 I know by @Ocramius that @EvanDotPro and other CR contributors are seeing this works :) We can only wait theirs opinion

[weierophinney]

@gianarb regarding this:

Speed is a problem :) in my old macbook pro are ~10h for component

now you see the motivation for subsplit-ng. :) This stuff is immensely time-consuming due to the imperative to keep history (which, for some components, goes back almost 10 years now!).

I'll be looking at your gist, and a few others that others have been pointing out to me in the past few weeks since the ZF3 announcement. I am under no illusions that this will be an easy or quick task; however, once done, we'll have a far easier story going forward with maintaining the individual components.

[gianarb]

@weierophinney the time is frightening 👻 but this split work only one time this stuff resize the problem :) But every others best proposal is good! 👍
I'm here for every tests or upgrades, for example I don't know if the history result is good or not :)

[carnage]

Speed is not really an issue. The split only has to happen once. There will be ways to improve speed anyway. eg: http://willi.am/blog/2014/10/14/unpacking-a-git-database/ and putting the repo on a ram disk to speed up I/O**

** Assuming that the speed of the split is I/O bound.

[gianarb]

@carnage yes thanks.. This night I'm trying to split a component in digitalocean, this is another life :P

[jaapio]

Not sure if this is still applicable. And some other git guru's could probably the same job. But I found this scrip https://github.com/simpliwp/git-splits/blob/master/git-splits which does its job quite good.

[gianarb]

Hi @jaapio impossibile?! None!
We are using this script to split framework here

[jaapio]

What I meant was that I wasn't sure if you were still looking for a solution. I see there is one, so my comment was not applicable. :-)

[weierophinney]

Accomplished with #7542