New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some questions about the future of the Doctrine Project #6211

Closed
javiereguiluz opened this Issue Jan 3, 2017 · 20 comments

Comments

Projects
None yet
@javiereguiluz
Contributor

javiereguiluz commented Jan 3, 2017

Hi Doctrine Project managers (@Ocramius, @beberlei and @guilhermeblanco),

This morning, in the Symfony Slack chat, I posted the following message:

I'd like to ask your opinion about the health of the Doctrine Project. It's a critical piece for Symfony and I'm concerned about it: the number of contributors looks alarmingly low, the development of new features is not very impressive lately and it accumulates more than 1,000 open issues. Thanks!

I won't copy the entire discussion because there were a lot of messages, but at the end someone suggested to ask you directly about the future of the Doctrine Project. So here are some questions:

  1. Are you planning to keep working on 2.x or are you putting that version in low maintenance mode and work only on 3.x?
  2. Which is the roadmap for the 3.x version? We have the ORM.NEXT GitHub Project and the 3.0 GitHub milestone.
  3. Is there an estimated date to publish the 3.0 version?
  4. Would you like to have more contributors to the project or do you prefer to work alone on the 3.x development? If you want more contributors, how can we help you?
  5. Is the +1,000 open issues figure realistic, or are there a lot of issue that could be closed right away? How could we help you with that?

Thanks for your time!

@Ocramius

This comment has been minimized.

Show comment
Hide comment
@Ocramius

Ocramius Jan 3, 2017

Member

Are you planning to keep working on 2.x or are you putting that version in low maintenance mode and work only on 3.x?

We're planning to release 2.6 and call it the last 2.x version, freezing any request for improvements.
ORM 3.x is being developed by @guilhermeblanco when he has free time.

Which is the roadmap for the 3.x version? We have the ORM.NEXT GitHub Project and the 3.0 GitHub milestone.

No roadmap: the roadmap is "we do what we can, when we can", sorry.
We're mostly thinking of:

  • dropping features
  • performance optimizations (especially considering PHP's new engine)
  • internal cleanups to reduce complexity

Most public API will stay as-is until we got to a decent cleanup state and can take decisions about the future of the interfaces.

Is there an estimated date to publish 3.0 version?

No - at this rate it will be likely no earlier than 2018, unless we just do a release due to a forced BC break/security issue leading to BC break.
We cannot commit to any schedule, since we're not a company and we do not profit from the tool.

Would you like to have more contributors to the project or do you prefer to work alone on the 3.x development? If you want more contributors, how can we help you?

I think the amount of prototyping work done by @guilhermeblanco doesn't play well with coordination work. Yes, he is going "cowboy-coder" here, but I really do trust him on most design decisions behind the scenes, as he's been very meticulous and informed with his work.

We also don't have the time to coordinate work, and the domain knowledge behind the ORM internals is huge and sadly only written down in the Java world.
Assuming we had the willpower for it (not the case at the end of a work day), nobody in the team would really be able to follow somebody's work closely, as the amount of issues being spawned already completely clutters our ability to work on new stuff in our free time.

Is the +1,000 open issues figure realistic, or are there a lot of issue that could be closed right away?

We had more than 1000 open issues for ages - the fact that we moved them to Github a bit over a year ago simply exposes the number to the "general public". There are no critical issues or known vulnerabilities at the moment, so there may be a big purge after the API changes (3.x).

How could we help you with that?

We could need somebody to go into open PRs and find good reasoning for rejecting them. Yes: I do intend reject there, as most issues on the ORM starting from version 2.4.0 are edge-case scenarios, mapping weirdness, exotic relational databases and structures.

The ORM is an extremely stable tool, so I wouldn't worry too much about the amount of issues, rather than the quality/relevance. Also, we are extremely alert on security issues, which we also didn't have in a while.

If somebody wants issue rights and go in and do a close-spree (with careful explanations and cross-referencing), then I'd gladly grant those.

Other than that, I can suggest sponsoring (actual 💰) @guilhermeblanco (if he's interested - this is my idea, not his) for working full-time on the ORM, and that's a quite substantial investment.

Member

Ocramius commented Jan 3, 2017

Are you planning to keep working on 2.x or are you putting that version in low maintenance mode and work only on 3.x?

We're planning to release 2.6 and call it the last 2.x version, freezing any request for improvements.
ORM 3.x is being developed by @guilhermeblanco when he has free time.

Which is the roadmap for the 3.x version? We have the ORM.NEXT GitHub Project and the 3.0 GitHub milestone.

No roadmap: the roadmap is "we do what we can, when we can", sorry.
We're mostly thinking of:

  • dropping features
  • performance optimizations (especially considering PHP's new engine)
  • internal cleanups to reduce complexity

Most public API will stay as-is until we got to a decent cleanup state and can take decisions about the future of the interfaces.

Is there an estimated date to publish 3.0 version?

No - at this rate it will be likely no earlier than 2018, unless we just do a release due to a forced BC break/security issue leading to BC break.
We cannot commit to any schedule, since we're not a company and we do not profit from the tool.

Would you like to have more contributors to the project or do you prefer to work alone on the 3.x development? If you want more contributors, how can we help you?

I think the amount of prototyping work done by @guilhermeblanco doesn't play well with coordination work. Yes, he is going "cowboy-coder" here, but I really do trust him on most design decisions behind the scenes, as he's been very meticulous and informed with his work.

We also don't have the time to coordinate work, and the domain knowledge behind the ORM internals is huge and sadly only written down in the Java world.
Assuming we had the willpower for it (not the case at the end of a work day), nobody in the team would really be able to follow somebody's work closely, as the amount of issues being spawned already completely clutters our ability to work on new stuff in our free time.

Is the +1,000 open issues figure realistic, or are there a lot of issue that could be closed right away?

We had more than 1000 open issues for ages - the fact that we moved them to Github a bit over a year ago simply exposes the number to the "general public". There are no critical issues or known vulnerabilities at the moment, so there may be a big purge after the API changes (3.x).

How could we help you with that?

We could need somebody to go into open PRs and find good reasoning for rejecting them. Yes: I do intend reject there, as most issues on the ORM starting from version 2.4.0 are edge-case scenarios, mapping weirdness, exotic relational databases and structures.

The ORM is an extremely stable tool, so I wouldn't worry too much about the amount of issues, rather than the quality/relevance. Also, we are extremely alert on security issues, which we also didn't have in a while.

If somebody wants issue rights and go in and do a close-spree (with careful explanations and cross-referencing), then I'd gladly grant those.

Other than that, I can suggest sponsoring (actual 💰) @guilhermeblanco (if he's interested - this is my idea, not his) for working full-time on the ORM, and that's a quite substantial investment.

@Ocramius Ocramius added the Question label Jan 3, 2017

@javiereguiluz

This comment has been minimized.

Show comment
Hide comment
@javiereguiluz

javiereguiluz Jan 3, 2017

Contributor

Thanks for your detailed answer! The status of the Doctrine Project is more clear now 👍

Contributor

javiereguiluz commented Jan 3, 2017

Thanks for your detailed answer! The status of the Doctrine Project is more clear now 👍

@lcobucci

This comment has been minimized.

Show comment
Hide comment
@lcobucci

lcobucci Jan 3, 2017

Member

@Ocramius do you think we could/should make this status a bit more visible?

Member

lcobucci commented Jan 3, 2017

@Ocramius do you think we could/should make this status a bit more visible?

@Ocramius

This comment has been minimized.

Show comment
Hide comment
@Ocramius

Ocramius Jan 3, 2017

Member

@lcobucci just do it if you feel like it 👍

Member

Ocramius commented Jan 3, 2017

@lcobucci just do it if you feel like it 👍

@lcobucci

This comment has been minimized.

Show comment
Hide comment
@lcobucci

lcobucci Jan 3, 2017

Member

@Ocramius ok, I will think on a nice way and propose it 😉

Member

lcobucci commented Jan 3, 2017

@Ocramius ok, I will think on a nice way and propose it 😉

@guilhermeblanco

This comment has been minimized.

Show comment
Hide comment
@guilhermeblanco

guilhermeblanco Jan 3, 2017

Member

Hi @javiereguiluz!

I'll answer the questions from my own POV. I think it's ok if we answer individually on our own perspective and then take this discussion offline and come up with a project response.

  1. Are you planning to keep working on 2.x or are you putting that version in low maintenance mode and work only on 3.x?

We have enough fixes/features on master branch that we could release 2.6 today. AFAIR, there was one outstanding fix needed for SLC to be reviewed and merged (which by now it already is), so there're no blockers to release 2.6.

I'm focused on working purely on 3.0. However, I couldn't touch it since October due to a variety of personal reasons (I switched jobs, onboarded a huge project on initial phase which required some extra dedication and now took a month of well deserved vacation) and will resume my development once I get back home.

  1. Which is the roadmap for the 3.x version? We have the ORM.NEXT GitHub Project and the 3.0 GitHub milestone.

We share a Google Doc describing our wish list and that is what we're doing on ORM.NEXT Project. Not everything is listed there, since as we evolve, we might change plans. There're thousands of lines of code to be written and thousands to be dropped.
I started a new branch called develop. You should be able to use it today (with some BC breaks already highlighted) if you want. I've lost count of how many unknown bugs (and edge cases) I've fixed so far, but you can count 100+.

As it currently stands, develop has embeddable support removed (actually commented out) since most of it was done in a dirty hooked way. Properly supporting it (and addressing most issues), would require a full re-development of the feature, so I removed its support for now.

I have yet to review 3.0 issues and close/answer them properly and annotate with similar elements on ORM.NEXT project. Should do that any day tbh.

  1. Is there an estimated date to publish the 3.0 version?

No, but I was committed to dedicate 2-3h daily on this project to work on 3.0.
Keep in mind Doctrine 2 took an year of planning and 2 years of development. We don't need to redo all the work, but I expect it to take at least full 2017 of development before it hits early releases of 3.0.

  1. Would you like to have more contributors to the project or do you prefer to work alone on the 3.x development? If you want more contributors, how can we help you?

There're a couple blocking changes required before we unleash the ability to split work with others.
Right now I'm in the middle of a massive change (transforming ClassMetadata mapping from associative arrays to OO). I've done most of the work already, but any other person that decides to jump in now would have to rewrite a lot of code until I finish this epic.

The change to ClassMetadata will unleash changes to be made everywhere. One good example is the ability to compile metadata PHP classes that can take advantage of opcache and remove the need for metadata cache forever. As I work on OOing Metadata, I'm also ensuring that Export to PHP is adequate, so consuming it later would be a joy.

  1. Is the +1,000 open issues figure realistic, or are there a lot of issue that could be closed right away? How could we help you with that?

Most of the issues are actually user issues, not bugs on ORM. As we evolve, we add new ways of implementing similar features, which lead people to use them in funky ways.
Our main wish is to reduce the amount of weirdness people can do while using Doctrine, which should save us substantial time and effort (well, and save us from some grey hair).

Member

guilhermeblanco commented Jan 3, 2017

Hi @javiereguiluz!

I'll answer the questions from my own POV. I think it's ok if we answer individually on our own perspective and then take this discussion offline and come up with a project response.

  1. Are you planning to keep working on 2.x or are you putting that version in low maintenance mode and work only on 3.x?

We have enough fixes/features on master branch that we could release 2.6 today. AFAIR, there was one outstanding fix needed for SLC to be reviewed and merged (which by now it already is), so there're no blockers to release 2.6.

I'm focused on working purely on 3.0. However, I couldn't touch it since October due to a variety of personal reasons (I switched jobs, onboarded a huge project on initial phase which required some extra dedication and now took a month of well deserved vacation) and will resume my development once I get back home.

  1. Which is the roadmap for the 3.x version? We have the ORM.NEXT GitHub Project and the 3.0 GitHub milestone.

We share a Google Doc describing our wish list and that is what we're doing on ORM.NEXT Project. Not everything is listed there, since as we evolve, we might change plans. There're thousands of lines of code to be written and thousands to be dropped.
I started a new branch called develop. You should be able to use it today (with some BC breaks already highlighted) if you want. I've lost count of how many unknown bugs (and edge cases) I've fixed so far, but you can count 100+.

As it currently stands, develop has embeddable support removed (actually commented out) since most of it was done in a dirty hooked way. Properly supporting it (and addressing most issues), would require a full re-development of the feature, so I removed its support for now.

I have yet to review 3.0 issues and close/answer them properly and annotate with similar elements on ORM.NEXT project. Should do that any day tbh.

  1. Is there an estimated date to publish the 3.0 version?

No, but I was committed to dedicate 2-3h daily on this project to work on 3.0.
Keep in mind Doctrine 2 took an year of planning and 2 years of development. We don't need to redo all the work, but I expect it to take at least full 2017 of development before it hits early releases of 3.0.

  1. Would you like to have more contributors to the project or do you prefer to work alone on the 3.x development? If you want more contributors, how can we help you?

There're a couple blocking changes required before we unleash the ability to split work with others.
Right now I'm in the middle of a massive change (transforming ClassMetadata mapping from associative arrays to OO). I've done most of the work already, but any other person that decides to jump in now would have to rewrite a lot of code until I finish this epic.

The change to ClassMetadata will unleash changes to be made everywhere. One good example is the ability to compile metadata PHP classes that can take advantage of opcache and remove the need for metadata cache forever. As I work on OOing Metadata, I'm also ensuring that Export to PHP is adequate, so consuming it later would be a joy.

  1. Is the +1,000 open issues figure realistic, or are there a lot of issue that could be closed right away? How could we help you with that?

Most of the issues are actually user issues, not bugs on ORM. As we evolve, we add new ways of implementing similar features, which lead people to use them in funky ways.
Our main wish is to reduce the amount of weirdness people can do while using Doctrine, which should save us substantial time and effort (well, and save us from some grey hair).

@Ma27

This comment has been minimized.

Show comment
Hide comment
@Ma27

Ma27 Jan 3, 2017

Contributor

as I already mentioned in the Slack discussion, I try to comment on some issues in order to help, but my time is quite limited right ATM.
I hope that I'll have a little bit more time this year, so I'll try to spend 1-2h in a week by looking through the issue tracker and checking if I can do anything.

Contributor

Ma27 commented Jan 3, 2017

as I already mentioned in the Slack discussion, I try to comment on some issues in order to help, but my time is quite limited right ATM.
I hope that I'll have a little bit more time this year, so I'll try to spend 1-2h in a week by looking through the issue tracker and checking if I can do anything.

@Jean85

This comment has been minimized.

Show comment
Hide comment
@Jean85

Jean85 Jan 3, 2017

Contributor

I will do the same! Happy to help such a great project.

Contributor

Jean85 commented Jan 3, 2017

I will do the same! Happy to help such a great project.

@Ocramius

This comment has been minimized.

Show comment
Hide comment
@Ocramius

Ocramius Jan 4, 2017

Member

Closing here - answers were given, I suppose :-)

Member

Ocramius commented Jan 4, 2017

Closing here - answers were given, I suppose :-)

@Ocramius Ocramius closed this Jan 4, 2017

@Ocramius Ocramius self-assigned this Jan 4, 2017

@guilhermeblanco

This comment has been minimized.

Show comment
Hide comment
@guilhermeblanco

guilhermeblanco Apr 3, 2017

Member

Looks like it's well deserved some updates around this.

I've been dedicating 15-20h/weekly on develop branch since February. Right now almost the entirety of ClassMetadata has being moved to object orientation. Our performance tests highlighted an improvement of close to 10% and memory optimization close to 20%.

Right now I'm in the process of re-building the ClassMetadataFactory and also the way we load, cache and build ClassMetadata instances. It's expected that we have a significant performance improvement over this, since we'll no longer have to validate and complete arrays. It'll also be opcached.

As soon as this work is finalized, we'll be able to unlock parallel efforts of optimizations and improvements. One of the big challenges I have today is to break down Embeddables and Inheritance across the board. I have a detailed plan on how to build them, but the amount of interdependencies are preventing me to start this work. Working on ClassMetadataFactory will help reducing most of them.

I've been discussing extensively of having improved Metadata implementors as a way to strategically optimize code that is duplicated across the board on UnitOfWork, Persisters and SqlWalker. This should have invisible changes to end-user, but will unlock the possibility of specifying how the eager fetching could happen (either through join or through follow-up select).

Right now UnitOfWork is also doing a lot when comparing changesets. We've discussed internally how we could make it OO, and also delegate the change detection to Metadata implementors. This is also a couple months ahead of the game.

We made a decision of trying to ditch Query\Parser and leveraging Hoa\Compiler. Anyone should be able to work on this today, but it's non-trivial and required deep understanding of how we built our current parser and also some SQL standards.

I'll attempt to keep this issue with some background information, but ORM.NEXT project is the way to get a more accurate status of the project.

Member

guilhermeblanco commented Apr 3, 2017

Looks like it's well deserved some updates around this.

I've been dedicating 15-20h/weekly on develop branch since February. Right now almost the entirety of ClassMetadata has being moved to object orientation. Our performance tests highlighted an improvement of close to 10% and memory optimization close to 20%.

Right now I'm in the process of re-building the ClassMetadataFactory and also the way we load, cache and build ClassMetadata instances. It's expected that we have a significant performance improvement over this, since we'll no longer have to validate and complete arrays. It'll also be opcached.

As soon as this work is finalized, we'll be able to unlock parallel efforts of optimizations and improvements. One of the big challenges I have today is to break down Embeddables and Inheritance across the board. I have a detailed plan on how to build them, but the amount of interdependencies are preventing me to start this work. Working on ClassMetadataFactory will help reducing most of them.

I've been discussing extensively of having improved Metadata implementors as a way to strategically optimize code that is duplicated across the board on UnitOfWork, Persisters and SqlWalker. This should have invisible changes to end-user, but will unlock the possibility of specifying how the eager fetching could happen (either through join or through follow-up select).

Right now UnitOfWork is also doing a lot when comparing changesets. We've discussed internally how we could make it OO, and also delegate the change detection to Metadata implementors. This is also a couple months ahead of the game.

We made a decision of trying to ditch Query\Parser and leveraging Hoa\Compiler. Anyone should be able to work on this today, but it's non-trivial and required deep understanding of how we built our current parser and also some SQL standards.

I'll attempt to keep this issue with some background information, but ORM.NEXT project is the way to get a more accurate status of the project.

@mnapoli

This comment has been minimized.

Show comment
Hide comment
@mnapoli

mnapoli Apr 11, 2017

Contributor

@guilhermeblanco would crowdfunding or sponsoring help in any way, or are you committed to some kind of full-time job (i.e. can money help free some time for you)? Given how widely used Doctrine is, I think a few companies and individual could help financially.

Contributor

mnapoli commented Apr 11, 2017

@guilhermeblanco would crowdfunding or sponsoring help in any way, or are you committed to some kind of full-time job (i.e. can money help free some time for you)? Given how widely used Doctrine is, I think a few companies and individual could help financially.

@Ocramius

This comment has been minimized.

Show comment
Hide comment
@Ocramius

Ocramius Apr 11, 2017

Member

@mnapoli we discussed crowdfunding a few times, but never got to actual concrete details.

For now, I personally applied to http://prototypefund.de/ to develop #5550 in the second half of this year, but that's just a personal bet.

If that fails, I'll probably catch up with the rest of @doctrine/doctrinecore and set up a non-profit organisation for Doctrine, so that we can accept funding.

Member

Ocramius commented Apr 11, 2017

@mnapoli we discussed crowdfunding a few times, but never got to actual concrete details.

For now, I personally applied to http://prototypefund.de/ to develop #5550 in the second half of this year, but that's just a personal bet.

If that fails, I'll probably catch up with the rest of @doctrine/doctrinecore and set up a non-profit organisation for Doctrine, so that we can accept funding.

@pstephenson02

This comment has been minimized.

Show comment
Hide comment
@pstephenson02

pstephenson02 Apr 12, 2017

I've heard good things about https://www.patreon.com/ as well to help with sustaining contributions. I believe Laravel uses this and I'm sure a number of other open source projects.

pstephenson02 commented Apr 12, 2017

I've heard good things about https://www.patreon.com/ as well to help with sustaining contributions. I believe Laravel uses this and I'm sure a number of other open source projects.

@Ocramius

This comment has been minimized.

Show comment
Hide comment
@Ocramius

Ocramius Apr 12, 2017

Member
Member

Ocramius commented Apr 12, 2017

@psihius

This comment has been minimized.

Show comment
Hide comment
@psihius

psihius Apr 14, 2017

@Ocramius I'd suggest taking steps and setting up infrastructure for that. Doctrine is a huge and very popular project that is one of the pillars of the PHP world. If somethong happens to it, it's gonna be a mess, especially if it results in project splitting into forks and all that stuff. There also needs to be a knowledge base built up and docs really need a lot if work (they are vague in a lot of places and sometimes you can't even understad the params of some methods even looking into the code)

psihius commented Apr 14, 2017

@Ocramius I'd suggest taking steps and setting up infrastructure for that. Doctrine is a huge and very popular project that is one of the pillars of the PHP world. If somethong happens to it, it's gonna be a mess, especially if it results in project splitting into forks and all that stuff. There also needs to be a knowledge base built up and docs really need a lot if work (they are vague in a lot of places and sometimes you can't even understad the params of some methods even looking into the code)

@Ocramius

This comment has been minimized.

Show comment
Hide comment
@Ocramius

Ocramius Apr 14, 2017

Member

I'd suggest taking steps and setting up infrastructure for that.

Please scroll up by 2 comments: #6211 (comment)

Member

Ocramius commented Apr 14, 2017

I'd suggest taking steps and setting up infrastructure for that.

Please scroll up by 2 comments: #6211 (comment)

@CarlosEduardo

This comment has been minimized.

Show comment
Hide comment

Great job guys! CC: @guilhermeblanco @Ocramius @lcobucci

@weaverryan

This comment has been minimized.

Show comment
Hide comment
@weaverryan

weaverryan Jan 23, 2018

Contributor

I noticed there's no develop branch (anymore) - has the 3.0 branch moved somewhere else? Thanks :)

Contributor

weaverryan commented Jan 23, 2018

I noticed there's no develop branch (anymore) - has the 3.0 branch moved somewhere else? Thanks :)

@Majkl578

This comment has been minimized.

Show comment
Hide comment
@Majkl578

Majkl578 Jan 23, 2018

Member

@weaverryan Develop has been merged into master which became 3.0-dev, see #6955. 2.x branches are in feature freeze now.

Member

Majkl578 commented Jan 23, 2018

@weaverryan Develop has been merged into master which became 3.0-dev, see #6955. 2.x branches are in feature freeze now.

@weaverryan

This comment has been minimized.

Show comment
Hide comment
@weaverryan

weaverryan Jan 23, 2018

Contributor

@Majkl578 - I finally found that - sorry for not noticing it :). Very exciting! Thanks!

Contributor

weaverryan commented Jan 23, 2018

@Majkl578 - I finally found that - sorry for not noticing it :). Very exciting! Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment