Incubators Proposal

[johnspackman]

This issue is to propose a project-governance protocol for Qooxdoo projects, based loosely on Apache “Incubator” projects, where new additions to the core framework can be developed with a rapid code/test/review/release cycle and at the same time have the oversight of the core team, following our core principals of quality control and longevity.

When proposing large additions to the core framework, using PRs is not ideal because integrating a PR into master is an all-or-nothing decision - the bigger the addition, the more it would benefit from real-world testing before integrating. OTOH it’s difficult to get real world testing because testers have to switch to using the PR’s branch (which is really hard if testers also need to use their own fork of the framework, or more than one feature under development), plus logistical difficulties such as the branch needs to be constantly rebased.

This puts pressure on the core-team to accept a PR before it is fully tested, and subsequent changes for improvements or bug fixes require a round trip of submission and approval by the whole core team each time; the newer a feature is, the more likely there are to be a need for “hotfixes” to solve urgent problems, which again is more pressure to accept changes without a proper review, and possibly a sign that the original review process wasn’t good enough.

Incubator Libraries

New features could be presented as an “Incubator” project, which would be a repo inside the https://github.com/qooxdoo organisation and which would follow our normal rules for quality control (i.e. a PR/review/merge cycle, concerns for backwards compatibility, policy on documentation, etc) and longevity. The project becomes an official sub project, so we’re putting commitment as a group into some kind of stewardship that may outlive the original developer’s involvement (which is not to say that the project cannot be deprecated and retired, just that the life cycle goes beyond one person).

As an official project (especially one which is being considered for inclusion into the core framework) it would be normally sponsored by at least one core-team member (or at least, someone who is a strong candidate to become a core team member). And if it will use the qx.* namespace, there should be some discussion on which parts of the namespace it should occupy.

Backwards Compatibility

As an official Incubator, the project would follow Qooxdoo guidelines for compatibility - EG freezing the API when we get to beta, or deprecating features in one version before removing them in the next, but this compatibility would be separate from that of the core framework.

For example, an Incubator could be compatible with Qooxdoo v6 and yet also take advantage of ES6 and therefore require the compiler (and therefore be incompatible with the generator).

Potentially, an Incubator could include an ES6->ES5 transpile as part of it’s release process to produce a version especially for generator compatibility; but some transpiled code is very hard to read: async/await is translated into a state machine inside the function where it occurs. The decision to provide this or not could be left the Incubator

Naming Convention

It would be useful to be able to identify repos with special “Incubator” status, the easiest way to do that could be to have a naming convention eg append “.incubator” to the repo name. We already use “qxl.” as a prefix for libraries like qxl.apiviewer and qxl.demobrowser - although a repo name like “qxl.apiviewer” implies that there is a namespace called “qxl.apiviewer”.

For Incubators which are proposed as an addition to the core framework, and which use the qx.* namespace, we could prefix with “qx.” - eg https://github.com/qooxdoo/qx.newfeature.incubator. This makes it clear that it is an officially supported project / proposal, but the incubator status shows a certain flexibility and independent development stream from the core framework.

Using the Incubator

Because the compiler supports dependencies it would be easy for anyone to incorporate Incubators into their development - if the Incubator is ES5 then the same can be achieved with the generator, albeit with some additional manual work to get it up and running.

Example Use Cases

In recent issues #9780 and #9781, @cboulanger and I have been working on a sub project which is in the qx.* namespace; we’ve chosen to do that (based on a loose discussion) because it’s a feature which seems like a really good fit for Qooxdoo core (the compiler allows namespaces to be split across libraries) and had a positive response; as it’s a large feature it made sense to develop it outside the core.

The code in the latest alpha version is getting fairly complex, and relies heavily on ES6 - not just nice-to-have syntactic sugar like arrow functions, but async/await handling is a huge improvement for asynchronous work, and in this case async is a very big part of the code. Translating it into ES5 would be a lot of grind with the only value being backwards compatibility for the generator.

If incubators are accepted, I would propose dividing the work so far into two separate Incubator projects under the https://github.com/qooxdoo organisation:

qx.persistence.incubator - provides persistence of Qooxdoo objects, allowing arbitrary depth of objects to be stored in a flat object storage, including support for complex data features such as recursive references.

qx.remote.incubator - provides the remote data binding discussed in #9781, building on the code in qx.persistence.incubator to synchronise objects across network boundaries (ie client/server and window-to-window) and add remote code execution.

@cboulanger and I would both have ongoing projects which would use these Incubators and therefore we have vested interest in getting them established imminently. When considered as a possible addition to Qooxdoo as a whole, these incubators would be exploring a foundation for full stack Javascript development and could grow to be a strong feature and selling point of the project.

[cboulanger]

I fully support this proposal. I wouldn't be concerned with generator compatibility, though, as this would put an unnecessary burden on the process with very little benefit. I think we should establish a release workflow that would allow to create qooxdoo variants that are based on the latest, backward-compatible main branch and include the incubator project code. We should then schedule the full inclusion of these features for version 7, which removes the generator and backward compatibility. With the release of version 6, we can document how to use the incubator variant for people that want to use the attractive features that you outline.

[johnspackman]

I wouldn't be concerned with generator compatibility, though, as this would put an unnecessary burden on the process with very little benefit

I completely agree; that's not my reason for proposing Incubators, but it was one of the things that started me thinking about it. The async/await is just so useful that ES5 compatibility is impractical by comparison

create qooxdoo variants that are based on the latest [...] and include the incubator project code. We should then schedule the full inclusion of these features for version 7

We could, and the compiler's recursive dependency resolution makes that trivial to try it out on the fly before committing to providing an official distro.

I would much prefer that approach to something creating a "Qooxdoo v7" branch or "Qooxdoo v8" branch that is coded on, because "the next great version" is a huge distraction - it's very tempting to work on the latest and greatest features, which leads to the BC version becoming neglected. We have enough difficulty as it is, knowing what to back port to v5 without compounding the problem.

Incubators would be a big staging area for changes, and major releases would be the ideal time to decide on "taking the plunge" and integrating an incubator into the core release.

[cboulanger]

Yes, what about a "qooxdoo.next" branch repo, together with a "@qooxdoo/next" NPM package, which would be automatically maintained and released by GitHub Actions? It would assemble the branch and the package from the framework, the compiler, and the incubator projects, and would have listing, testing, etc.

[cboulanger]

(historically interesting, but with completely different goals: https://github.com/qooxdoo/next)

[derrell]

I like the general goal introduced here, but I'm not sure I'm fully on board with this approach. The problems I see are that we have an already-overworked team, and this approach adds more (eventually) to qooxdoo core (or at least, required to be supported by the qooxdoo team). I really like the library scheme that we've implemented. We have the qxl namespace for libraries that are highly related to core functionality. I'm curious why what's being discussed here doesn't fit directly into that architecture? Why does it require a different architecture. And what projects fit into incubator vs qxl library, and why?

[cboulanger]

@derrell I'd say: qxl.* libraries are self-contained packages which are in general not aimed to be included into the core, whereas incubator projects, if I understand @johnspackman correctly, would have the qx namespace and will, when finished and if there is consensus among the core devs be merged into the core.

One major benefit of the incubator approach would be that we can enforce quality checks that are not necessarily enforceable with a qxl.* library. This would work, for example, if write some scripts that would merge core, compiler. and the incubator projects into one repository which would then run checks etc. and could be checked out and regularly released. Creating such a workflow with individual qxl.* libraries would be very tedious.

The core team would not be concerned with the individual incubator project except when 1) it is first accepted as such and 2) it is ready to be reviewed. At least that is how I would envision it - maybe @johnspackman has a different idea for implementation...

[derrell]

My inclination would be to keep them as qxl.* libraries. We could even create those same scripts to merge core, compiler, and qxl projects and run checks on them, for quality control... yet still keep the libraries as separate entities without imposing additional maintenance requirements on the core team. I'm not totally against the incubator idea; I just have a strong suspicion that it will come back to bite us in terms of maintenance in the future.

[johnspackman]

Yes, what about a "qooxdoo.next" branch repo

@cboulanger I think we should stay away from a branch, because I think that it's important to focus on the current release of qooxdoo core framework; the qooxdoo/next was a disappointing diversion, and everyone's curious about the "what's 'next'?" A rollup package would be OK, but if users have to use the compiler that would be an optimisation-of-distribution.

@derrell I think that there is a problem in how we do things now when someone submits a PR which is a substantial change and with dozens of files - I've done it myself, and it's always a dilemma trying to balance the need for change (driven ultimately by my own private needs) vs time constraints & how long to hold off. It's really difficult to share code in practice if it is only available in a PR, and I hope this is something that will make it easier and less work for the team to review because a whole section of "core" new feature can be matured before final agreement to join the core.

My thoughts started from the point of view of having a qx.* namespace for things, and deliberately wanting to keep them out of the workflow of the core team. When it's components which are contributions to "the Qooxdoo experience" so to speak, then it's difficult to promote components which are external libraries, and if we're promoting internal libraries such as qxl.*, then we're already going down the route of building libraries that are part of the github.com/qooxdoo organisation

A big part of the incubator idea is that it allows substantial additions to be trialed without requiring the core team to approve via the core framework repo. Incubators would be a mechanism for saying "we think this is a good solution, but it needs to prove itself before we integrate it into the core".

It's like saying that the integration status of feature "X" is in alpha or beta, even if feature "X" is itself considered stable in its own right by a number of people.

When v6 is released and we're working towards v7, that could be a good point to consider whether an incubator is stable enough to incorporate into core. I think this is @cboulanger's point, that we should have a road map for when the proposed new libraries are integrated into the core, ie they are not allowed to languish at github, they have to have a specific destiny - of course this implies that if they don't make it, they get deprecated and then retired.

IMHO it is important to prevent github.com/qooxdoo from becoming a dumping ground of half finished repos, and most importantly prevent qx.* from being populated with code that's since fallen out of favour. I can't remember the timings or how it happened, but there always seems to have been a deprecated qx.io.remote.* in favour of qx.io.request.*.

I do have a vested interest in this - I'm finding that more of my work lately is leading me to take Qooxdoo more and more into the server, and where there is an opportunity to collaborate this is a good two-way thing. A case in point is @cboulanger 's issue about remote binding, happened to be something I was already working on (in part) and our joint need has accelerated things somewhat; given that it was interesting to others too, it seemed logical to progress it.

[cboulanger]

@johnspackman I mistakenly said "branch" when I really meant "repo" (I have edited my comment since then). I fully agree a branch is not the right solution. What I meant is some completely automated way of distributing an "experimental" version of qooxdoo which also does the Quality Assurance. That's why suggested using the new GitHub Actions feature to assemble this version. The repo itself would not get any manual update / change, all work is done either in the framework, the compiler, or the incubator repos. The difference between incubators and qxl.* libraries would be that incubators aim to be included into the core eventually and have the qx namespace, whereas qxl.* libraries wouldn't. This seems a plausible architecture that would not place an additional burden on the core team.

[derrell]

Ok. I'm coming around to it. I still have reservations and concerns, but I'm getting there.

What do we do, if the namespace includes .incubator (or some such thing) and then it moves out of the incubator? Does this mean that all apps that were using it need to modify their code to remove that portion of the namespace?

[hkollmann]

The namespace should'nt have a .icubator in it. IMHO the repository name .incubator - or even better for sorting purpose - incubator. are enough.

[johnspackman]

I mistakenly said "branch" when I really meant "repo" [...] automated way of distributing an "experimental" version of qooxdoo

ah, sorry - an npm module called @qooxdoo/experimental works better for me - I think my main concern is not implying anything too exciting about it - when Qooxdoo "next" was discovered (it was not announced, someone found the repo and asked on the list) lots of people including me wanted to know what this exciting new project was? When can we try it? Is there any docs? etc etc.

@derrell What do we do, if the namespace includes .incubator (or some such thing) and then it moves out of the incubator? Does this mean that all apps that were using it need to modify their code to remove that portion of the namespace?

I agree with @hkollmann I was thinking that the repo has the .incubator suffix but the namespace in code does not - specifically so that when (if) it gets integrated, there is no change to user code. So as a user, your taking the risk that the incubator may never be integrated, but OTOH if it is integrated then the merge should be transparent. Perhaps that should be part of the manifesto, that it breaking changes when integrating an Incubator are not allowed (or perhaps avoided if at all possible).

With the dependency management of compiler, the incubator could be disabled at the same time that it is integrated - this could be a tag on the repo or something in qx.newfeature.incubator/qooxdoo.json that says "I'm now in the Qooxdoo core, don't use me" (or the repo could have a commit deleting all files except for README that says the same).

[johnspackman]

@hkollmann or even better for sorting purpose - incubator.

Yes, I think I prefer that style

[cboulanger]

@johnspackman @qooxdoo/experimental works for me, too!

@derrell The point for me is that such a repo/package would be a functional equivalent of a "development" branch, which we don't have, and a much better implementation of it at that. It would mean that the master branch would only receive smaller, manageable updates, and the "big" new features would only be merged once they have proven to be totally stable. As @johnspackman has argued, the current problem is that currently, the temptation is to merge new features once they appear stable, which is often not the case, and then the core team has to be involved when fixing them. With the proposed setup, master will be kept free from such experiments.

[derrell]

This has been well-presented, and my concerns have been addressed. I'm in!

[johnspackman]

great :) Please see qooxdoo/qx-dev-manual#5

[hkollmann]

closed with qooxdoo/qx-dev-manual#5