New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposed Packaging in Qooxdoo v6 #9369

Open
johnspackman opened this Issue Aug 10, 2017 · 12 comments

Comments

Projects
None yet
5 participants
@johnspackman
Member

johnspackman commented Aug 10, 2017

This issue is to discuss the proposed packaging for Qooxdoo v6.

One-Step Install and Setup

It’s anticipated that Qooxdoo will primarily be delivered via npm, which will allow a really nice and simple getting started guide like this:

npm install qooxdoo
qx create myapp
cd myapp
qx compile
## Now open source-output/myapp.html in your browser ...

Note that npm install qooxdoo should be all that is necessary to setup the entire Qooxdoo development environment - it will add the qx command line app and incorporate the entire of the framework necessary to start developing Web (aka Desktop or Mobile) and Server applications.

Use in Node.JS

We would also like users to be able to easily use Qooxdoo in nodejs applications, starting with our own components qooxdoo-cli and qooxdoo-compiler; they should be able to use Qooxdoo by just npm install qooxdoo-sdk.

Note that in this scenario (Zero-Compilation), we would not require that there is a separate compilation step in order to use Qooxdoo in a nodejs application - you might choose to add the compilation step later, for example if you wanted automatic dependencies/class loading, or you wanted to use the ES6->ES5 transpilation, etc, but we would not force you to use the extra compilation step unless/until you wanted to. This is important because the working practice for just about every node tutorial and npm module ever written is to just use npm install ... and require().

Zero-Compilation Web Applications

Another similar request is that there is an option to create web applications without using a compilation step at all; this would mean providing the whole of Qooxdoo for use by a single <script> tag in source & build versions, available via a CDN or a users’ own server.

Python Toolchain (generate.py)

We would like to be able to remove the python toolchain altogether from v7.0, but deprecate it during v6.0 because our existing user base will want an upgrade path (and the javascript toolchain needs real world testing). We could consider packaging the python toolchain as a separate npm module - this would be useful to allow the generator/python tools to be maintained outside of the main release cycle of Qooxdoo and would allow us to massively slim down the main distribution … however, note that the generator does not support ES6 syntax and when 7.0 is released, ES6 will be permitted for use inside the Qooxdoo core framework and at that point the generator will no longer work.

Note that it is planned that we replace the API generator/viewer and the TestRunner in v6.0 because these are tools used by the end user and the generator does not support ES6 syntax - effectively this requires that we reimplement the API generator/viewer and TestRunner using the new tool chain (WRT the API generator, we also want to support “true” JSDoc standard so there are potentially changes that need to be made there anyway).

The new API generator/viewer and TestRunner could both be separated out into separate npm modules; there might also be good reason to separate out the Python toolchain including the API generator/viewer and TestRunner so that there is clear separation of concerns and an immediate upgrade path from 5.x to 6.x.

The disadvantage is that end users would have to change their batch processes for API & TestRunner - I am unclear as to how much of an issue this would be, presumably it is pretty easy? It might be possible to do it anyway by modifying the config.json files so that the “test-runner*” jobs are farmed out to the external modules?

Distribution Packaging

So far, our proposal is to have these npm modules:

  • qooxdoo-sdk is the framework, suitable for desktop, mobile, & server development
  • qooxdoo-compiler depends on qooxdoo-sdk
  • qooxdoo-cli depends on qooxdoo-sdk and qooxdoo-compiler
  • qooxdoo is a wrapper that collects all of these into one place so we get a nice, one-step install package for users (see the “getting started” sample above)

Potentially, we could also consider these new modules:

  • qooxdoo-api-generator depends on qooxdoo-sdk and qooxdoo-compiler
  • qooxdoo-test-runner depends on qooxdoo-sdk and qooxdoo-compiler
  • qooxdoo-legacy-toolchain contains the python based generate.py, 5.x API viewer/generator, and 5.x TestRunner

A complication to consider is that a "zero-compilation" version of qooxdoo-sdk (whether for node.js or for web apps) would need to use pre-compiled versions of Qooxdoo classes - the framework would be compiled into a package of source code and resources. However, for “normal” development of applications, users are expected to use the toolchain on their own code (combined with contribs and Qooxdoo framework itself) and the toolchain requires access to the original source code, and not the transpiled version in the zero-compilation distribution.

Rather than have a qooxdoo-sdk module for development and qooxdoo-sdk-zero-compilation, we would incorporate the transpiled code with the source code and just have qooxdoo-sdk.

Version Numbers

Can we say that in semver of x.y.z, x.y should be synchronised across all modules, but the module increments its own .z as often as it likes?

Questions

  • Can we move the entire Python toolchain to a separate module? This would complicate the 5.x->6.x upgrade process a little but makes great "Separation of Concerns" type of sense
  • Should we bother to provide legacy version of API generator/viewer and TestRunner? Or can we just assume that our next version will "just work" ;)
@cboulanger

This comment has been minimized.

Show comment
Hide comment
@cboulanger

cboulanger Aug 10, 2017

Contributor

Moving the python toolchain into a separate module makes a lot of sense, since it could be separately maintained for legacy applications.

Another idea would to farm out the "components" not to npm modules, but to contribs. This would remove one layer of maintenance complexity: you could work with the GitHub repos only and skip the npm publish step. This would require putting each component into its own repository. But there are pros & cons to this approach.

Contributor

cboulanger commented Aug 10, 2017

Moving the python toolchain into a separate module makes a lot of sense, since it could be separately maintained for legacy applications.

Another idea would to farm out the "components" not to npm modules, but to contribs. This would remove one layer of maintenance complexity: you could work with the GitHub repos only and skip the npm publish step. This would require putting each component into its own repository. But there are pros & cons to this approach.

@johnspackman

This comment has been minimized.

Show comment
Hide comment
@johnspackman

johnspackman Aug 10, 2017

Member

aha, yes I like this approach more - npm is a pain, with version numbers and republishing and now that we have qx the direct repo approach is trivial to explain and document.

Member

johnspackman commented Aug 10, 2017

aha, yes I like this approach more - npm is a pain, with version numbers and republishing and now that we have qx the direct repo approach is trivial to explain and document.

@cboulanger

This comment has been minimized.

Show comment
Hide comment
@cboulanger

cboulanger Aug 10, 2017

Contributor

Plus, it allows to control for qooxdoo versions, which is inherently difficult in npm (see the controversy over peer dependencies)

Contributor

cboulanger commented Aug 10, 2017

Plus, it allows to control for qooxdoo versions, which is inherently difficult in npm (see the controversy over peer dependencies)

@tcsaddul

This comment has been minimized.

Show comment
Hide comment
@tcsaddul

tcsaddul Aug 11, 2017

+1 for the Zero Compilation Web Applications. It will also enable Typescript support.

BTW, the following files are the only two (2) files needed in our Visual Studio IDE. Thanks to jbaron for updating it to Qooxdoo 5.0.2

Definition File: https://github.com/jbaron/cats/blob/master/src/typings/qooxdoo.d.ts
Javascript File: https://github.com/jbaron/cats/blob/master/resource/qooxdoo.js

tcsaddul commented Aug 11, 2017

+1 for the Zero Compilation Web Applications. It will also enable Typescript support.

BTW, the following files are the only two (2) files needed in our Visual Studio IDE. Thanks to jbaron for updating it to Qooxdoo 5.0.2

Definition File: https://github.com/jbaron/cats/blob/master/src/typings/qooxdoo.d.ts
Javascript File: https://github.com/jbaron/cats/blob/master/resource/qooxdoo.js

@johnsky

This comment has been minimized.

Show comment
Hide comment
@johnsky

johnsky Sep 8, 2017

@johnspackman, the proposal regarding npm packages is reasonable, but SDK includes compiler, cli and any other stuff, so sdk contains everything a developer need:

  • qooxdoo-sdk - qooxdoo developer toolkit (now and hopfully in future) is framework itself and a number of tools (demobrowser, apiviewer, generate.py or replacement);
  • qooxdoo-toolkit - includes compiler, CLI, demobrowser, apiviewer and other replacements of generate.py
  • qooxdoo-toolkit-legacy - includes python based toolkit, demobrowser, apiviewer

Is it really necessary to have so many npm packages you proposed?
Maybe, is that possible to move some tools to separate positories, and it would probably be convinient for supporter/contributor? Usually, it needs to have tools and framework all in one place.

Please note, that qooxdoo-sdk is fundamental for qooxdoo SPA application developer, at present.

johnsky commented Sep 8, 2017

@johnspackman, the proposal regarding npm packages is reasonable, but SDK includes compiler, cli and any other stuff, so sdk contains everything a developer need:

  • qooxdoo-sdk - qooxdoo developer toolkit (now and hopfully in future) is framework itself and a number of tools (demobrowser, apiviewer, generate.py or replacement);
  • qooxdoo-toolkit - includes compiler, CLI, demobrowser, apiviewer and other replacements of generate.py
  • qooxdoo-toolkit-legacy - includes python based toolkit, demobrowser, apiviewer

Is it really necessary to have so many npm packages you proposed?
Maybe, is that possible to move some tools to separate positories, and it would probably be convinient for supporter/contributor? Usually, it needs to have tools and framework all in one place.

Please note, that qooxdoo-sdk is fundamental for qooxdoo SPA application developer, at present.

@johnspackman

This comment has been minimized.

Show comment
Hide comment
@johnspackman

johnspackman Sep 8, 2017

Member

@johnsky the reasons we're proposing dividing Qooxdoo up into a number of modules is partly to keep download size down, and partly for separation of concerns inside the project. Components (like the demo browser) sprawl across the repository, and ideally would follow a separate (although similar) release schedule to the main framework anyway - these are client applications that use the Qooxdoo Framework, even though they are developed as part of the Qooxdoo Project.

This separation should make it easier for new users to get to know the project as well as make for an easy transition for more established users (and help us clear up a lot of files that are simply not used any more along the way).

What we're proposing is that qooxdoo has everything a developer needs, so very similar to the qooxdoo-sdk in your comment, except that the demobrowser and apiviewer are held separately. The apiviewer is a bit of a special case because users of the framework want to build their own version with their own API documentation - this will be possible thanks to @cboulanger 's contrib mechanism which will allow the qx command line to download and build a custom api viewer with the single command qx apiviewer enable. I suspect that users of the Qooxdoo Framework don't care about where the apiviewer (or demobrowser) is stored, so this is a perfectly good solution - and the simplicity of qx apiviewer enable is more engaging than editing config files etc.

The reason that qooxdoo-compiler is an npm module (currently called qxcompiler, but that will change) is because it is an API that is used to compile framework based apps; it uses the framework itself, so we want the framework to be a separate module that it can call upon (without dragging in copy of itself recursively, or the demobrowser/apiviewer etc)

The reason that qooxdoo-cli is an npm module (currently called qx-cli, but that too will change) is because npm install is a great and easy way to distribute and install applications. If we were not using node/npm already we would probably use a different mechanism, but npm is a fundamental part of the node ecosystem, so that fact combined it's ability to stitch together modules dependencies makes it ideal.

The reason that qooxdoo-legacy-toolchain is separate is just to keep it separate - the same as your qooxdoo-toolkit-legacy.

While the other two, apiviewer and demobrowser should be separate (IMHO), I tend to agree with you that making them npm modules does not necessarily make best sense. I think now that they will probably be best delivered as contribs.

Member

johnspackman commented Sep 8, 2017

@johnsky the reasons we're proposing dividing Qooxdoo up into a number of modules is partly to keep download size down, and partly for separation of concerns inside the project. Components (like the demo browser) sprawl across the repository, and ideally would follow a separate (although similar) release schedule to the main framework anyway - these are client applications that use the Qooxdoo Framework, even though they are developed as part of the Qooxdoo Project.

This separation should make it easier for new users to get to know the project as well as make for an easy transition for more established users (and help us clear up a lot of files that are simply not used any more along the way).

What we're proposing is that qooxdoo has everything a developer needs, so very similar to the qooxdoo-sdk in your comment, except that the demobrowser and apiviewer are held separately. The apiviewer is a bit of a special case because users of the framework want to build their own version with their own API documentation - this will be possible thanks to @cboulanger 's contrib mechanism which will allow the qx command line to download and build a custom api viewer with the single command qx apiviewer enable. I suspect that users of the Qooxdoo Framework don't care about where the apiviewer (or demobrowser) is stored, so this is a perfectly good solution - and the simplicity of qx apiviewer enable is more engaging than editing config files etc.

The reason that qooxdoo-compiler is an npm module (currently called qxcompiler, but that will change) is because it is an API that is used to compile framework based apps; it uses the framework itself, so we want the framework to be a separate module that it can call upon (without dragging in copy of itself recursively, or the demobrowser/apiviewer etc)

The reason that qooxdoo-cli is an npm module (currently called qx-cli, but that too will change) is because npm install is a great and easy way to distribute and install applications. If we were not using node/npm already we would probably use a different mechanism, but npm is a fundamental part of the node ecosystem, so that fact combined it's ability to stitch together modules dependencies makes it ideal.

The reason that qooxdoo-legacy-toolchain is separate is just to keep it separate - the same as your qooxdoo-toolkit-legacy.

While the other two, apiviewer and demobrowser should be separate (IMHO), I tend to agree with you that making them npm modules does not necessarily make best sense. I think now that they will probably be best delivered as contribs.

@cboulanger

This comment has been minimized.

Show comment
Hide comment
@cboulanger
Contributor

cboulanger commented Sep 8, 2017

@johnsky

This comment has been minimized.

Show comment
Hide comment
@johnsky

johnsky Sep 8, 2017

It's hard to image developer workflow changes overall, and the idea seems good. Thank you for the details, @johnspackman!

johnsky commented Sep 8, 2017

It's hard to image developer workflow changes overall, and the idea seems good. Thank you for the details, @johnspackman!

@cboulanger

This comment has been minimized.

Show comment
Hide comment
@cboulanger

cboulanger Nov 30, 2017

Contributor

Concerning the qooxdoo-sdk- package - It just occurred to me that it really makes no sense to upload the whole qooxdoo-sdk to npm for each and every small change that is made to the framework, in particular before we have any stable release (the only occasion this really makes sense). Instead, the qooxdoo-sdk package should consist of a stub that executes a script when installed that downloads the ZIP from github and unpacks it to ./qooxdoo @johnspackman @hkollmann

Contributor

cboulanger commented Nov 30, 2017

Concerning the qooxdoo-sdk- package - It just occurred to me that it really makes no sense to upload the whole qooxdoo-sdk to npm for each and every small change that is made to the framework, in particular before we have any stable release (the only occasion this really makes sense). Instead, the qooxdoo-sdk package should consist of a stub that executes a script when installed that downloads the ZIP from github and unpacks it to ./qooxdoo @johnspackman @hkollmann

@johnspackman

This comment has been minimized.

Show comment
Hide comment
@johnspackman

johnspackman Dec 1, 2017

Member

The idea is that qooxdoo-sdk is the npm package that you add to an npm module in order to use Qooxdo in your npm based tool (eg compiler and cli would use const qx = require("qooxdoo-sdk");). So that download script would have to be part of npm's install/setup process, and presumably npm will run it every time a new version is released, possibly after deleting the old version so even if we wanted to do an incremental download somehow I don't know if it would be possible to do one.

We don't release that often (although once we've reached v6 we should probably release more frequently), and if anyone needs a more fine grained delivery they would use git.

I think that the qooxdoo-sdk package should definitely have the whole framework (ie the contents of framework/source/) as a JS package. CLI would find it's qxpath as ./node_modules/qooxdoo-sdk, and it's npm's problem how to obtain it.

Member

johnspackman commented Dec 1, 2017

The idea is that qooxdoo-sdk is the npm package that you add to an npm module in order to use Qooxdo in your npm based tool (eg compiler and cli would use const qx = require("qooxdoo-sdk");). So that download script would have to be part of npm's install/setup process, and presumably npm will run it every time a new version is released, possibly after deleting the old version so even if we wanted to do an incremental download somehow I don't know if it would be possible to do one.

We don't release that often (although once we've reached v6 we should probably release more frequently), and if anyone needs a more fine grained delivery they would use git.

I think that the qooxdoo-sdk package should definitely have the whole framework (ie the contents of framework/source/) as a JS package. CLI would find it's qxpath as ./node_modules/qooxdoo-sdk, and it's npm's problem how to obtain it.

@cboulanger

This comment has been minimized.

Show comment
Hide comment
@cboulanger

cboulanger Dec 1, 2017

Contributor

I agree with you that real releases should be ready-to-go npm packages. I wasn't clear enough: I mean as long as we are in the alpha/beta phase, we should already be offering an NPM package so that the workflow is established, but it seems so incredibly wasteful (and a unneccessary work) to be releasing NPM packages of a moving target. My idea was that during this time, we could have a qooxdoo-sdk package that let's us skip the git clone step and provides the newest qooxdoo master there is without any additional work for the developer using the qooxdoo npm package. Does that make more sense to you?

Contributor

cboulanger commented Dec 1, 2017

I agree with you that real releases should be ready-to-go npm packages. I wasn't clear enough: I mean as long as we are in the alpha/beta phase, we should already be offering an NPM package so that the workflow is established, but it seems so incredibly wasteful (and a unneccessary work) to be releasing NPM packages of a moving target. My idea was that during this time, we could have a qooxdoo-sdk package that let's us skip the git clone step and provides the newest qooxdoo master there is without any additional work for the developer using the qooxdoo npm package. Does that make more sense to you?

@hkollmann

This comment has been minimized.

Show comment
Hide comment
@hkollmann

hkollmann Dec 1, 2017

Member

That's the idea i am working on. We have a stable qooxdoo-sdk package version 5.0.2 on npm . We could use this to integrate into qx-cli package. We can't use qooxdoo for this - as suggested above - because the qooxdoo package is used for the qooxdoo Server components for a long time.

If you want to use the develop version of qooxdoo you can use npm link ../qooxodoo
For this to work the package name in package.json in qooxdoo must change to qooxdoo-sdk because npm links works with package names.

Im just testing if changing the package name will break the build system before i submit a PR for this

Member

hkollmann commented Dec 1, 2017

That's the idea i am working on. We have a stable qooxdoo-sdk package version 5.0.2 on npm . We could use this to integrate into qx-cli package. We can't use qooxdoo for this - as suggested above - because the qooxdoo package is used for the qooxdoo Server components for a long time.

If you want to use the develop version of qooxdoo you can use npm link ../qooxodoo
For this to work the package name in package.json in qooxdoo must change to qooxdoo-sdk because npm links works with package names.

Im just testing if changing the package name will break the build system before i submit a PR for this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment