New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PLIP: Resource registry improvements #1955

Open
thet opened this Issue Feb 27, 2017 · 8 comments

Comments

6 participants
@thet
Member

thet commented Feb 27, 2017

PLIP: Resource registry improvements #1955

Responsible Persons

Proposer: Johannes Raggam

Seconder:

URL: #1955

There is a Definitions section later in this PLIP, which explains some of the terms used here.

Abstract

  • Get rid of the need of bundle resources.

  • Instead, let bundles, which are technically just bunch of plone.registry settings, define their dependencies to resources directly instead via bundle resources.
    This makes the dependencies also extendible via just registry manipulation.

  • Store the creation and modification dates of a registry entry.

  • The feature of storing registry metadata must be implemented for plone.registry, as it lacks this feature.
    Via the Zope transaction history we can even get a diff between registry versions, if needed - but only for history aware ZODBs.

  • If the modification date of the filesystem or plone.resource based Resource (or any of it's dependencies) is older than the modification date of the bundle registry settings,
    then show a notification for the integrator to start a recompile of the bundle (through the web or via plone-compile-resources).

  • If there is no compiled bundle or the RR is in development mode: then just include an automatically generated RequireJS require call to include all dependencies (which might be an option also in production mode with HTTP2).

  • Add a button to delete the setting for a path to a compiled version of the bundle.

  • Explicitly add an entry to show/delete a compiled plone.resource based version of the bundle.
    Those resources overload the ones in the filesystem.

  • If recompiling TTW fails, automatically roll back.
    Offer a manual roll-back button anyways.

  • Let RequireJS handle the JS dependency resolution and what to include in the compiled output via the settings of enabled, depends-on (dependencies to other bundles) and dependencies (dependencies to Resources).
    If this isn't possible, we can calculate all the Stub Modules in Python code too.
    This makes the Stub Modules obsolete.

Advantages

  • No need for Bundle Resources anymore - a concept which is very hard to explain.

  • No need for Meta Bundles anymore.
    We just include all bundles.

  • Addon developers are encouraged to add their resources just to one of the standard bundles (plone or plone-logged-in).

  • Addon developers can also create a new bundle and provide a compiled version of it.

  • Integrators get notified when a bundle needs to be recompiled, because filesystem versions have changed.

  • Integrators can better see if a resource or a bundle is shipped from filesystem or from plone.resource.

  • Stub modules do not need to be defined anymore, and any JavaScript isn't included twice that way.

Motivation

Wipe away all the growing pains of our resource registry and the feeling to need to apologize for everything.

Assumptions and Risks

The PLIP #1653 should be considered too.

  • See how Webpack fits into that concept...

  • Maybe RequireJS gets obsolete by Webpack? AFAIK they can well coexist. But I'm missing a bit of knowledge about the concepts and internals of Webpack.

  • This still assumes to use LESS. But SASS seems to replace LESS everywhere.

  • Maybe TTW compilation should be abandoned at all.

Definitions

  • Resources: JavaScript or LESS filesystem resources.
    The have a name to refer to, in registry.xml defined as plone.resources/NAME.

    In case of JavaScript files, these are also RequireJS modules: JavaScript files, which make use of RequireJS define function to create a RequireJS module, associated with a name which can be imported with require.

    We make use of this with our Mockup Patterns [http://plone.github.io/mockup/dev/]. These are patternslib Patterns [http://patternslib.com/] with a JavaScript API in form of a object, which is returned by this module and gets initialized automatically when a class like pat-mymodule if found in the DOM.
    Also any JavaScript resource, which we depend upon. Be it a RequireJS module or not.

  • Bundle resources: Special Resources, which actually define a bundle.

    It defines all the bundle's dependencies, initializes the patternslib registry and can run some other initialization code.

  • Bundles: Bundles define dependencies to resources which are included in the rendered output.
    The have a name to refer to, in registry.xml defined as plone.bundles/NAME.
    The switch compile allows the creation of two types of bundles:

    • If compile is True: This makes it a RequireJS bundle, which should use the RequireJS define method to define the set of dependencies.
      This is then compiled with the command line script plone-compile-resources or Through the Web with r.js, the RequireJS compiler.

    • If compile is False: This makes Plone just output the JavaScript resources which are defined as Resources.
      It's also possible to just define one JavaScript file, which is then just output as is.

  • Meta bundles: There are 3 meta groups: None, default, logged-in. Bundles of a meta group (which is a property of Bundles) are concatenated in the HTML response.
    In case of None, they are not concatenated at all.

  • Stub modules: List of resources, which are already included in another bundle and therefore are only included as "stub modules" without all their code in the compiled output of the bundle.
    he list of stub modules is an attribute of a bundle.

  • Development Mode: If the resource registry is in development mode (this is a setting in plone.registry, in case someone needs to manually set it), then no meta bundles are used and no concatenation or production mode caching URLs (++production++ ...) are used.

    • In development mode you can set a bundle to "Develop JavaScript" and "Develop CSS". In that case, the bundle will not use the compiled version but the resources - which the bundle depends on - directly.
    • In case of Develop JavaScript and compile is True, RequireJS is used to load the resources.
    • In case of Develop CSS and compile is True, the LESS JavaScript compiler is used to include and compile LESS files directly.

Proposal & Implementation

Probably a significant amount of work has to go into the implementation of this PLIP.
The proposal is outlined above.

Deliverables

...

Risks

See above.

Participants

...

Disclaimer

I didn't check against how settings are really named in the UI and in code. I just recalled them, so there might be some errors..

@hvelarde

This comment has been minimized.

Member

hvelarde commented Feb 27, 2017

do you mind adding a definitions part to your proposal: what are bundle resources, meta bundles, stub modules and why we need/needed them?

also:

  • what's the difference between resource registry bundles in production and in development mode?
  • why we need to recompile resources?
@thet

This comment has been minimized.

Member

thet commented Mar 3, 2017

@hvelarde @gforcada I have reworked everythign and included some definitions, made everything more clear and removed some errors.

@vangheem @bloodbare @robgietema @datakurre @ebrehault I would appreciate a review from you guys too :)
It sounds all good to me, but maybe I'm missing something.

@datakurre I don't know how webpack could play a role here.

@ebrehault This would make stub modules obsolete, more or less just for simplicity and because it's essentially not needed because people should extend one of the default bundles anyway. What do you think?

@pbauer That could maybe part of a discussion in Sorrento too (but I'm not there...).

/cc @jensens @rnixx

@ebrehault

This comment has been minimized.

Member

ebrehault commented Mar 4, 2017

@thet sounds good to me :)
And I agree it might be quite a big job.

@datakurre

This comment has been minimized.

Member

datakurre commented Mar 5, 2017

@thet Webpack plays well with RequireJS. I don't believe that you need to think Webpack yet in this PLIP.

My webpack builds depend on config.js and less-variables.js -views, but config.js never included bundles and LESS cannot be removed yet, so I expect to be unaffected by these changes.

@hvelarde

This comment has been minimized.

Member

hvelarde commented Mar 6, 2017

@thet keep it simple: don't mess around RequireJS now and don't try to guess anything. also, I still don't understand why we need to recompile resources.

IMO, Plone should ship with resources bundled; when a upgrade is done, a command line script managed by buildout could take care of the bundling of those resources only. any resources external to Plone (third party resources) must be just rendered as declared: if an add-on declares 3 JS, and 3 CSS, make no assumptions and just render that; all heuristics is broken.

is responsibility of each add-on developer to make thing play well; it's a decision of add-on developers if they want to use RequireJS or no; but Plone should not give support to add-ons not playing by the current rules.

what we need to have is a clear path to cook resources on add-on upgrades as we had before.

@djay

This comment has been minimized.

Member

djay commented Nov 17, 2017

@thet Does this proposal solve the problem of requiring a plugin reinstall to have updated filesystem compiled js/css be active? I can't see that explicitly stated but I might be missing it as a sublte implication.
This problem is a huge pain. You should be able to deploy a plugin and just have it work. Currently on a shared platform you have to go to every plone site and reinstall. If there are unrelated issues with reinstalling then you are in trouble, as your site is broken. I only want to reinstall a plugin if it has an actually upgrade step, not to deploy updated JS.

@hvelarde

This comment has been minimized.

Member

hvelarde commented Nov 17, 2017

that's why I want to get rid of all the bundling process that happens on runtime; I insist that resources should be just registered and rendered without any need of interaction among them; I know that's far from optimal but this problem is really complex and we have no good solution available now.

just look at the insane process of updating any JS component in the core; the repos are continuously and the production cache of any site is destroyed every time a single line of code is changed. that's worst than what I was proposing, IMO.

please just make things easier.

@gp54321

This comment has been minimized.

gp54321 commented Jan 31, 2018

I noticed today that when uninstalling an add-on the resources can't be removed (there is no actual removing done in the exportimport directory modules). This seems well known since an addon like collective.glossary does not even include resource instructions in the uninstall profile. However when the uninstall is done, the plone-legacy bundle is not cooked, and the javascript of the removed add-on keeps loading in the browser. The resources can be deleted by hand in the resource registry but you have to be aware of the problem.

I wonder if a new version (possibly enhanced) of the addon is installed, the new javascript will replace the old in the plone-legacy. That would be very annoying if it is not the case.

Unless I am mistaken I don't see this requirement in your PLIP.
To me it looks more a missing feature than a bug, but I can file an issue if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment