Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal to rewrite + modernize Static Content Deployment 馃槺 #168

Merged
merged 1 commit into from Jun 12, 2019

Conversation

@DrewML
Copy link
Member

commented May 24, 2019

Rendered Proposal

@magento-cicd2

This comment has been minimized.

Copy link

commented May 24, 2019

CLA assistant check
All committers have signed the CLA.

@DrewML DrewML changed the title Proposal to rewrite Static Content Deployment 馃槺 Proposal to rewrite + modernize Static Content Deployment 馃槺 May 24, 2019

@jedmao
jedmao approved these changes May 24, 2019

There are a number of issues with the current implementation of static content deployment.

1. It's prohibitively slow, sometimes taking 10+ minutes just to copy and process files

This comment has been minimized.

Copy link
@joni-jones

joni-jones May 24, 2019

Contributor

It always depends on the number of files to deploy but for Magento vanilla installation (CE + EE) it takes 13sec on my laptop.

This comment has been minimized.

Copy link
@DrewML

DrewML May 24, 2019

Author Member

Right, you're on the happy path it was designed for, which is part of the problem: those speeds are only available on stores with a limited number of files and only 1 locale.

We also need to consider those that can't have a top of the line machine for their work or personal computer. 13 seconds on a MacBook Pro will not be 13 seconds on a < $1k USD laptop from Best Buy.

If it takes several minutes for multiple locales, and the majority of our user base in Europe is using multiple locales, then the speed of a basic store isn't super relevant.

We'll be at MMDE in a few days, though - let's ask around!

- [Theme/module/i18n/"lib/web" file inheritance flattening](https://devdocs.magento.com/guides/v2.3/frontend-dev-guide/themes/theme-inherit.html)
- Locale splitting
- Each locale gets its own static assets under a theme dir
- Less compilation

This comment has been minimized.

Copy link
@joni-jones

joni-jones May 24, 2019

Contributor

I agree about Less compilation and JS minification but not sure about other tasks like moving files, concatenation, and locales generation. I suppose the problem that you need to interact with the file system frequently and as node.js is also single threaded will it be a big difference between node.js and PHP? From my experience python or Go are more suitable for these tasks.

This comment has been minimized.

Copy link
@DrewML

DrewML May 24, 2019

Author Member

I suppose the problem that you need to interact with the file system frequently and as node.js is also single threaded will it be a big difference between node.js and PHP

An extremely large difference. PHP blocks when reading a file, node does not.

File reads in node are not done in the main thread - only your code is single-threaded: node itself can spawn as many threads as it likes (and it does).

node's original selling point was async network and disk I/O - that's where it thrives. Since more of static content deploy is copying/resolving/moving, rather than pure CPU-bound tasks, it's a fairly good fit

File reads happen in the thread pool on the right in this example:
image

This comment has been minimized.

Copy link
@DrewML

DrewML May 24, 2019

Author Member

From my experience python or Go are more suitable for these tasks.

This leads us back to the same problem as PHP (outlined in this document): lack of a good interop story with node. All modern front-end tooling is written assuming node as a runtime.

This comment has been minimized.

Copy link
@Igloczek

Igloczek May 25, 2019

It would be great to think about just styles compilation and be able to extend those tools with other compilers like SASS, Stylus or some PostCSS based process.
I think people will be happy to migrate from Frontools to some offical tools 馃槉


### What opportunities open up with a re-write of SCD using node.js?
- Compilation of modern JS with [Babel](https://babeljs.io/)
- Automatic live reloading of CSS during development

This comment has been minimized.

Copy link
@joni-jones

joni-jones May 24, 2019

Contributor

Could you clarify what do you mean? Now, in the development mode, the css/js will be generated in runtime.

This comment has been minimized.

Copy link
@DrewML

DrewML May 24, 2019

Author Member

Example:

You are on the checkout page. You have 3/4 of the checkout form filled out. You now want to tweak some values in CSS.

Currently, your options are:

  1. Make the change in DevTools, then copy/paste into your store CSS
  2. Make the change on the file system, then refresh the whole page and re-enter all form data (note this applies to all page state, not just forms. Think styling an open dropdown)

Live reloading injects the CSS changes into the live document - you get to keep your browser state, but use your own editor.

This comment has been minimized.

Copy link
@joni-jones

joni-jones May 24, 2019

Contributor

Should be there some kind of file watcher? Will it be a separate tool or an existing one?

This comment has been minimized.

Copy link
@DrewML

DrewML May 24, 2019

Author Member

Should be there some kind of file watcher?

Yep! That would be the idea for development mode (when actively working on front-end tasks). You typically have a persistent process running that keeps a warm cache and does small incremental rebuilds when files change (which typically/optionally includes running linters/unit tests/etc).

Will it be a separate tool or an existing one?
The typical file watcher lib used in node tooling supports the native watch mechanisms on most operating systems, but can fall back to polling for mounts (for our friends developing with vagrant).

This comment has been minimized.

Copy link
@Igloczek

Igloczek May 25, 2019

Do you still plan to have some development mode auto processing of missing assets?

From PoV of a agency that is doing things own way and have dedicated skilled frontend devs - kill it with fire

From an imaginary possition of full stacks, backends and other people who might not know modern frontend tooling - "it doesnt't work after changes, just got a page without sytes and can't click anything" - at least that's what I see from questions asked over years about SASS Blank which requires to manually run some tools to get styles.

This comment has been minimized.

Copy link
@DrewML

DrewML May 28, 2019

Author Member

I think we need some solution here. Just like I don't want to setup docker + a bunch of microservices locally, most backend folks won't want to be running a persistent node process.

We'll find a good middle-ground here.


The performance of `bin/magento setup:static-content:deploy` is _not great_, which leads to developers being unhappy about doing front-end work.

It is also a very significant bottleneck for Magento Cloud deployments.

This comment has been minimized.

Copy link
@joni-jones

joni-jones May 24, 2019

Contributor

I think we need some benchmarks to compare different PL.

This comment has been minimized.

Copy link
@DrewML

DrewML May 24, 2019

Author Member

Do we? The document states multiple reasons for using node - perf is 1 of several. Interop with other node tooling and developer familiarity are others

This comment has been minimized.

Copy link
@DrewML

DrewML May 24, 2019

Author Member

Also want to verify you saw this section:


Accessible for Front-End Devs

When building something new, it's important to choose the right programming language for the job.

Often, the "right" language is based on the skillset of those that will be maintaining the tool. As an example, the Magento CLI is written using PHP. PHP is not a language designed with the intent of being used to write CLIs, but it is capable of it. Magento chose to write bin/magento in PHP because it makes contribution accessible to those using the tool.

A re-write in JavaScript means that the code for static content deployment can be maintained by front-end developers, who spend most of their time in Magento working on JavaScript/HTML/CSS.

This comment has been minimized.

Copy link
@jedmao

jedmao May 24, 2019

Contributor

In other words, benchmarks or not, this one point should not be the dealbreaker for this proposal.

This comment has been minimized.

Copy link
@joni-jones

joni-jones May 24, 2019

Contributor

Yes, I have seen this section. That's why I raised this question about benchmarks. I'm not forcing to use some specific language, my point is the usage applicable language especially if we are going to introduce a new stack dependency.

This comment has been minimized.

Copy link
@DrewML

DrewML May 24, 2019

Author Member

I'm not forcing to use some specific language, my point is the usage applicable language especially if we are going to introduce a new stack dependency.

Yep! Definitely agree, and glad you're bringing it up. If we were just shooting for raw performance here, I'd definitely agree we should just choose whichever language + runtime gives us the best running times.

But, one of the critical parts of the goal here is developer experience. Leveraging the ecosystem of common tools for front-end work will allow front-end devs in the Magento world to reapply their skills they've learned elsewhere, and get productive quicker.

1. Iterate, iterate, iterate

## Prior Art
- https://github.com/SnowdogApps/magento2-frontools/issues/25 Static Asset Processing (used by many Magento FE developers and agencies)

This comment has been minimized.

Copy link
@Igloczek

Igloczek May 25, 2019

馃槏

This comment has been minimized.

Copy link
@DrewML

DrewML May 28, 2019

Author Member

Y'all did good work! Deserve some credit :)

### What opportunities open up with a re-write of SCD using node.js?
- Compilation of modern JS with [Babel](https://babeljs.io/)
- Automatic live reloading of CSS during development
- [Autoprefixer](https://github.com/postcss/autoprefixer) support for CSS

This comment has been minimized.

Copy link
@Igloczek

Igloczek May 25, 2019

I think the whole PostCSS should be there and Autoprefixer as one of the options.

This comment has been minimized.

Copy link
@DrewML

DrewML May 28, 2019

Author Member

Definitely have no disagreements there. BUT, right now I'm focused 100% on minimal API surface area for feature parity. We need this thing to be correct + fast, then we can have fun!


### What opportunities open up with a re-write of SCD using node.js?
- Compilation of modern JS with [Babel](https://babeljs.io/)
- Automatic live reloading of CSS during development

This comment has been minimized.

Copy link
@Igloczek

Igloczek May 25, 2019

Do you still plan to have some development mode auto processing of missing assets?

From PoV of a agency that is doing things own way and have dedicated skilled frontend devs - kill it with fire

From an imaginary possition of full stacks, backends and other people who might not know modern frontend tooling - "it doesnt't work after changes, just got a page without sytes and can't click anything" - at least that's what I see from questions asked over years about SASS Blank which requires to manually run some tools to get styles.

@DrewML

This comment has been minimized.

Copy link
Member Author

commented Jun 12, 2019

I haven't seen any objections here, and I have a (so far promising) prototype going of this work locally, so I'm going to merge this and get it out of the queue.

Happy to iterate if folks have further feedback, but this is outside of core so it's low-risk 馃帀

@DrewML DrewML merged commit 4692024 into master Jun 12, 2019

1 check passed

licence/cla Contributor License Agreement is signed.
Details
@DrewML

This comment has been minimized.

Copy link
Member Author

commented Jul 19, 2019

Good progress being made - this work now lives here while it's still a POC

https://github.com/DrewML/scd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can鈥檛 perform that action at this time.