Moving towards a new Frontend Builder

[usirin]

Hello,

Following is the builder's requirements to work. I tried my best to describe each step in words, so that we can discuss and come up with better ideas, and mostly because if we know the requirements of it right now we can move to a new one with confidence and ditch bant once and for all.

/cc @koding/frontend

---

• Build Sprites
• Build Styles
• Build Scripts
• Copy Assets
• Copy Third Party

Build Sprites

Sprite Specification

• Each app declares a sprites array in their bant.json. (the default is lib/sprites for each app right now.)

// bant.json
{
  "sprites": [
    "./lib/sprites"
  ]
}

• Each entry in sprites array consists of 2 folders:
  • lib/sprites/1x
  • lib/sprites/2x
• For each file in 1x folder, there should be another file present in the 2x folder with the same name:

# if these two doesn't present at the same time, builder should throw.
lib/sprites/1x/foo.png
lib/sprites/2x/foo.png

Sprite build step

• Builder goes through each app's sprites folders that has been defined in the sprites array in bant.json
• Creates a 1x css spritesheet and saves it under opts.spriteTmpCssOutdir
• Creates a 2x css spritesheet and saves it under opts.spriteTmpCssOutdir
• Creates a 1x sprite image and saves it under opts.spriteImgOutdir
• Creates a 2x sprite image and saves it under opts.spriteImgOutdir
• It watches each sprites folders of each app, and rebuilds sprites for each change event.

Build Styles

Style Specification

• Each app declares a styles array in their bant.json

// bant.json
{
  "styles": [
    "./lib/styl/*.styl",
    "./lib/components/**/styl/*.styl"
  ],
}

• Each entry in styles array can be a glob relative to the root folder of app folder.
• Each entry in styles array corresponds to a file or list of files that can be compiled via stylus compiler.

Style build step

• Copies over KD Framework css file.
• Includes STYLES_COMMONS_GLOB ('app/styl/**/*.styl') (for each app :/)
• Goes through each app's style files that has been defined in the styles array of bant.json
• Creates a single css file for each app under opts.stylesOutdir/<app-name>.css
• If builder is running watch mode for styles:
  • It watches files under STYLES_COMMONS_GLOB
  • When a change event happens with globals, it rebuilds all styles.
  • It watches each file that is described in the styles array of bant.json of each app.
  • When a change event happens under single file, it rebuilds individual app style.
• If builder requires sprites compilation:
  • It looks for each spritesheet file under opts.spriteTmpCssOutdir
  • Includes each spritesheet file into compiled css.

Build Scripts files

• Each app declares a main key which corresponds to a file that is relative to the app root folder.

// bant.json
{
  "main": "lib/index.coffee"
}

• The file in main key will be used as an entry file for bant compilation.

Scripts build step

• It provides a way to require each app with its name with absolute require paths.

# We have 2 files
# client/activity/lib/foo.coffee (A) file under 'activity' app.
# client/ide/lib/bar.coffee (B) file under 'ide' app.

# (B) can require (A) by using absolute paths, and vica-versa.

# client/ide/lib/bar.coffee
# no need to do require '../../../activity/lib/foo'
foo = require 'activity/foo'

• It creates a rewrite map that removes the necessity of using lib for requiring single modules under <app-name>/lib folder.

# no need to write down `require 'activity/lib/foo'`
foo = require 'activity/foo'

• It compiles CoffeeScript.
• It imports CoffeeScript files without the necessity of .coffee extension.

# no need to require like: require './foo.coffee'
foo = require './foo'

• It compiles JSX.
• It compiles Pistachio.
• It minifies output JavaScript file if opts.minifyJs
• It provides below globals to the browser:
  • All app modules under globals.modules
  • Remote api schema under globals.REMOTE_API
  • Ace library path under globals.acePath
  • Ace library config under globals.aceConfig
• It dedupes the kd.js modules. (wtf?)
• It extracts sourcemaps if opts.extractJsSourcemaps option is true.
• If watch mode is enabled:
  • It watches over each required file starting with entry files of the each app.
  • When a change happens in the dependency tree, it rebundles.

Copy Assets

Copy assets builder step

• Copies assets under opts.assetsDir to opts.assetsOutdir with the same folder structure.

Copy Thirdparty

Copy thirdparty builder step.

• Copies files and folders under opts.thirdpartyDir to opts.thirdpartyOutdir with the same folder structure.

---

Here are my suggestions:

1. Remove bant, use webpack

• No maintenance cost for our "own" builder.
• webpack is widely supported by the developers from big companies: facebook, twitter, etc.
• webpack supports awesome debugging tools out of box, hot-reloading
• webpack is the defacto builder in React community.
• Widely support plugins, loaders.
• Loaders for bundling images, style files, etc.
• It automatically supports creating optimized chunks out of builds, and loads them asynchronously.

If we switch to webpack requiring a style file for a component will become like this:

require './fooview.styl'

• This way we can tell webpack to create chunks, and only serve the required js and css files on initial load, and the loading the rest can be deferred.

2. Also @cihangir had some ideas about about uploading sprites to a cdn, and remove that task out of builder as well, I am supporting it but i don't have any clue how it should be implemented, any idea is appreciated here.

Any feedback/help will make our lives as frontend engineers way much easier, cause waiting for 284712894 seconds for a simple js file change shouldn't be a normal practice. 👻

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[szkl]

Generating a sprite image file is a build task.
Uploading sprites to a CDN is a deployment task.

It doesn't add up on my mind how a deployment task can eliminate a build task regarding sprite image file.

[szkl]

Also, development iteration of sprite editing should be considered before going into it.

[usirin]

Also, development iteration of sprite editing should be considered before going into it.

Can you elaborate?

[szkl]

If builder is not going to generate a sprite image then what developer is supposed to do?

[usirin]

That was exactly the thing i was trying to come up with a solution. But have literally no idea. I guess sprite generating still needs to be the part of the builder, but builder should be smart enough to not recompile when building first if there are no change in the sprites.

[szkl]

That's a completely different matter :)

[usirin]

What would be the best solution for sprites? Because clearly it's one of the bottlenecks at the initial build and i think we need to address that.

[szkl]

webpack is a good idea maintenance wise but a daunting task. Finer details of corresponding weback steps should be designated first before Project I.G.I.

[usirin]

Finer details of corresponding weback steps should be designated first before Project I.G.I.

This is exactly the reason of me creating this issue, let's come up with the steps all together towards our need.

[szkl]

I still think we can investigate slowness regarding sprite generation and JavaScript watcher. It should take less than webpack replacement 💥

[szkl]

Initial proof of concept goal should be an optimal {Coffee,Java}Script file watcher and sprite image generation process.

[usirin]

I agree with you, we should first investigate those and fix those problems, since sprite generation will require extra attention regardless of us changing builder.

I also think that we should have this issue open to figure out what the ultimate needs of ours are to have a better/faster/stronger builder. For example, having a hot-reloading mechanism would really reduce the development time spent.

[fatihacet]

2. Also @cihangir had some ideas about about uploading sprites to a cdn, and remove that task out of builder as well, I am supporting it but i don't have any clue how it should be implemented, any idea is appreciated here.

Here is an idea, let's put already created sprite images to s3 and download them to dev's machine while building the client and use it instead building the sprites again.

Here is the flow of the idea above,

• You woke up and pulled the master, latest commit id was 1111 on master before you go sleep and after the pull it is 2222.
• You run make development on client folder as regular. Builder start working and it will get the latest commit id and it will check whether 2222 exist in your website folder.
• If not it will download the folder from s3.
• If yes, in other word you have to sprite images for latest commit id, builder won't need to do anything.

Let's say you need to add new sprite image then you will need to generate the sprites again as usual and as is right now then your 2222 folder content will be changed with the sprites you created manually.

This is just an idea how we can utilize remote sprite images but this may work or not and/or contain some pitfalls, since it's an idea in my mind and I wanted to add it here.

bye.

[sinan]

#6208

[usirin]

#6208 was not working correctly, i made lots of changes which is living in my local, will send it when i have time.

[gokmen]

I've lots of things to say for this one but I'm unable to have time to list them, it's my bad. But let me start from somewhere;

• Why are we keeping sprites in individual folders? Can't we simply put them in same place? Since we need to use them from each other.
  • This will let us to create a sprite package easily and we won't need to build them all the time like mentioned in the proposal.
• I'm ok with webpack, but we need to make sure if it fits with our requirements https://webpack.github.io/docs/comparison.html we can probably go alone with browserify as well, but I'm ok as I said.
• We can add integration for browsersync for hot-reload
• It's not the exactly same thing but we can create a simple builder like I did with kd-generator https://github.com/gokmen/kodemirror it's using kd.js and codemirror for example with a small requirement set which provides all the functionality we need.

[sinan]

browserify was fine when we first did it, but then to make everything work (globals, remote, exposing _kd etc) tetsuo have done some dirty patches to it, and he did Haydar. On top of it we had to bring bant and all those Makefiles. Those patches, extras, and bant is something we don't want to maintain. Also we had react in question when we started this discussion. Webpack provided all those by default, but just not sprites.

Their solution to sprites is different, you don't create spritesheets but it checks the images by size and if smaller than 8kb (adjustable) it embeds into the compiled css as base64 if bigger it puts it to the dist folder of your choice and links the background-image to it (paths are also adjustable). IMO this is a clever solution than sprites as it created weird problems on retina displays and it was always an hassle during deployments.

I like your https://github.com/gokmen/kodemirror/blob/master/gulpfile.coffee if you see our landing build it is more or less the same thing. But when things get bigger that simple approach may become messy.

As a last note re: Hot-reload, yes that's an ongoing debate on which is nicer, browsersync vs webpack's. Webpack came with it from the beginning then browserify people did a similar thing. I am not going to take a side on that because I think it's less important than the issues above.

[gokmen]

as an update http://blog.tighten.co/unpacking-webpack just see this, fyi.