WIP: Greenfield web app

[akotlar]

Let's build it from scratch, but better, faster, ...

Philosophy: Minimal magic, minimal reliance on outside work, don't use it unless we understand it.

Goal: <16ms interactions, including <16ms page transitions. Should feel identical to a desktop app in terms of performance, but maintain state like a website (i.e get variables).

TODO:

• Profile/logout should be responsive: no user icon / dropdown until narrow view
• Default to redirect rather than popup
• Clicking on login should clear state if auth failed
• Write test for token verification on backend
• Add profile page
• Finish auth/redirect notebook logic in gateway
• Add notebook state endpoints in gateway
• Add notebook state view in frontend
• Break this up into ~10 commits, targeting <= 200 LOC each (with first commit being checking in package-lock.json)
• Deal with cross-origin tracking issues in Safari. This may require using the "custom domains" feature of auth0, paid. Workaround could be to poll/websocket request to api server to refresh tokens.

To run:

cd packages/web-client
docker build . -t blah
docker run --env-file=env-example -p 3000:3000 blah  npm run start

then navigate to http://localhost:3000

# lines: Most come from the package.json.lock files. These maintain versioning information.

• It is recommended to check in .lock files
• They're huge, sorry.

Documentation

JS

https://javascript.info

We use the subset termed ES2018.

Compatibility across all browsers is ensured by transpilation using BabelJS, to some lower JS target. Polyfills should not be used, except when impossible to support a browser (this is configurable).

I mostly don't care about anything that isn't an evergreen browser, so I think we should support: Edge, Safari, Chrome, Firefox.

Among those we may want to care about, IE11 has ~2% global use (more if only desktop browsers)

NodeJS

We use 10.15. This is the latest LTS release

Versioning / dependency management

TL;DR: npm

npm init # creates a package.json file, which tracks dependencies
npm install next react react-dom # install 3 packages and save them to the dependencies property

package.json

The file that tracks dependencies, and their semantic versioning numbers

Shape:

{
  "name": "hail-web-client",
  "version": "0.2.0",
  "scripts": {
    "dev": "next",
    "build": "next build",
    "start": "NODE_ENV=production SSL=true next start"
  },
  "author": "Hail Team",
  "license": "MIT",
  "dependencies": {
    "next": "^7.0.2-canary.50",
    "react": "^16.7.0",
    "react-dom": "^16.7.0"
  },
  "devDependencies": {
  }
}

• Scripts are thing that can be run by typing, in shell npm run. Ex: npm run dev

Async, Await, Promises and callback (WIP)

Javascript is async-first. This is most obvious in Node.js, which is the most popular library for server-side JS.

• How event loop works

• The event loop call stack: https://www.youtube.com/watch?v=8aGhZQkoFbQ

At a high level, a function that defines a callback will return immediately. The callback is pushed on to the event-loop stack, and on each tick, is checked to determine whether it has returned or not.

Blocking operations within the callbacks will block the event loop. This is how CPU viruses, like blockchain manage to slow down web pages that are hijacked to include some mining script: hashing something 30 million times, takes a long time, and JS cannot do anything besides waiting for those operations to finish in a synchronous fashion.

Luckily, asynchronous functions are the norm in the JS ecosystem, such that both in the browser, and nodejs, IO functions are (mostly?) asynchronous.

• For NodeJS: Transparently to the user, blocking operations (IO) are executed from kernel threads that Node maintains in the background, effectively making these operations non-blocking (until the thread pool is exhausted).

Browsers and NodeJS use different event loops:

NodeJS: libuv event loop

• Node maintains a hidden worker thread pool (kernel threads) through which it issues sys calls, to avoid blocking the event loop.

Web: depends on the underlying Javascript Engine

• Chromium: V8: libevent: https://stackoverflow.com/questions/25750884/are-there-significant-differences-between-the-chrome-browser-event-loop-versus-t
• Firefox: Spidermonkey: ?
• https://developer.mozilla.org/en-US/docs/Web/JavaScript/EventLoop#Event_loop

Using callbacks

# Callback-based
function asyncCall(arg, cb => {
  const (err, result) = someSynchronousOperation();

  cb(err, result);
}

asyncCall(arg,(r, err) => { if(err){ throw new Error(err); doSomething(r)} )

Using async/await

Deeply nested callbacks are hard to follow. This is called "callback hell". To help combat this, JS, in both NodeJS and Web context, developed Promises. Promises flatten the callback tree.

function asyncPromise(arg) {
  return new Promise((resolve, reject) => {
     const (err, result) = someSynchronousOperation();
    
     if(err) {
        reject(err);
        return;
     }    
     
     resolve(result);
  });
}

asyncPromise(arg).then( r => doSomething(r) ).catch( err => throw new Error(err) )

This has one problem. Chaining promises leads to a potentially hard to follow chain of .then .catch. As in many other languages, the solution to "transforming" async call syntax to sync ones, is to color async functions with a "async" and "await" clauses.

This can be used with any functions that return promises (but not those that just return a callback). Luckily again, JS libraries have been moving towards the Promise-land (sorry) for ~5 years, before Promises were in stdlib (bluebird).

async function usePromise() {
  const arg = someSyncOperation();
  
  let result;
  try {
    result = await asyncPromise(arg);
  } catch(e) {
    // without wrapping catch, will just throw on reject(), unwinding the call stack
    doSomethingWIthError(e) 
  }
  
  doStuffWithResult(result);
}

React

What is a react component? A function that returns JSX.

React components accept props (HTML attributes <Component propName={propValue} />)
Stateless vs stateful components

# Stateful
class Stuff extends React.Component {
  static getInitialProps() {
  
  }

 render() {
  return <div>Hello World</div>
 } 
}

# Stateless
# Note that arrow syntax has an implicit return if you don't create a function block, i. => { return <div>Hello World</div> } is valid too.
() => <div> Hello World </div> 

Stateless ones are typically cheaper, but not necessarily:

• See: PureComponent
• See: shouldComponentUpdate lifecycle method

JSX differences from html

1. className : "class" is a reserved word in JSX; used to specify the component class (every HTML element is modeled as an object). This will go away in 2019, maybe
2. There should alway be one and only one non-leaf node in the component tree. Leaf siblings are allowed; note that this requires that there is only one root node as well.

export default () => <div>OK</div>
export default () => (<div>OK</div><span>Not OK</span>)

To get around this, wrap in a "", an html element, or array. Basically, react wants a top level / root object. https://reactjs.org/docs/fragments.html, https://pawelgrzybek.com/return-multiple-elements-from-a-component-with-react-16/

import { Fragment } from 'react';

const good = () => [<div>OK</div><span>GOOD!</span>];
const good2 =  () => <Fragment><div>OK</div><span>GOOD!</span></Fragment>;
const good3 = () => <span><div>OK</div><span>GOOD!</span></span>;

JSX naming conventions

1. Lowercase components are just built-in html elements. i.e <span> is a an HTML <span> on output.
2. Uppercase components are javascript functions. This makes composing components really simple.

const CoolComponent = () => <span>Hello World</span>;

export default () => <CoolComponent>;

You can pass state to these user-defined components, much like you would in HTML, using attributes. These attributes can have arbitrary names, except they must start with a lowercase letter, and follow camel-case convention. These attributes are called props

const CoolComponent = (props) => <span>Hello {props.name}!</span>;

export default () => <CoolComponent name="Alex">; #mounts <span>Hello Alex!</span> in DOM

PureComponent / shallow watch

React's reconciler is triggered whenever this.setState is called, resulting in a walk down the descendent nodes, based on either the presence of that state variable as a "prop" (i.e <MyComponent name={this.state.name}/>), or its use directly within the component (i.e `{this.state.name === 'Alex' ?

Do stuff

:

Do other stuff

).

To give the reconciler less work to do, when accepting objects as props, use a <PureComponent>. This will tell React to check the reference for diff, rather than deep value compare. Obviously much faster to do the latter.

You can do even better than PureComponent. Use a regular Component, and specific a shouldComponentUpdate() { } method in that component. Within that method, write whatever checks needed, so that when a prop, or state changes, you return true, otherwise false. When true, the component will re-render. However, this allows you to react in a more fine-grained way, i.e instead of checking reference, check for the update of a specific property, or don't react to that object changing at all.

Behind the scenes, PureComponent is in effect implementing a shouldComponentUpdate that checks reference equality (prevProp !== currentProp).

References:

1. https://reactjs.org/docs/react-api.html#reactpurecomponent

Memo stateless components

Stateless/functional components (i.e those than don't extend React.Component or React.PureComponent, i.e (props) => <div>Hello {props.name}</div>), can be memoized. As in a typical memoized function, given one set of input (props), the result is cached, and the cached result is returned for n + 1 calls.

References

1. https://reactjs.org/docs/react-api.html#reactmemo
2. https://scotch.io/tutorials/react-166-reactmemo-for-functional-components-rendering-control

Typescript

2. https://reactjs.org/docs/react-api.html#reactmemo

And React Component prop definitions

https://levelup.gitconnected.com/ultimate-react-component-patterns-with-typescript-2-8-82990c516935

NextJS

https://nextjs.org/docs/
Next has 4 deviations from normal react:

1. _app.js: Can be omitted. Wraps all other components. Is useful for global functions, because it is not reloaded when you change pages. Good place to place a header component, a footer, global data stores, or handle page transitions.
   it has this shape:

<Container>
          <Header />
          <Component {...pageProps} />
          <Footer />
</Container>

2. _document.js: Optional. Rendered only on the server, exactly one time. Wraps _app. Good place to define external resource you want to load, such as some external stylesheet, font, whatever.

3. getInitialProps: a lifecycle method that is only available to components in the pages/ folder. getInitialProps runs once during server-side rendering, and again if you navigate to the page that defines it. Only components in pages can specify this property. This is because it is effectively a function triggered during routing and:

• getInitialProps is of course only available if you define a stateful component. See functional components (just JSX wrapped in a function, rather than a class)

4. NextJS includes a light, fast router. Routes are matched based on the names of files in pages/, with index.js mapping to /

For instance, to navigate to domain.com/scorecard/users, you'd make the folder structure:

pages/

• scorecard.tsx
• scorecard/
  • users.tsx

These 'pages' components are just like normal react components, except they expose getInitialProps, described above. Each page file must export 1 default component:

#Page file
import React from 'react';

const index = () => <div>Hello World</div>
export default index;

There is nothing else to do to get routing to work, a quite nice solution.

JS pragma

1. this is different than in most (every?) other language. scope of this is bound to caller, not object containing the method

• Solution: use arrow functions

class Something {
  constructor() {
   this.bar = 'foo';
 } 
  //Do
  onSubmit = () =>  {
     console.log(this.bar) //prints foo
  }

  // Don't
  onSubmitBad() {
   console.log(this.bar) //may be undefined
  }
}

const barrer = new Something();
console.info("good", barrer.onSubmit());
console.info("bad", barrer.onSubmitBad());

Tips

Client-side routing

Wrap a normal anchor tag in <Link ></Link>
ex:

<Link href='/path/to/page'><a>Page Name</a></Link>

This simply adds the client-side routing logic, and passes the href to <a href=.

Prefetching

One of the neat things about Next is how easy it makes prefetching pages. This allows perceived page loading times on the order of 5ms, even when the page requires very complex state (say a GraphQL or series of REST calls with large responses).

<Link href='/expensive-page' prefetch><a>Expensive Page</a></Link>

Make your app do ONLY server-side routing

Meaning every time you click on a link in your page, you hit the server, just like the first visited page.

Simply use <a> directly.

Caching and sidecar requests

Broadly, there are three strategies: browser caching, server caching, and service-worker caching.

In this project we will likely use all three. Server caching is an excellent strategy for pages that serve only public data. In this strategy we pre-generate the static html, serve that, and invalidate the cache once in a while. An example of this can be found in e131a93

• Care needs to be taken with the server-side option, not to leak authentication state, since this will, at least by default, be shared across all users.

Styleguide

1. Typescript everywhere

Performance

1. React SSR vs Nunjucks

• React SSR performance (well, React DOM in general) is a focus for 2019
  

2. React vs VanillaJS. Depends on what you measure, it's either 50% slower or many times faster.

• https://github.com/krausest/js-framework-benchmark
  • React authors claim this is an unrealistic environment, and that their scheduler is tuned to provide smooth/non-hitching UI interactions, at some cost to the speed with which 100,000 elements can be appended to a page.
• Some consider this to be more reliable: https://localvoid.github.io/uibench/
  • Here React performs many times better than vanilla JS for some operations.
• I should probably figure out exactly why

In practice, React in 2019 will likely be the best performing UI solution available, with of course the exception of some very well optimized JS. This is because of React Fiber's time slice mode, which will effectively allow UI operations, like user input, to preempt other operations

How React works

React Fiber, the new reconciling/scheduling algorithm: https://github.com/acdlite/react-fiber-architecture

[cseed]

Thanks, Alex! Look forward to digging into all of this.

[tpoterba]

for future pull requests, can you open a PR from your forked repository? We like to keep the main repo free from non-released branches.

[akotlar]

Yep

[akotlar]

Because of the Webpack config, auth can be imported multiple times across components, while also being included only once across the final collection of bundles.

The current Webpack config handles the mapping from components to flat bundles quite well.

[akotlar]

@cseed As I briefly mentioned in an email on Saturday, I moved notebook loading to a prefetched model. Data and page loading are decoupled: When one clicks on the Notebook link, routing to the page happens without waiting for data to come in from notebook-api.hail.is. If data is available, it is shown, else a loading indicator.

To avoid loading screens, I call notebook-api.hail.is asynchronously immediately after a user hits app.hail.is, whether they're on the Notebook page or not. Therefore, typically a user will never see a Notebook page loading indicator, when they click from one page to Notebook (say they land on the home page: by the time they click on Notebook, the data is fetched, so no loading screen).

Furthermore, web sockets keep that Notebook data up to date, again regardless of whether a user is on the Notebook page.

So in all instances except when a user lands on Notebook directly, the time it takes to reach the Notebook page is << 16ms (seemingly ~3ms).

A similar approach is now used for scorecard, except Scorecard will render data in the SSR phase (because I consider scorecard less important, and because most users of the web app won't need it), and will not prefetch data when on another page. However, clicking on "Scorecard" from another page will not cause routing to block while waiting for the https://scorecard.hail.is/json response; instead a loading indicator will be shown. In practice I find this preferable, because the GUI feels more responsive, and users get more feedback.

This demonstrates the core benefit of universal rendering approaches, and how they can improve page loading times over even the leanest server-side rendered application.