Hot module replacement

[zodern]

Hot module replacement (HMR) updates modified files in the app without having to reload the page or restart the app. This is usually faster (many times updating the page before the build has finished), and allows state to be preserved, though it is also less reliable. In situations where HMR is not supported, it uses hot code push to update the page.

HMR is only available for apps that use the modules package. It will probably not work correctly with apps that use globals (though globals from packages or npm dependencies are fine) since it uses import/require to detect which modules need to be re-evaluated.

This pull request only enables it for app code in the modern bundle. After this PR is done I will work on supporting other client architectures and packages.

A partial list of situations it falls back to hot code push:

• Files were modified that do not have a module id, are bare, are json data, or do not have meteorInstallOptions
• Modules were removed from the bundle
• A package was modified

Current status: It works well with many apps, but there are still many details that need to be handled.

Use:

This PR is included in the Meteor 2 betas: #11206

meteor update --release METEOR@2.0-beta.2
meteor add hot-module-replacement

Integration with View layers

React - Components are automatically configured to update using React Fast Refresh. Learn more about it in the React Native docs. React 16.9 or newer required. I haven't checked what happens with apps using an older version. The react runtime is included in production builds, though it is never evaluated. This should be fixed with tree shacking.

Svelte - Replace svelte:compiler with zodern:melte.

Hot API

This API is likely to completely change. It is currently based on the webpack hot api, with only the minimum implemented.

module.hot.accept() - The module will be rerun whenever it or any of its dependencies are modified

module.hot.decline() - If the module or any of its dependencies are modified, hot code push will be used instead of HMR.

module.hot.dispose((data) => {}) - Adds a callback to run before the module is replaced. Any state that should be preserved can be stored in data. The data will be available when the module is rerun at module.hot.data.

module.hot.data - If the module was reloaded, will have any data set by dispose handlers. If the file was not reloaded, or there were no dispose handlers, it will be null.

module.onRequire({
	before(module) { return {};  },
	after(module, data) {}
});

onRequire is only available when module.hot is. Use it to run code before and after each module. Both before and after are optional. Anything returned by the before function will be passed as the second argument to the after function.

These should be wrapped by

if (module.hot) {
  // Use module.hot here
}

Minifiers are able to remove the if statement for production, though none of the common minifiers for Meteor currently do.

How it works

1. The Meteor tool has a new runner - the HMR server. In development this is enabled automatically if the app or any of its packages uses hot-module-replacement. The HMR server provides a websocket server (on the same port as the proxy) that apps can connect to.
2. Compilers can detect if the hot api will be available for a file by calling file.hmrAvailable(). Even if it returns true, the file might not be updatable with HMR since all of the information needed isn't available at this step in the build process, but it does guarantee module.hot is set. If a compiler changes its output depending on if HMR is available, it should use it in its cache key.
3. After compilation and the import scanner and before linking the bundler calls a function in the HMR server which compares the in-progress build with the previous build to check if all changes can be updated through HMR and preparing the data needed for the update.
4. It immediately sends the changes to any clients even though the build hasn't finished (this is called an eager update). If this fails, the client will handle the failure after the build finishes
5. Once the build finishes and the server is restarted or autoupdate creates the new versions, the client will request any changes since the last non-eager update and handle any failures from step 4.

Updates are applied with this process:

1. It applies the changes to the modules. Added modules it installs, and changed modules it updates the content of and removes cached exports.
2. It checks if the modules are replaceable. It kept track of which modules imported which modules, and using this it makes sure all of the modules it was imported by accept updates, or that all of the modules those modules were imported by accept updates, and so on until either all paths reach a module that accepts updates, it reaches the main module ,or it reaches a module that declines updates.
3. For all of the modules between the ones that were modified up to the ones that accept updates it removes cached exports and marks them to be evaluated the next time they are required
4. It then requires all of the modules that accepted the update, causing them and the modules from step 3 to be re-evaluated

Remaining tasks

There are some larger details that should probably be discussed:

Hot API So far I have been copying webpack's api so I can focus on other aspects. Many other bundles have implemented their own api's, and to me some of them seem more intuitive. One of the next steps is to do more research and experiment with this, and I would appreciate any input from the community.

Integration with the meteor tool It currently uses an event emitter, which is a temporary solution. Originally I had considered creating an API for build plugins to handle HMR, but this part of HMR seems straight forward enough I am not sure if there is a reason for there to be more than one implementation. We also need to consider how packages will enable HMR for themselves, and for compilers to know if HMR is available for a unibuild. Allowing build plugins for HMR might help with both.

Communication with app Currently it uses a websocket server on a separate port. The Meteor tool already has an http server for the proxy, which could be extended to include a websocket server. A related feature request is at meteor/meteor-feature-requests#354. I think this can wait until later since we should probably review all of the auto reload packages at the same time. With this PR we will have 4, and there are other PR's open adding an additional package.

How much should be enabled by default Is there a reason to use hot-module-reload without React Fast Refresh?

Before beta:

• Exclude HMR client code from production builds
• Reload page when module is removed from client
• Add api for compilers to check if HMR is enabled and if a file is replaceable
• Updated modules should use same sourceURL as the module they replace
• Websocket should handle connection errors or disconnects
• Clear dispose handlers and _hotAccepts when a module is updated, though it should preserve _hotAccepts if the updated module fails
• Limit number of change sets stored in memory
• When adding files, group by meteorInstallOptions
• Compare all relevant details of files instead of only hash when checking for changes
• hot-module-replacement should not be a build plugin. All of that functionality should be moved into the meteor tool.
  • Enable HMR when the app directly uses hot-module-replacement
  • This will allow hot-module-replacement to be a dev only package so it is excluded from production builds
  • Add a websocket server to the proxy to send HMR updates to client. This will allow us to use the same port as the Meteor app.

Before stable:

• Move import tracking from install to hot-module-replacement package
• Clear old entries in importedBy whenever a module is updated
• It shouldn't disable linker cache
• Make React Fast Refresh optional
• Require client to send secret before sending changes over websocket
• Decide on and implement hot api specifically with Meteor bundler in mind
• Improve logging in client during HMR update
• Write Tests
• Documentation
• Make sure all details in How should we set up apps for HMR now that Fast Refresh replaces react-hot-loader? facebook/react#16604 (comment) are correctly handled, mainly the Nuances topic
• React Fast Refresh should debounce updates (it currently rerenders for each component that was modified)
• Handle changes outside the js and css bundles that require a full reload. These are normally handled by HCP, but HMR blocks it.
• Check how core packages handle HMR
• Support react components in typescript files

Later:

• Create HMR api to add listeners for update
• Remove use of install's root object.
• Submit PR to install to add necessary api's for HMR
• Submit PR to reify to allow removing cache entries
• Create simple package to show on screen when page updated (if a change didn't do what you expect or isn't a visible change, you might not realize when it applied the changes)
• Enable for cordova apps
• Look into including a timestamp for the build in bundle (without it, there might be a race condition)
• Enable for legacy bundle
• Enable for packages
• Look into ways this could help with faster server restarts
• Show reason when we do a full reload because the HMR was not possible
• Use meteorInstallOptions to control which modules have access to module.hot. Currently HMR is only used for apps or packages that directly depend on hot-module-replacement, but module.hot is available for everything.
• Remove if (module.hot) { ... } when minifying (requires Dead code elimination #11107)

[filipenevola]

Amazing work @zodern! Thank you!

Watch a short demo of this already working in one of my Meteor apps https://youtu.be/riroCNt9I98

I also have tested already in another one that is bigger and almost 3 years old and it is working there as well 😉

If you can please start using it and provide feedback here. I'm working closely with Zodern to get this released as soon as it is ready.

[pmogollons]

Awesome. I just started testing it in a large react project and it works perfectly without any config or change. Without hot-reload it takes almost 6s from change to reload. With hot-reload it takes around 3.5s - 3.7s.

Ill start using this for our dev env and give some feedback.

Thanks @zodern

[evolross]

HMR newbie here. Question - is this a development-only tool? How would this work in production on Galaxy? You redeploy your app and it doesn't need to refresh if HMR can handle all the changes/updates?

[pmogollons]

AFAIK is only a dev tool. It updates the components in place without reloading all the page. Shouldnt do anything on production.

[aogaili]

I've tested it and it seems to be working perfectly! great work @zodern

My only wish is that for the HMR to work also within packages.

[zodern]

@afrokick The next beta will disable react fast refresh if it detects you are using an older version of React, which should fix the issue with the dev tools. If react fast refresh is disabled, it won't automatically update react components with HMR. It requires React 16.9 or newer.

Apps that use an older version of React can use this in the file that mounts the root component:

if (module.hot) {
  module.hot.accept();
}

When the file re-runs, ReactDOM.render will detect the old react tree, unmount it, and render a new react tree using the updated components. It won't preserve component state, but before react fast refresh many apps used this for HMR since it works reliably.

[filipenevola]

Meteor 2.0-rc.0 is published.

You can use it now updating your app:

meteor update --release 2.0-rc.0

Or you can create a new app:

meteor create my-app --react --release 2.0-rc.0

Please provide feedback here.

[pmogollons]

Im having an issue where hmr is failing to reload a component with this error on the browser console. Its happening on the second update of the component, the first one works ok.

hot-module-replacement.js?hash=b99aaee973e26637b168141edcb2337028f027a8:348 Uncaught TypeError: Cannot read property 'forEach' of undefined
    at checkModuleAcceptsUpdate (hot-module-replacement.js?hash=b99aaee973e26637b168141edcb2337028f027a8:348)
    at hot-module-replacement.js?hash=b99aaee973e26637b168141edcb2337028f027a8:361
    at Set.forEach (<anonymous>)
    at checkModuleAcceptsUpdate (hot-module-replacement.js?hash=b99aaee973e26637b168141edcb2337028f027a8:348)
    at hot-module-replacement.js?hash=b99aaee973e26637b168141edcb2337028f027a8:471
    at Array.forEach (<anonymous>)
    at applyChangeset (hot-module-replacement.js?hash=b99aaee973e26637b168141edcb2337028f027a8:464)
    at hot-module-replacement.js?hash=b99aaee973e26637b168141edcb2337028f027a8:183
    at Array.every (<anonymous>)
    at handleMessage (hot-module-replacement.js?hash=b99aaee973e26637b168141edcb2337028f027a8:182)

This is the component

import React from 'react';
import styled from 'styled-components';
import { withQuery } from 'meteor/cultofcoders:grapher-react';

import { assemblyMessagesLiveQuery } from '../../../../api/assemblyMessages/queries';

import Icon from '../Icon';
import ErrorBoundary from '../ErrorBoundary';


function ChatTab(props) {
  return (
    <ErrorBoundary content={ true }>
      {
        props.data.map((message) =>
          <div key={ message._id }>
            <h4>{ message.user.fullName }</h4>

            <div>{ message.createdAt }</div>

            <p>{ message.text }</p>
          </div>
        )
      }

      <MessageInput>
        <textarea rows={ 4 }/>

        <button>
          <Icon
            name="send"/>
        </button>
      </MessageInput>
    </ErrorBoundary>
  );
}

const MessageInput = styled.div`
  position: absolute;
  bottom: 0;
  left: 0;
  width: 100%;
  padding: 16px 16px 80px;
  min-height: 160px;
  background-color: ${({ theme }) => theme.colors.bg};
  
  textarea {
    width: 100%;
    color: ${({ theme }) => theme.colors.text};
    padding: 8px 16px;
    border-radius: 8px;
    background-color: ${({ theme }) => theme.colors.bgLight};
  }
  
  button {
    position: absolute;
    bottom: 92px;
    right: 24px;
    color: ${({ theme }) => theme.colors.green};
    border: none;
    font-size: 24px;
    background: none;
  }
`;


export default withQuery(({ assemblyId }) => {
  return assemblyMessagesLiveQuery.clone({
    skip: 0,
    limit: 200,
    assemblyId
  });
})(ChatTab);

[rijk]

I have the same issue, after updating a hook (not a component).

[nicubarbaros]

It still fails with lazy imports in React.

hot-module-replacement@0.2.0-rc200.
react 17.0.1

[nicubarbaros]

😢

[zodern]

@nicubarbaros could you please create a reproduction? React fast refresh accesses all of the exports from modules, and it looks like one of the exports is a reference to an iframe or something else that React Fast Refresh needs to be more careful with.

[filipenevola]

😢

Don't worry, we are going to continue working on HMR, fixing bugs and improving it as a whole.

The merge means that we are ready to have it in a new Meteor final release.

[rijk]

Further debugging this error:

Uncaught TypeError: Cannot read property 'forEach' of undefined

I found out the moduleId when it happens is /node_modules/meteor/hot-module-replacement/client.js. So presumably this is fixed with 59754c4.

[filipenevola]

I found out the moduleId when it happens is /node_modules/meteor/hot-module-replacement/client.js. So presumably this is fixed with 59754c4.

Hi @rijk, did you try Meteor 2.0-rc.3? This commit should be included there.

[rijk]

Sorry, had no idea we were at 3 already 😄

[filipenevola]

Always moving!

[nicubarbaros]

I tracked it down to these fellows that are creating the problem mentioned above.

Video.js & emoji-mart.

I will try to create a reproduction.

[pmogollons]

I can confirm that the issue I reported previously is now fixed in final release.

[rijk]

@filipenevola Is this still the right place to report issues with HMR?

[filipenevola]

Hi @rijk no. It's not.

You should open new issues then we can track them individually.

Please provide reproductions, it's very hard to reproduce HMR problems when we don't have the code as modules can interact in different ways. Of course, if a reproduction is hard to be provided explain why 😉

Thanks!