New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Full static generated mode #22

Open
manniL opened this Issue Dec 23, 2018 · 21 comments

Comments

Projects
None yet
10 participants
@manniL
Copy link
Member

manniL commented Dec 23, 2018

Current state

Nuxt's Static Site Generator (nuxt generate) is growing! I love the static mode when it comes to portfolio pages (or generally, pages that don't include a lot of dynamic data).

Problem

Usually, you use statically generated pages together with a Headless CMS or another external API.

Currently, you can generate the HTML (with static + universal mode) but asyncData calls are still made during client-side navigation, which means that an external API will likely be called on such a static page.

While it's worth here and there to make these calls even after static generation, it's absolutely not needed (from my POV) for the majority of the pages. Instead, the author/developer could simply issue another build (eg. through Netlify) to update the content.

Also you might encounter different content as asyncData is not called on the entry route of you static app. (Going from /b to /a can lead to different content than directly accessing /a)

Proposal

As announced by @Atinux on Vue Toronto (see his talk at 26:18), I want to propose the full static generation option for nuxt generate.

Instead of relying on the API calls, we should inline the response of the asyncData method(s) inside a .json file (per page) when the generate.fullStatic flag is enabled in the nuxt.config.js (the flag name is debatable). Then we can use the .json file as data source without issues.

Related issues

nuxt/nuxt.js#4607

3rd party modules that apply this approach

https://github.com/DreaMinder/nuxt-payload-extractor
https://github.com/stursby/nuxt-static/

@manniL manniL added the nuxt label Dec 23, 2018

@YamenSharaf

This comment has been minimized.

Copy link

YamenSharaf commented Dec 24, 2018

As someone who's working to create a fully static site, this has been an issue for me as it looks like there's no way currently to have a fully static site with a headless CMS.
Nuxt has a lot of tools that help with that, but not quite in a way that ties them together to create something as fast as a static site should be.

These are the options I've stumbled across so far:

Relying on asyncData in page components

Pros

  • This is the natural /encouraged approach by Nuxt docs
  • Each page pulls in its own static content. Refreshing the page yeilds static content without extra AJAX requests.

Cons

  • When navigating through nuxt-link to pages with async-data function defined, an AJAX call is fired before each page navigation to, again, load the data that is already statically on the page in a needless fashion. I'm no expert, but I guess that entails also shipping the vue engine to bind data coming from the server (also needlessly).

Relying on payload

Pros

  • Nothing I can think of really, since the earlier approach seemed to work fine for pulling data for individual pages.

Cons

  • More complex to implement.
  • When navigating through nuxt-link, payload property is always undefined, which means I would have to have an asyncData method that would fire anyway on each page navigation.

Utilizing the vuex store and nuxtServerInit to get needed data

Pros

  • No AJAX call is triggered when pulling data from the store (seems fair), even when navigating with nuxt-link.

Cons

  • Store is basically global and not confined, which means that pulling data upfront. eg, posts, will cause them to load on each page even if not needed, but you do get faster navigation afterward.

Conclusion

I would have had rather be a straightforward way to pull data from an API without the overhead of extra AJAX requests or loading unneeded data.

@pi0

This comment has been minimized.

Copy link
Member

pi0 commented Dec 24, 2018

Nice writeup @manniL.

For more specific terminology let's call this behavior/mode static and the offline data source files cache data.

Adding some more important notes:

  • We should consider we have fetch, asyncData, nuxtServerInit and SSR plugins that all should be addressed also can cause side effects like triggering store dispatches that they can also fetch/process data.
  • Cache items have no one-to-one relation with pages or route entries. Wildcard routes and query parameters require different responses. (edited: this is a rare case as we generate one HTML per each)

Regarding implementation and future opportunities I can think of 3 ways:

1. Memorize asyncData

This way, during generating we can spy result of asyncData during SSR generate and store it somewhere like .nuxt/dist/cache. Then by modification of asyncData implementation for client-side instead of the original method we import corresponding cache item. This also addresses the need for duplicate HTTP requests.

Pros

Most straight-forward approach.

Cons

We can't guarantee everything is cached. fetch and store may have their own logic.

2. Cache window.__NUXT__

This way, during generate we cache nuxt states into .json files and like method (1), change the behavior of client-side for fetch and asyncData

Pros

Straight-forward and covers store state too.

Cons

  • The possibility of breaking change for exciting projects.
  • How to handle non pojos?

3. Cache network requests

This way was the one that me and @Atinux talking about it almost one year ago for static mode. Instead of internal nuxt hacks, we can find a way to spy on network requests and statically cache them somewhere like .nuxt/dist/cache/[hash].json and instead of doing actual requests we can import json chunks.

Cons

  • Spy part is delegated to official nuxt modules. (However, if changes required to the core, we make them)

Pros

  • Updating API cache is independent of nuxt generate. We can update it without rebuilds
  • We have full control over each ajax/http call to be cached. Not caching a black-box
  • No internal hacks and breaking changes
  • Transparent caching means making it optional without the need to changes of fetch/asyncData and store actions for offline support
  • More service-worker spec friendly for caching and PWA like approach.
  • Works with other modes (@manniL)
@pi0

This comment has been minimized.

Copy link
Member

pi0 commented Dec 24, 2018

After talking with @Atinux, both (2) and (3) options seem good approaches. Most of the static generated websites do not need complex logic and simply storing NUXT object will be enough but also network level caching has it's own benefits too.

The suggested approach is introducing a new $cache API to the nuxt core which is responsible for storing and loading cache entries. This API may be used directly by users or module like axios. The cache layer may leverage a cache adapter too.

I do believe we can include a really basic default implementation to the core. Adapters should be probably different for server and client (cache.client.js and cache.server.js. Like nuxt/nuxt.js#4574 which made by Sebastien for nuxt-link)

For page level caching during generating full static version, we can add a cache property to the pages or global config (Like transition) that can enable/customize page level caching and probably globally enabled when using nuxt generate --full-static. We can then store ssrContext.nuxt.data to a JSON file that will be loaded instead of calling asyncData and fetch on the client side. This should be also clearly documented that those functions are no longer being called on full static mode.

For changing client-side behavior we can use lodash template to replace default asyncData/fetch calls with $cache.get($route) calls (vue-app/template/client.js)
For client-side support we can add logics to read cache option from page or global and use $cache.get($route) instead of calling functions. (vue-app/template/client.js)

@Atinux

This comment has been minimized.

Copy link
Member

Atinux commented Dec 25, 2018

I strongly validate @pi0 comment.

This caching system could be used by other modules and inside services too in the future.

Nuxt is build on modules, we simply need to have @nuxt/cache now.

The thing I want to work on is to give more possibilities to modules to customize the Nuxt logic of client-side. This could be done by extending defaults components, or having some hooks system. So Nuxt plugins could also interact more the client & server logic of the default Nuxt logic (middleware, async data, transitions, store, etc)

@manniL

This comment has been minimized.

Copy link
Member Author

manniL commented Jan 4, 2019

One aspect we shouldn't forget: Making a page "full static" by actually removing the API calls from the code also means that the underlying API is harder to attack (via DDoS, exploits, ...) as a possible attacker doesn't know where the target API lives.

Of course, Security through Obscurity is no sophisticated security concept at all.

@pimlie

This comment has been minimized.

Copy link

pimlie commented Jan 5, 2019

Could we take this rfc maybe a step further and also include an option to build an all inclusive, single (static) html file? There are times this would be useful, e.g. when you create a demo of something so you can just email a single html file to a client or upload multiple demos to your website without the need for subfolders etc. But also an Electron application could benefit from this when all html/js/cs is loaded at application start-up, or a better example if you want to put a single html file on a usb-stick to hand-out as a promotion.

E.g. in the past I've used a grunt script to achieve this (albeit for a non-Nuxt project). As combining all the css/js/font files resulted in a single 1.3MB html file, I embedded all files zipped & then base64 encoded which reduced the file size to 550KB. (Of course the unzip javascript code and the js for the loading dialog was not zipped).

@pi0

This comment has been minimized.

Copy link
Member

pi0 commented Jan 5, 2019

@pimlie Nice idea. We may use MHTML format to encapsulate all resources of each page. And it is easier to compress.

@pimlie

This comment has been minimized.

Copy link

pimlie commented Jan 5, 2019

Although mhtml would in theory be better, it lacks the full support which plain html file has (eg Firefox doesnt support it by default). But if all the hooks are in place, we could choose to create modules to generate both mhtml as html.

@Atinux

This comment has been minimized.

Copy link
Member

Atinux commented Jan 7, 2019

Indeed, this could be made with the help of a module @pimlie, this module could:

  • Overwrite all webpack file/url loader to force include as data64
  • Find any url that point to a static file and include it's base64
  • Remove code-splitting to have only one huge JS file?

Also, with 2.4, modules can create sub commands 🚀

@DreaMinder

This comment has been minimized.

Copy link

DreaMinder commented Jan 9, 2019

I've made a module for this purpose, didn't know that a simillar one already exists.
Its very simple, but requires custom code in asyncData like payload mechanism does.
Repo: https://github.com/DreaMinder/nuxt-payload-extractor
Working demo: https://dreaminder.pro/ru/blog/nuxt-vs-vue-spa-battle (just a sketch of my blog)
Maybe you'll find usefull some ideas I used.

@manniL

This comment has been minimized.

Copy link
Member Author

manniL commented Jan 14, 2019

Another problem that comes to my mind is queryString. But something like /articles?page=2 won't work properly when you have no server behind it.

We should still think about that

@manniL manniL changed the title Fully static generated page Full static generated page Jan 23, 2019

@manniL manniL pinned this issue Jan 27, 2019

@Atinux Atinux changed the title Full static generated page Full static generated mode Jan 28, 2019

@DirkStevens

This comment has been minimized.

Copy link

DirkStevens commented Jan 30, 2019

Wonderful thoughts!

We're using Nuxt as the front-end for data from a Headless Drupal. We'd love to be able to render a static site.

These are some of the challenges we face with static rendering:

We've got >40,000 pages managed and maintained by a team of 25+ writers

  • Most pages are updated regularly several times a year (think: 25 * 40,000 = 1,000,000 updates/year)
  • Pages are added on a daily basis
  • Some pages are removed

How and when are re-renders performed of added/changed pages? How do you know about new pages? Removed pages?

How can we let Nuxt know what data changed?

Routes are managed in the CMS by the writers

  • This is very different from common router scenarios where the routes are known upfront and can be defined beforehand.
  • Pages can be moved from one route to another
  • This requires a re-generate of several pages
  • Routes/slugs can be renamed - this touches all pages that link to that page also
  • Our routes can not be defined with patterns at all...

How can we let Nuxt know a route changed?

Search pages

  • We've got search pages with fuzzy and facet bases search that require server side calls
  • We've experimented with browser bases search on full-data fetches and have excellent results in the UX but problem is to get the data to the client and keep the data current

Some pages combine data from multiple sources

  • Page A uses DataA
  • Page B uses DataB
  • Page C uses parts of DataA and DataB

Ideally the generate takes re-use of data into account.
If DataA changes, Page A and C need to be regenerated. If DataB changes Page B & C need to be regenerated.

We're not familiar with the deep insides of Nuxt but would love to help.

Cheers!

@DreaMinder

This comment has been minimized.

Copy link

DreaMinder commented Jan 30, 2019

@DirkStevens Wow, sounds like worst case scenario for static-generated project =)
I'm not nuxt contributor, but I think complicated invalidation logic varies with every project and can't be included in framework core.
If you want speed up your project, I'd suggest to go for nginx-cache or varnish. There are ways to communicate with between nuxt and proxy-cache to invalidate updated content, it would be much better reliable solution.

Just imagine that you changed navigation item in your layout and you now have to generate 40k pages, because every page contains navigation.

@pimlie

This comment has been minimized.

Copy link

pimlie commented Jan 30, 2019

@DirkStevens You'd probably should read this question: nuxt/nuxt.js#2370 and the nuxt generate docs in general.

@DirkStevens

This comment has been minimized.

Copy link

DirkStevens commented Jan 30, 2019

@DreaMinder @pimlie Thanks for your thoughts.

True, invalidation logic should happen in the project and not in Nuxt. For large, high-impact production static sites it would be good if Nuxt exposes events that let us trigger partial re-generate of items that we mark as invalid.

Right now we use Akamai CDN. @DreaMinder Could you point me to information about proxy-cache invalidation with Nuxt?

@DreaMinder

This comment has been minimized.

Copy link

DreaMinder commented Jan 30, 2019

@DirkStevens sorry, no. Maybe I'll make an article about it, it is complicated. Shortly, nginx-proxy-cache (or cloudflare) respond with cache by default but checks in background if content is updated with limited interval. I forgot how this strategy is called...

@sebtiz13

This comment has been minimized.

Copy link

sebtiz13 commented Feb 4, 2019

I'm just starting on nuxt, but for my blog i have create an modules to download the list of my articles and I do my treatment in hook "build:before" for save that in json file.

After in pages I import this json file and return the part of data which interest me in asyncData.
Currently I have only 5 articles on my blog so I do not know the impact on the performance of generation and its most a proof of concept for the introduction to the static site for me.

But i think its not bad because the file is transformed in js file with hash by nuxt, so the browser can keep it in cache.

I know this solution its not adapted for big site, but I think we can create chunked file with the specific data for each pages.

Finally Its true that it would be nice if nuxt managed the creation of these files through asyncData and this function its only called on server side.

@yann-yinn

This comment has been minimized.

Copy link

yann-yinn commented Feb 5, 2019

My two cents

Nuxt is actually an awesome Static Site Generator and i built several sites it as a static generator.

My guess is that a solution like Gridsome will probably soon get most of the hype concerning JAMStack with Vue because :

  • It is focused on JAMStack only, so it will probably move much faster in this direction
  • It will offer, out of the box, some plugins to pull data to generate pages (yaml, wordpress, contentful etc), so it will probably work faster for beginners
  • There is a GraphQL layer to get the data, people seems to love the idea and tha's part of what made Gatsby so successfull (mixing the good words together : JAMStack + React + GraphQL, bingo ^^').

I would say the most important things for the users is to have plugins to fetch data from sources in a easy way, out of the box.

For example : today, how do you fetch data from markdown files to generate html files ? Which is the best way to create a "Hugo Like" or "Jekyll Like" static site ? This is not something that is easy to find. I began to code my own package for that, nuxtent was a solution but seems not to be maintained anymore.

The asyncData / JSON files issue is more a technical issue than a way to vastly improve the Developper experience when generating static pages with Nuxt.

@manniL

This comment has been minimized.

Copy link
Member Author

manniL commented Feb 5, 2019

@yann-yinn Thanks for the feedback ☺️

A few words on your post:

It [Gridsome] is focused on JAMStack only, so it will probably move much faster in this direction

Could be possible, but doesn't have to be. We are also concerned with JAMstack as you can see 😋

It will offer, out of the box, some plugins to pull data to generate pages (yaml, wordpress, contentful etc), so it will probably work faster for beginners

True, that's something we can start with / focus more on. Easier integration with Headless CMSs. There are some nice initiative (e.g. this module or this module) out there.

Also, more tutorials on that topic would be great, I agree.

The asyncData / JSON files issue is more a technical issue than a way to vastly improve the Developer experience when generating static pages with Nuxt.

I don't agree with that. A "full static mode" would increase the performance of the page and would lead to less mismatching content (as explained in the intro post).

@yann-yinn

This comment has been minimized.

Copy link

yann-yinn commented Feb 5, 2019

I don't agree with that. A "full static mode" would increase the performance of the page and would lead to less mismatching content (as explained in the intro post).

I agree a "full static" generation without additional asyncData call on client Side would be cool and also help to have more "atomic deploys" , which is part of JAMStack philosophy. My concerns about source plugins and comparison with Gatsby-like static site generators should be in another issue than this one.

@cesasol cesasol referenced this issue Feb 18, 2019

Open

Roadmap for v3 #182

6 of 21 tasks complete
@cesasol

This comment has been minimized.

Copy link

cesasol commented Feb 18, 2019

I do think that the static or event the ssr mode could be improved, this article points that website made with techniques like hydrating are actually bad but could be improved with partial hydrating. In my case I often find myself looking at big page components that only render static html but takes twice the time to get to an interactive state (hydrating) for the huge code that is makes, so far I managed two options, one is to use a functional per view component that renders only static html, the second (currently named nuxtent) is to have the html or markdown in another file and use it as an api.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment