Recompiling all entry templates after changes of a non-entry partial file leads to long project rebuild

[exocornet]

Hello.
Website have 30 pages and 50 components.
Changing something in the component recompiles for a long time up to 3 minutes. Can you tell me how to incrementally build pug files and components?
Thank you.

[webdiscus]

Hello,

could you describe what you are doing, concrete steps?
For example, one page contains Pug includes (mixins, layout, etc), styles and scripts.

• when is changed main Pug file, it take 10 sec
• when is changed included Pug file (footer), it take 20 sec
• when is changed a SCSS file, it take 3 mins
  etc.

P.S. we only speak english, please use google translate

[exocornet]

I hope I understood you correctly.

Pug-plugin settings:

When adding a div element on the page, precompilation completed in 34439 ms.

When a low-level component changes. Used on all pages. The component is called about 60 times. Recompilation takes 255897 ms.

When changing the sass file, the recompilation was completed in 36273 ms.

Many thanks for the help!

[webdiscus]

@exocornet

Thanks, I will research your case.

Note

when you change a globally used SCSS file, e.g. base.scss, then all pages will be rebuilt because the generated CSS filename is changed and all pages must contain the new filename in the link tag.

Workaround

If you have a large project, it is recommended that you define only one Pug file in the entry that you currently work on.
Just manually specify one Pug file in the entry.

[webdiscus]

@exocornet
Is the problem with using Pug in TS solved, the #62 issue?
Possible should be used the esModule: true in loader option.

[exocornet]

@exocornet Is the problem with using Pug in TS solved, the #62 issue?
Possible should be used the esModule: true in loader option.

I will return to this problem a little later. So far, I still do not have time to fully immerse myself in the task.

Thank you for your interest and for your concern!

[exocornet]

@exocornet

Thanks, I will research your case.

Note

when you change a globally used SCSS file, e.g. base.scss, then all pages will be rebuilt because the generated CSS filename is changed and all pages must contain the new filename in the link tag.

Workaround

If you have a large project, it is recommended that you define only one Pug file in the entry that you currently work on. Just manually specify one Pug file in the entry.

I didn't quite understand it. I sometimes use this live hack, but I want to get a good job with the compilation of all pages.

Thanks for helping!

[webdiscus]

@exocornet

The new pug plugin v5.0.0 is released and based on the html-bundler-webpack-plugin.

Try to upgrade your project to v5. See please the BREAKING CHANGES.

If the issue is still yet actual, feel free to reopen this issue.

[exocornet]

@exocornet

The new pug plugin v5.0.0 is released and based on the html-bundler-webpack-plugin.

Try to upgrade your project to v5. See please the BREAKING CHANGES.

If the issue is still yet actual, feel free to reopen this issue.

Hello. I tried the new version of the plugin v5. Nothing changed. It still takes a long time to reassemble everything. :'((

[exocornet]

I’m also trying to switch to RSPack and it shows an error adding a plugin as in this answer webdiscus/html-bundler-webpack-plugin#55 (comment) .

I would be grateful if we can fix this too. I wouldn’t want to switch to old solutions and plugins since your plugin is good and I really like it. Thank you for its development and timely support.

[webdiscus]

@exocornet

using the version 5.x please change your Webpack config:

• remove all Pug rules in module.rules where is used PugPlugin.loader.
  The new pug plugin does not need the Pug loader config.
• new pug plugin detects all Pug pages automatically, use the entry plugin option as a path to pages:

  new PugPlugin({
    entry: 'src/pages/',
  });

P.S. This does not affect performance, but simplifies plugin config.

[webdiscus]

@exocornet

back to the topic...

Please create a repository with a reproducible performance issue.
To simulate a problem, you can use 10 components and 5–10 pages.

I don't have a project as big as yours, so it's difficult for me to reproduce the problem. But it is very important for me to research and fix the performance issue.

[exocornet]

@exocornet

back to the topic...

Please create a repository with a reproducible performance issue. To simulate a problem, you can use 10 components and 5–10 pages.

I don't have a project as big as yours, so it's difficult for me to reproduce the problem. But it is very important for me to research and fix the performance issue.

Of course, here's the repository - https://github.com/exocornet/pug-plugin-test . This template is my personal customization and if you are interested in something, you can take it for your personal use. But for now I would like to do a quick reload of pug during development.

[exocornet]

I'm also interested in the question. I did a little research on the "html-bundler-webpack-plugin" plugin and since pug-plugin is written in it. Is there a need for pug-plugin now? As for me, “html-bundler-webpack-plugin” gives all the same capabilities or are there still some differences between these two plugins?

[webdiscus]

are there still some differences between these two plugins?

No differences.

But later I will add back the pug specifically pretty option to format minimised HTML in development mode.

See src/index.js

The pug-plugin >= v5.x is the html-bundler-webpack-plugin preconfigured for using pug, not more.

pug-plugin == html-bundler-webpack-plugin + pug package + preprocessor: 'pug' (preconfigured option for pug)

You can read the History of Pug Plugin.

• pug-plugin package is stay only for compatibility with projects which already uses the package
• pug-plugin is nice short name

[exocornet]

But later I will add back the pug specifically pretty option to format minimised HTML in development mode.

If we talk about pretty in pug, then this setting was most suitable. But she doesn’t do everything as it should :(. I haven’t found the ideal method yet and I use the one on the screenshot.

Here are examples of defective pretty html.

[webdiscus]

wow, you found the workaround to format HTML.
Yes, the pretty option uses in old pug-plugin the js-beautify.

But this package formatted HTML not perfectly.

P.S. If you find a better npm package for HTML formatting, let me know.

[webdiscus]

@exocornet

I looked your test project...
The webpack config is too heavy.

The complete project was compiled in 188000ms (over 3 minutes).
I have created a simplest hello world pug template with 3 lines, w/o assets and the compilation took 3.5 seconds on my macBook M1 Max 64 GB. It's not OK, it's realy overhead.

Firstly, I try to optimise your config.

[exocornet]

Yes, first strat (cold start). The fact that it compiles this way I think is normal. Because you need to collect everything first and then send it for withdrawal. But subsequent reassemblies, when they take the same time as the start, are sad and very strange.

And I’m trying to improve this point and somehow raised the question and thought that maybe if your plugin is made for the configuration with an implementation for the request, then maybe this will improve the situation at least a little. (#93)

And speed is one of the reasons why I decided to try running my assembly via express + webpack + pug-plugin and am also already planning to switch to RSPack. But so far I don’t understand how to connect it all and make it work.

Since your plugin is not implemented for on-demand rendering (#93), and there is still an error in RSPack.

For now, I'm making do with the temporary solution 'npm run watch', where you can enter one or more project pages. But this solution also causes a lot of inconvenience.

[webdiscus]

@exocornet

I can reproduce the problem with needless recompilation.

For example, there are project files:

pages/home/index.pug <= entrypoint
pages/home/includes/header.pug (included in entrypoint)

pages/about/index.pug <= entrypoint
pages/about/includes/header.pug (included in entrypoint)

After changes in pages/home/index.pug, will be rendered only this file. OK
After changes in pages/home/includes/header.pug, will be rendered all pages:

• pages/home/index.pug (with all dependencies) OK
• pages/about/index.pug (with all dependencies) NOT OK, this must not be rendered

When any file included in the entrypoint file changes, webpack calls the loader for all pages. This is a BUG!

I don't know why webpack does this. I'm trying to look deeper into the problem.
This is very very hard to fix it, I can't promise that it will be fixed. The problem is deep in the webpack.

[exocornet]

Yes. I understand what you're saying. And here it is possible that the problem is not only in webpack, but also in pug itself. While studying this problem with the pug template engine, I learned that it was created as the main server-side template engine. And I tried to run it in combination with express + pug and it worked very well and quickly. But to my regret, I also need webpack. And it seems to me because of the features of pug and webpack. It would be possible to compile it through a request using the page URL. How I tried to do this in #93

If the plugin - "pug-plugin" could exist in exactly this setting, it seems to me that with such an implementation it would be possible to speed up pug. And also use pug-plugin on the server side with express. But I could be wrong, I don’t have much experience in this and here I’m just guessing from everything I tried and tried to understand the logic of pug + webpack recompilation.

[exocornet]

Yes. I understand what you're saying. And here it is possible that the problem is not only in webpack, but also in pug itself. While studying this problem with the pug template engine, I learned that it was created as the main server-side template engine. And I tried to run it in combination with express + pug and it worked very well and quickly. But to my regret, I also need webpack. And it seems to me because of the features of pug and webpack. It would be possible to compile it through a request using the page URL. How I tried to do this in #93

If the plugin - "pug-plugin" could exist in exactly this setting, it seems to me that with such an implementation it would be possible to speed up pug. And also use pug-plugin on the server side with express. But I could be wrong, I don’t have much experience in this and here I’m just guessing from everything I tried and tried to understand the logic of pug + webpack recompilation.

But here’s how to implement the first launch and assembly via webpack, and subsequently only upon request from the page url. There may also be difficulties here, but perhaps this is not the most difficult situation, like accelerating the reassembly of pug 😄 :)

[webdiscus]

@exocornet

By compilation single HTML file w/o any template engine will be sometime the compilation twice called.

The same issue with html-webpack-plugin - Double compilation with webpack 5 and webpack-dev-server.

This is really webpack bug, not pug.

[exocornet]

@exocornet

By compilation single HTML file w/o any template engine will be sometime the compilation twice called.

The same issue with html-webpack-plugin - Double compilation with webpack 5 and webpack-dev-server.

This is really webpack bug, not pug.

Perhaps you are right. But what I see in this problem is not very similar to my situation. Since for me it simply reassembles all pages, as it should be on the server side without using HMR. And in the problem that you found, you simply called the same rebuild twice. It happens to me once, but that’s all. And plus, this problem is already 4 years old, it may have been fixed a long time ago.

[exocornet]

Yes. I understand what you're saying. And here it is possible that the problem is not only in webpack, but also in pug itself. While studying this problem with the pug template engine, I learned that it was created as the main server-side template engine. And I tried to run it in combination with express + pug and it worked very well and quickly. But to my regret, I also need webpack. And it seems to me because of the features of pug and webpack. It would be possible to compile it through a request using the page URL. How I tried to do this in #93
If the plugin - "pug-plugin" could exist in exactly this setting, it seems to me that with such an implementation it would be possible to speed up pug. And also use pug-plugin on the server side with express. But I could be wrong, I don’t have much experience in this and here I’m just guessing from everything I tried and tried to understand the logic of pug + webpack recompilation.

But here’s how to implement the first launch and assembly via webpack, and subsequently only upon request from the page url. There may also be difficulties here, but perhaps this is not the most difficult situation, like accelerating the reassembly of pug 😄 :)

Tell me, is it possible that your plugin (pug-plugin) can be made to work like this?

[webdiscus]

this problem is already 4 years old, it may have been fixed a long time ago.

To this day this problem still exists. I can reproduce this reliably when compilation is called twice after change single html file.

Yes, this is a slightly different problem, but it may still be indirectly related to the current problem.

[webdiscus]

If the plugin - "pug-plugin" could exist in exactly this setting, it seems to me that with such an implementation it would be possible to speed up pug. And also use pug-plugin on the server side with express. But I could be wrong, I don’t have much experience in this and here I’m just guessing from everything I tried and tried to understand the logic of pug + webpack recompilation.

no, the pug-plugin works only with webpack and cannot work with express as midleware, like on your screenshot. This is absolutely two differente worlds.

[exocornet]

Eheh, I understand. Then I will try to further fix the problem.

[webdiscus]

I'm still searching for a solution to the problem when the dependent file of one page is changed, but all pages are recompiled.

[webdiscus]

Notes

Twice recompilation

To fix twice recompilation the same file just add the aggregateTimeout webpack option:

watchOptions: {
    // fix twice recompilation the same file
    aggregateTimeout: 600,
},

Compiling all entrypoint files after changes of a non-entry file.

When is changed the entrypoint template file, then will be recompiled only this entry file (with dependencies).
When is changed any template partial file (file is not entrypoint), then webpack will recompile all entrypoint files, because hasn't inner dependency tree for templates and don't know which entrypoint is parent to the partial file.

This is a big problem. Currently I don't see a clever solution, sorry. Perhaps, we need develop own caching system with own dependency tree of template files (possible not for all templating engines).

[exocornet]

Although it was not possible to solve the problem, we have now found the cause. And we know what to do. Thank you very much for your help and your interest. I think it will be possible to solve not only this problem. Thank you very much again!

[exocornet]

Can you tell me whether "pug-plugin" will support RSPack and how soon this might happen?

[webdiscus]

Can you tell me whether "pug-plugin" will support RSPack and how soon this might happen?

The problem is that RSPack is not 100% webpack ready yet.
Therefore, RSPack should support the webpack API used in the html-bundler-webpack-plugin (pug-plugin is only the config for html-bundler-webpack-plugin).
Please, follow the RSPack support and compatibility #55 issue.

[webdiscus]

@exocornet

Although it was not possible to solve the problem...

in the version 5.0.1 following bug is fixed:

• recompiling all entry templates after changes of a non-entry partial file

Note:

• If you change a page specifically partial e.g. src/pages/index/styles-and-scripts/index.link.pug, then will be recompiled only index page (ca. 5 sec). Now, this case works as expected.

• If you change common used partial, e.g. a component mixin or layout, then will be recompiled all entries, over 3 mins, it is OK

[webdiscus]

@exocornet

does the fix work for you?
Can be closed the issue?

For other issue, please open new issue.

[exocornet]

Thank you. I will definitely update to this version.

Have you seen this plugin?
https://rsbuild.dev/plugins/list/plugin-pug

What's interesting is that it's on rsbuild. Although I am attracted by the more native component of rspack. But suddenly this solution will help solve some problems.

[webdiscus]

https://rsbuild.dev/plugins/list/plugin-pug

It is very primitive and don't have reach functionality as my plugin ;-)
For me the plugin is unusable.

[exocornet]

@exocornet

does the fix work for you? Can be closed the issue?

For other issue, please open new issue.

Well, the problem itself was never solved. The problem was the long reassembly of the project. When in other frameworks or template engines this all happens instantly. :( But if you have nothing else to help you with, then of course close it. I am grateful to you for such help. Thank you!

[exocornet]

https://rsbuild.dev/plugins/list/plugin-pug

It is very primitive and don't have reach functionality as my plugin ;-) For me the plugin is unusable.

Got it, okay. I was just thinking maybe what mechanism could be implemented so that your plugin would be even better 😊

[webdiscus]

The problem was the long reassembly of the project.

Your project is very big, over 40 pages, over 800 source files.
The first compilation of one page takes ~12 seconds! Recompilation takes ~4.5-5 seconds.

[exocornet]

The problem was the long reassembly of the project.

Your project is very big, over 40 pages, over 800 source files. The first compilation of one page takes ~12 seconds! Recompilation takes ~4.5-5 seconds.

Well, I think you've improved anyway. I will still try to find something that would work faster 😁 and HMR would be million-bearing. I think I will turn to you more than once for advice or opinion. Thank you!

[webdiscus]

I was just thinking maybe what mechanism could be implemented so that your plugin would be even better

No special rspack mechanism was found.
In the rsbuild pug-plugin is only "3 lines" pure JS code:

• get config
• run compilation of pug source
• output

[webdiscus]

I think I will turn to you more than once for advice or opinion.

Yes, sure.

Thank you for help to improve this plugin!

[webdiscus]

@exocornet

I found why your project is builded too long.
The problem is with including the src/sections/StatisticsMap/include-svg/world-map.pug file.
It is very big SVG file in Pug format with paar variables.

I have saved the rendered SVG into separate svg file and added one in the src/sections/StatisticsMap/StatisticsMap.pug template as img tag:

- include include-svg/world-map
+ img(src='IMAGES/statisticsMap.svg' class=styles.statisticsMapWorldMap)

Build time for one page with SVG as Pug: ~9 seconds, recompilation - ~5 seconds
Build time for one page with SVG as img: ~4.5 seconds, recompilation - ~0.7 seconds only (!)

Total compilation of all pages takes ~24 seconds.

My recommendation

You can pre-render all variants of SVG images (with all variables) and use a variable name of the SVG file instead of rendering SVG "on fly".