Reduce memory usage

[WebFreak001]

I can't really continue to work on my website anymore because dmd crashes because it's out of memory. I am only using 38 diet files which are 52kb in total file size and my code is also only about 6.2kloc (counted with dscanner -l, so its about 6200 expressions like function calls, definitions, etc). Though I think this only happens on windows because dmd has no 64 bit version here so the maximum memory it can allocate is 2GB before it crashes. It might also be because of the way how dub compiles the dependencies and the project, but I think the biggest factor is the diet compilation. That takes a lot of time and memory.

Maybe somehow add support for caching the compiled diet files or add runtime compilation? Most of my diet files don't use any D code except for reading out of a file anyway. (I am reading out of a file and embed it into the page because otherwise cache is a problem for me)

[WebFreak001]

ok sorry the crash actually happened because of this super complex compile time regex:

ctRegex!`\W`

I know, hardcore stuff that should use up 2GB of memory. So I just replaced it with [^a-zA-Z0-9_] and it works now. Thought it happened because I added another diet template when I also added that regex.

[WebFreak001]

Now I actually get out of memory from diet templates - one fix was using 32 bit instead of 64 bit code generation but I'm afraid that this will not save me from issues in the long run. The issue title is "Reduce memory usage" so I guess I should reopen this because it wasn't actually fixed. Soon I might need to use --build-type singleFile but that will massively increase compilation time so I don't want to do that

[s-ludwig]

The problem is that I'm a bit unclear on what to do from within the library. Diet-NG (surprisingly) has turned out to use less memory during compilation than the original module, in which I've used a lot of dirty tricks to reduce memory consumption at the time of its original creation. I don't think there is anything substantial to gain there, except by the new CTFE implementation that will hopefully be a reality one day.

There is this caching mechanism (DietUseCache), but it will save memory only starting with the second build.

But one way to mitigate this issue is to simply separate parts of the application into different sub packages. If a Diet template instantiation is wrapped within a (non-template, non-auto-return) function, its semantic phases should only run while its containing package is built, but not when called from another one. So if there are any logical blocks that can be separated, that would be an easy measure that could also speed up the general edit-compile cycle.

[WebFreak001]

How about adding runtime compilation of diet templates without any D code? 80% of my templates are static anyway because they use javascript/polymer for the most stuff. (Oops forgot to reopen too)

[s-ludwig]

I've opened a separate ticket for the runtime HTML generation. Definitely something that would be useful.