New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory usage during compilation #24

Open
donatello opened this Issue Jan 30, 2018 · 6 comments

Comments

Projects
None yet
4 participants
@donatello
Copy link

donatello commented Jan 30, 2018

When I use either this package or wai-app-static to embed a file around 25MB in size (an executable to be served over http), I see that compilation takes over 3-4 GB of RAM and sometimes crashes my laptop.

Is this expected and can anything be done to make it more efficient? It seems that such high memory usage should not actually be needed during compliation.

@mgsloan

This comment has been minimized.

Copy link
Contributor

mgsloan commented Jan 30, 2018

This is probably a GHC issue. Ideally, Template Haskell should let you directly use a ByteArray or similar. Currently, it is run through a [Word8]. I'm surprised the memory usage would be that high, though.

@donatello

This comment has been minimized.

Copy link
Author

donatello commented Jan 30, 2018

Here is a somewhat minimal demonstration of the issue - https://github.com/donatello/file-embed-exp

Should I report this as an issue with ghc?

@mgsloan

This comment has been minimized.

Copy link
Contributor

mgsloan commented Jan 30, 2018

Sure, I think it'd be reasonable. Unfortunately, base does not have any type like Bytestring or Text for TH to use. It'd make sense to have something that could use ByteArray# (probably wrapped in a datatype), and something that could use (Ptr Word8, Int) or something like that.

@donatello

This comment has been minimized.

Copy link
Author

donatello commented Jan 30, 2018

I've opened an issue here - https://ghc.haskell.org/trac/ghc/ticket/14741

@steshaw

This comment has been minimized.

Copy link

steshaw commented Aug 9, 2018

@donatello Would be good to see a solution via GHC but in the meantime, maybe file-embed's inject system would suit your purpose? The closest thing I can find to a tutorial on it is here.

@hsyl20

This comment has been minimized.

Copy link

hsyl20 commented Jan 15, 2019

I have documented an alternative approach that could be used to fix this issue: https://hsyl20.fr/home/posts/2019-01-15-fast-file-embedding-with-ghc.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment