Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a "resource file adapter" for Git LFS/Netlify’s Large Media support #5749

Open
bep opened this issue Mar 12, 2019 · 6 comments

Comments

@bep
Copy link
Member

commented Mar 12, 2019

https://discourse.gohugo.io/t/netlifys-large-media-support/17259/9

version https://git-lfs.github.com/spec/v1
oid sha256:e6df77690697794deacb9d6963e574044448bef923f7d688aab1d856c54388a3
size 4074082

I assume there are Git LFS Go libraries out there so we should be able to do this effectively.

@bep bep added the Enhancement label Mar 12, 2019

@bep bep added this to the v0.56 milestone Mar 12, 2019

@fool

This comment has been minimized.

Copy link

commented Mar 13, 2019

In netlify's build environment, by the time hugo is running, there won't be permissions to pull new files from the repo (all git permissions are dropped before build, after cloning), so while this might be useful in other circumstances, it won't solve build problems at Netlify for people using the large media feature.

As far as I could determine don't currently have plans to make those files available directly during build.

@mkasu

This comment has been minimized.

Copy link

commented Mar 15, 2019

Just a comment, I'm not sure whether it would fit here or in the Discourse thread.

The way Netlify's Large Media support currently works is that the actual git repository and the git-lfs remote are completely separate. On each push, it will upload all LFS files to a separate server. On CI, it will only pull the main repository and some LFS metadata stubs, but no binary files, so Hugo would have no way to ever actually process data. As Netlify also makes money by selling Image Transformations as add-ons, I would assume that this is more or less on purpose and they wouldn't want to change it.

On the other hand, if using a normal GIT-LFS setup, you can set up Netlify to actually pull all binary LFS files into the build environment. Then, it'll just do a post-processing step replacing all metadata stubs with the real binary data after pulling the repo into the CI. Then, GIT LFS + Netlify + Hugo already works pretty flawlessly. My website uses this without issues. I have almost 4000 images when including all transformations done by Hugo. Although, doing all transformations would took about 10-15 minutes on each deploy, so I now usually commit the cache into the repo (also using git-lfs for this.)

@bep

This comment has been minimized.

Copy link
Member Author

commented Mar 15, 2019

@mkasu thanks for the input, some of these issues I create without too much thought, as a reminder. Sometimes they are good ideas, sometimes they can lead to other good ideas.

Although, doing all transformations would took about 10-15 minutes on each deploy,

If you configure your image to use :cacheDir, e.g.:

[caches.images]
dir = ":cacheDir/_gen"

Hugo will use Netlify's cache dir, which, in my tests, should survive deploys.

@mkasu

This comment has been minimized.

Copy link

commented Mar 16, 2019

@mkasu thanks for the input, some of these issues I create without too much thought, as a reminder. Sometimes they are good ideas, sometimes they can lead to other good ideas.

Sure, just giving some viewpoints as I have played around with GIT LFS/Netlify Large Media Support quite a bit.

I think a proxy format would still be a good idea, not in LFS context, but if somebody wants to deploy images separately (e.g. their CDN or another server.) I played around with making a photography website in the past. I wouldn't necessarily want to deploy the images together, but rather just link to them and upload them somewhere else. Of course I could just link them in articles, but having a proxy format and being able to use asset management would allow for more integration with other Hugo stuff.

[caches.images]
dir = ":cacheDir/_gen"

Hugo will use Netlify's cache dir, which, in my tests, should survive deploys.

Ah yeah, thanks. I think I already did that once in the past but forgot about it.

Now I just need to clean up my GIT LFS repository (Hint: LFS does not provide a way to actually prune old files, ever, so repositories just keep growing. Makes sense from a git-standpoint of preserving all history but gets ridiculous for in LFS context targeting big binary blobs.)

@bep

This comment has been minimized.

Copy link
Member Author

commented Mar 16, 2019

LFS does not provide a way to actually prune old files, ever, so repositories just keep growing.

Oh my ...

@mkasu

This comment has been minimized.

Copy link

commented Mar 16, 2019

LFS does not provide a way to actually prune old files, ever, so repositories just keep growing.

Oh my ...

Sorry for going off-topic but I can't resist to add the workaround GitHub recommends for pruning old files:

https://help.github.com/en/articles/removing-files-from-git-large-file-storage#git-lfs-objects-in-your-repository

To remove Git LFS objects from a repository, delete and recreate the repository.

@bep bep modified the milestones: v0.56, v0.57 Jun 14, 2019

@bep bep modified the milestones: v0.57, v0.58 Jul 31, 2019

@bep bep modified the milestones: v0.58, v0.59 Aug 13, 2019

@bep bep modified the milestones: v0.59, v0.60 Sep 6, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.