Merge pull request #575 from cloud-gov/ddb-large-file-handling

Add documentation regarding large file handling

_data/navigation.yml

@@ -114,6 +114,8 @@ sidenav:
         href: /documentation/site-wide-alert/
       - text: Supported site engines
         href: /documentation/supported-site-engines/
+      - text: Large file handling
+        href: /documentation/large-file-handling/
       - text: Renaming your site's repository
         href: /documentation/renaming-site-repository/
       - text: Monorepos on Federalist

pages/documentation/large-file-handling.md

@@ -0,0 +1,34 @@
+---
+title: Large File Handling
+permalink: /documentation/large-file-handling/
+layout: page
+sidenav: documentation
+
+---
+
+There are a few guidelines and restrictions to be aware of when dealing with large files, or a large number of files.
+
+## General Guidelines
+
+Due to restrictions imposed by Github and cloud.gov, as well as the internal infrastructure of Federalist, sometimes builds fail when running out of disk space. This usually shows up as an error in build logs like `"Disk quota exceeded"` or `"No space left on device"`. The Federalist team is looking into longer-term upgrades to help ease these restrictions. In the meantime, here are a few general rules and tips:
+- Regular build container provide 2GB of total user space. Because dependencies like those included in your `Gemfile` or `package.json` will be installed here, it's a good rule of thumb to keep your total Github repository size under 1GB.
+  - You can minimize your dependency size by removing development dependencies from `package.json` and using using [`npx`](https://www.npmjs.com/package/npx) to call development scripts.
+  - You can also minimize your total build size by optimizing images and compressing files that are intended for download.
+- If your site starts hitting this initial limit, you can request an upgrade to 4GB of user space. Send an email to [federalist-support@gsa.gov](mailto:federalist-support@gsa.gov).
+- If your site is going over both of these limits, one temporary workaround is to create a new site just to store certain large assets like images, PDFs, or slide decks. For example, you could host large reports in a separate repository which deploys to `reports.example.gov` and then link to these assets from your primary site.
+
+## Details
+
+### Federalist Build Containers
+
+Federalist offer two sizes of build containers ("regular": 4GB and "large": 6GB). The maximum size is limited by the size of containers available on [cloud.gov](https://cloud.gov). The first 2GB of each container is taken up by dependencies installed by Federalist (Ruby, Node, Python, and additional packages for each) leaving either 2GB or 4GB for the user space.
+
+### Github
+
+Github is not intended to store large files or a large number of files. There are caps on the size on an individual file and the total size of a single repository:
+- Individual file size: 100MB<sup>*</sup>
+- Single repository size: Recommended to keep below 1GB
+
+More details can be found [on their website](https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-large-files-on-github).
+
+_*[Git LFS](https://git-lfs.github.com/) can support larger file storage but currently does not integrate with the Federalist build process_

pages/documentation/migration-guide.md

@@ -69,5 +69,4 @@ You are able to start working while IAA is being signed.
 
 ## Additional info
 
-If your site has over 1 GB of images to serve you should look into another way to store those files. Github is not intended to store large amounts of files.
-- Github has no problem storing less than 1GB of different files types. For hosting over 1GB of files, we suggest getting access to a public database service that will load the files. One provider of this type of service is [cloud.gov](https://cloud.gov/). More information on repository size limitations is available [here](https://help.github.com/articles/what-is-my-disk-quota/).
+If your site has over 1 GB of files to serve you should look into another way to store those files. Github is not intended to store large amounts of files. More details are available in the [large file handling guide](/documentation/large-file-handling/).