Reliability of local storage (vs. documentation), single-node production deployment

[ppar]

The documentation warns against using local storage, but the reasoning is not very explicit. At the risk of sounding obtuse, I'd appreciate if someone could shed a bit of light on this.

• The HA deployment guide at https://docs.projectquay.io/deploy_quay_ha.html states:

  "Warning
  Do not use "Locally mounted directory" Storage Engine for any production configurations. Mounted NFS volumes are not supported. Local storage is meant for Project Quay test-only installations."

• The architecture document at https://docs.redhat.com/en/documentation/red_hat_quay/3.14/html-single/red_hat_quay_architecture/index#core-prereqs-storage states:

  "Red Hat Quay can work with local storage, however this should only be used for proof of concept or test setups, as the durability of the binary blobs cannot be guaranteed."

For context, I'm planning a production-grade, but non-HA deployment running only a single instance of quay. The deployment should be reliable, but we accept the risk of losing the entire server (and the backup restore downtime). In this use case, using local filesystem storage (ext4 fs on the same server running the quay container) instead of an object storage service would simplify things.

For a true HA setup, the need for a separate object storage independent of the Quay nodes is obvious (as should be the fact that one shouldn't take chances with NFS).

If we assume the hardware and filesystem in a standalone (single-node) production setup are stable, and there's exactly one quay instance accessing the storage, is there any reason not to use local filesystem storage for images?

[michaelalang]

@ppar the reason local storage is not considered Production ready is that a "single disk failure" will bring down your Production Quay instance.

S3 in contrast typically has replica of 3 for all objects as well as multiple Service endpoints (like AWS region us-east-1 and us-west-1)

If you do not expect anyone jumping on a case and fixing your local disk, you can go with disk storage.

Be aware that no one has tested Locking issues with NFS for the storage, no one has tested any other HA file systems, the Quay mainly focuses on S3 storage components in regards to features and reliability.

As stated in the Docs, from Quay perspectiv you can use local storage but it's considered none production from support point of view.

[ppar]

Ok, thanks. To me the docs sounded like there's something within Quay('s local storage implementation) itself that makes using local storage unreliable.

IMO single disk & other component failures aren't really in the documentation scope --- any competent Ops person is already aware of these things, and OTOH, a locally mounted ext4 filesystem might well be redundant (RAID, SAN, etc) without this being visible (or relevant) to userland processes.

So NFS and HA filesystems aside, is local storage receiving the same kind of testing as S3 & friends?