[13.x] Add filesystem driver option for SQS overflow storage

[Orrison]

Follow-up to #59734.

In the original PR I proposed offloading large SQS payloads to a filesystem disk (typically S3). Before merge that was changed over to cache stores. I asked in the PR comments whether you'd be open to making this configurable, and this is that PR. It adds a driver option under overflow so people can pick between cache (the current default) and filesystem.

A few reasons we want filesystem available:

• AWS parity: S3 is the storage AWS itself documents for this pattern. The Amazon SQS Extended Client Library for Java uses S3, and most engineers reaching for SQS overflow are coming from that pattern.
• Persistence semantics: Cache stores often have eviction or TTL behavior that doesn't suit job payloads which may sit on a queue for extended periods before a worker picks them up. Redis or Memcached behind a memory limit can drop the payload before the worker ever reads it. S3 and SQS, on the other hand, are both designed to hold data durably.
• Compliance and audit: For audited environments (SOC 2, ISO, HIPAA, etc.), putting job payloads on S3 is straightforward to document since it is an AWS-recommended standard. Putting them in a cache store adds audit surface and more work with auditors. At our scale that probably means we can't adopt the feature with cache as the only option.

The default stays driver => 'cache', so nothing changes for anyone already using the feature as merged. (if that makes it out before and if this is merged)

Config

'overflow' => [
    'enabled' => env('SQS_OVERFLOW_ENABLED', false),
    'driver'  => env('SQS_OVERFLOW_DRIVER', 'cache'), // cache or filesystem
    'store'   => env('SQS_OVERFLOW_STORE'),           // when driver=cache
    'disk'    => env('SQS_OVERFLOW_DISK'),            // when driver=filesystem
    'prefix'  => env('SQS_OVERFLOW_PREFIX'),
    'always'  => false,
    'delete_after_processing' => true,
],

When driver is filesystem, the queue writes the payload to the configured disk at {prefix}/{uuid}.json and stores that path in the SQS message body as the pointer. Workers read the payload back from the same disk. clear() also removes the overflow directory when delete_after_processing is on.

When driver is cache, behavior is exactly what shipped in #59734.

Tests

All existing cache-driver tests stay as-is. Filesystem mirrors added for:

• push at threshold
• always => true
• custom prefix
• below-threshold no-op
• clear() calls deleteDirectory when cleanup is on
• clear() skips it when cleanup is off
• pop() forwards filesystem config to the job
• pointer resolution and result caching on the job side
• file deletion on job completion (and the disabled case)

[Orrison]

Apologies, it seems some of the CI jobs are failing with GitHub oauth token issues unrelated to my changes. I rebased pushed up a few times to hopefully get it to pass but one always seems to fail with this token error.

If someone with the ability to do so could please re-run the failed jobs for me.

I assume this may be related to #60095

[Orrison]

Thank you @crynobone!

[taylorotwell]

I wonder if it's actually simpler to just have an S3 / disk cache driver that uses Storage under the hood?

[Orrison]

I wonder if it's actually simpler to just have an S3 / disk cache driver that uses Storage under the hood?

Perhaps, but then you are bleeding cache stores and filesystem disks together a bit unnecessarily imo.

And having to set up and deal with disk configuration that is already taken care of in filesystems. Though I guess since it uses Storage under the hood as you said, it would pull from that config and functionality anyways, so that's a moot point.

Would this S3/disk cache driver be something first-party in the framework or are you suggesting that people that need this would make a custom one?
Either way, in the framework or a custom cache driver, this does feel a little weird to me just referencing the filesystem. And choosing an overflow driver and the current suggested branching paths don't seem that complex to me.