[INT-674] Spark compatibility in the S3 gateway

[lukemarsden]

These are the changes needed to the S3 Gateway to make it work with Spark, and an example of how to use it.

• Handle paths with a trailing slash as if they didn't have one.
• Don't error on trying to read the timestamp on new files in open commits in the S3 Gateway. Spark needs to be able to read them.
• Don't fail GetObject for paths that are simultaneously an imaginary directory and a real file. Instead, read the file. This requires dancing around the fact that InspectFile doesn't work correctly in this case, but ListFile[0] works just fine.

This is important for a customer, so it would be great if we could get it into 2.3.2 please :-)

Master PR - #8129

[codecov]

Codecov Report

Merging #8115 (7488d93) into 2.3.x (77f598a) will decrease coverage by 0.00%.
The diff coverage is 72.22%.

@@            Coverage Diff             @@
##            2.3.x    #8115      +/-   ##
==========================================
- Coverage   22.24%   22.23%   -0.01%     
==========================================
  Files         415      415              
  Lines      112184   112189       +5     
==========================================
- Hits        24955    24949       -6     
- Misses      82940    82942       +2     
- Partials     4289     4298       +9     

Impacted Files	Coverage Δ
src/server/pfs/s3/bucket.go	45.73% <66.66%> (ø)
src/server/pfs/s3/object.go	43.08% <73.33%> (+4.95%)	⬆️
src/server/pfs/s3/error.go	21.42% <0.00%> (-14.29%)	⬇️
src/server/pfs/pfs.go	44.36% <0.00%> (-3.53%)	⬇️
src/internal/collection/postgres_listener.go	76.75% <0.00%> (-2.64%)	⬇️
src/server/pfs/server/compaction.go	69.06% <0.00%> (-1.66%)	⬇️
src/internal/task/etcd_service.go	81.49% <0.00%> (-1.58%)	⬇️
src/internal/storage/fileset/postgres_store.go	82.65% <0.00%> (-1.03%)	⬇️
src/internal/collection/postgres_collection.go	76.92% <0.00%> (-0.84%)	⬇️
... and 16 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[albscui]

This all LGTM! Just a couple comments about not losing err values in debug message. My comments about the ListFile vs InspectFile is mainly for my own understanding.

[albscui]

Also, is the plan to merge this into master later?

[lukemarsden]

Also, is the plan to merge this into master later?

I'll make a PR into master now.

[lukemarsden]

Here's the master PR: #8129