Async IO for HttpsDataVault, Fox Error Handling

[fm3]

• VaultPath.readBytes now returns Fox[Array[Byte]]
• Adapted usages and implementations accordingly
• Used the actually async http client play ws in the HttpsDataVault
• 404 is now propagated as Fox.empty and fill-value is inserted there. Other errors are propagated as errors (leading to neutral gray rendered by the front-end rather than the fill value’s black). Note that this only works for the HttpsDataVault at this point

TODO

• adapt vault path and data vault array interfaces
• adapt usages
  • adapt opening arrays and reading headers
  • adapt reading chunks
• adapt tests
• use non-blocking http request in HttpsDataVault
  • pass ws in there
  • use ws to get info and data
  • basic auth
  • what about connect timeout?
  • ensure this works with gzip and brotli encodings
• 404 should be propagated as Fox.empty and use fill value where appropriate
• merge master into this after Shared Chunk Cache for all DatasetArrays, CacheWeight for AlfuCache #7067 is in there

URL of deployed dev instance (used for testing):

• https://vaultreadfox.webknossos.xyz

Steps to test:

• Explore and view some remote datasets with different stores/data types, should still show data as before
• Self-streaming should now work (create zarr link in annotation, explore that link in the same webknossos instance)

Issues:

• fixes Error handling in DatasetArray: move away from Exceptions, use Fox/flatMap #6906
• fixes Zarr-Streaming from a datastore into itself is broken #6872
• fixes Should Fox have an Empty-preserving variant of ?~>? #7141

---

• Removed dev-only changes like prints and application.conf edits
• Needs datastore update after deployment

[fm3]

@frcroth do you happen to know a test dataset that uses brotli compression and can be accessed via https directly?

[fm3]

I was able to validate that both gzip and brotli work ok with the adapted HttpsDataVault. Looks like play ws does decode gzip itself, but it then also “eats” the encoding header, so our own decode code does not attempt to decode it again. On the other hand, when ws encounters an encoding it does not know how to decode, it keeps the header and does not decode. So that also works with our VaultPath code.

To test this I used this simple flask server:

from flask import Flask, make_response
import gzip
import brotli

app = Flask(__name__)
app.config["COMPRESS_REGISTER"] = False

@app.route("/")
def hello_world():
    return "Hello, World!"

@app.route("/gzip")
def test_gzip():
    s = "Hello, World!"
    b = s.encode('utf8')
    content = gzip.compress(b, 5)
    response = make_response(content)
    response.headers['Content-length'] = len(content)
    response.headers['Content-Encoding'] = 'gzip'
    return response

@app.route("/brotli")
def test_brotli():
    s = "Hello, World!"
    b = s.encode('utf8')
    content = brotli.compress(b)
    response = make_response(content)
    response.headers['Content-length'] = len(content)
    response.headers['Content-Encoding'] = 'br'
    return response

and the test route in commit bdef9ce

[frcroth]

Not totally happy with the name, can we find something more distinguishable?