IPIP: Add IPLD Gateway Specs

[RangerMauve]

WIP: Drafting up specs for /ipld/ support in gateways

[rvagg]

nice

@lidel where do we stand with writable gateways? I don't believe we have writability enabled anywhere (yet) and there's been some debate about pulling the trigger on this. If we put POST and PATCH in the spec, what does that mean for implementations? Does that just become an optional part, and we could handle it in code by allowing the config to turn on writability?

[RangerMauve]

I'm actually gonna be working on the writable gateway spec this week based on the stuff we did in the Agregore IPFS Daemon. AgregoreWeb/agregore-ipfs-daemon#10

[RangerMauve]

I've been signaling writability support by returning different HTTP method names for HEAD requests (e.g. only show PUT if it's a writable gateway)

[RangerMauve]

Err, should I submit a new PR along with the required doc once this is more thoroughly flushed out?

[lidel]

@RangerMauve no, this was a bug in Github (closed PRs against my branch instead of rebasing them against main).
I'll fix it manually.

[lidel]

@rvagg The idea is to flesh the writable gateways optional. Just want to specify behavior is someone wants to implement it.

Potential user:

• Public services such as Gateways or Pinning Services could decide to allow writes only if requests include Authorization: Bearer <token>
• localhost gateway provided by things like Kubo, Brave, Agregore could have it enabled by default (tbd)

[BigLep]

2022-07-19 conversation: there are things from IPFS Thing that influence this:

1. Demonstrations by @aschmahmann
2. Discussions that were had with @mikeal

@RangerMauve wants to understand the WASM story more and how that would impact this.

[softwareplumber]

Couple of comments:-

I think the parameters block (enclosed by square braces) is essentially doing what Timbl's old Matrix URI idea: on w3.org was trying to do. Matrix URIs are sparsely supported in some parts of the Web2 ecosystem (the Java JAX-RS API, I believe). Getting implementers of web toolkits (express, etc) to provide native support for IPFS URIs may be easier if we can 'invoke Timbl' rather than attempting to convince them a prori of the rightness of our approach.

More practically, there is an awful lot of infrastructure out there (proxies, reverse proxies, kubernetes ingress controllers, etc) which depends on cruddy regular-expression based parsing of URIs. Re-using '&' as a separator is an admirable idea from the point of view of code re-use but it risks breaking any bad regex that is relying on '&' appearing first in a query string. Moreover using '[' as a meaningful separator in the URI will make any implementer of said cruddy regex parsers cry into their beer as they try to navigate thickets of escape characters.

For these reasons I'd suggest that ';' as a separator, per the Matrix URI spec, is preferable.

[aschmahmann]

@softwareplumber IIUC the Matrix URI puts ; only at the end of a path and not in the middle and as a result looks an awful lot like query parameters ?a=b&c=d.... Part of the idea here is that including the escaped path information in the middle allows the paths to be easier to understand.

For example: /ipld/bafyroot/[ADL = HAMT]/entry1/field2/[ADL = FBL] can render the bytes for a picture, video, etc. Where HAMT and FBL are the abstract data layouts described in https://ipld.io/specs/advanced-data-layouts/.

[aschmahmann]

Noting that there have been proposals to use other signaling mechanisms than out-of-band including it in the path. I'd recommend those interested take a look at https://ipld.io/docs/advanced-data-layouts/signalling/ for some background (including following through to the naming and dynamic loading sections if you're interested). While none of the opinions on that website are "law" they may provide some useful context in either forming your own opinions or understanding those of others. If folks have other useful resources I'd drop them here as well.

[RangerMauve]

Btw, I did a talk last week at the IPFS thing and here are the slides from it: https://blog.mauve.moe/slides/ipld-gateway/#1

You can press p to open up the speaker notes for some of the stuff I said (will link to the recording here once it is published)

[softwareplumber]

Well, the JAX-RS spec *definitely* provides for matrix params within the path. I admit it's not so clear from the original Timbl musing, but the Java world is shot through with examples of matix params used in the middle of the path (e.g. https://www.logicbig.com/tutorials/java-ee-tutorial/jax-rs/jaxrs-matrix-param.html). So the example would become: /ipld/bafyroot;ADL=HAMT/entry1/field2;ADL=FBL

[RangerMauve]

I'm personally all for reusing an existing standard. One thing I like about this JAX thing is that it disambiguates which "side" the metadata goes on. e.g. ipld://cid/[foo=bar]example/ and ipld://cid/example[foo=bar]/ could serialize to the same thing, and could also lead to confusing situations like ipld://cid/[foo=bar]example[foo=baz]/.

With this syntax, we know for sure that the extra data goes *after the segment name.

Using semicolons to separate bits means that we can't just dump the segment into URLSearchParams, but I think that's easy enough to work around.

We can then say that just ; needs to be escaped rather than [ and ] in path segments.

Also, the thing about proxies seems like a good call since those things have caused issues in the past.

I'm not particularly married to [ ] so I'm down to spec paths with this JAX syntax instead. I'll reference the Matrix URI spec here: https://www.w3.org/DesignIssues/MatrixURIs.html

(I'll do some tests with existing URL parsers to see if they complain about it)

[softwareplumber]

@RangerMauve that's awesome; even if the idea doesn't work out I'm really pleased it's being considered.

If I can make one other suggestion, for future-proofing it might be an idea to somehow 'namespace' keywords like 'ADL' (perhaps prefixing with '$') and, maybe, reserve some kind of wildcard character in the spec. I have a gut feeling that eventually we're going to want a path-like syntax to represent selectors (or something that replaces selectors) and providing upward compatibility in the Gateway URI spec so that Gateway URIs are a subset of Selector URIs would be a good thing.

For example, I'm thinking that in the fullness of time a path like <CID>/folder/*;owner=alice might represent all the descendants of the 'folder' node with an attribute 'owner' equal to 'alice'. The path after the CID is of course a human readable representation of a simple selector; if we ever want a gateway to have this functionality it would make sense to reserve '*' and ensure that our operators (such as ADL) can be distinguished from attribute names.

[RangerMauve]

Hmm. Extra keywords in the path seem interesting, I wonder if it'd be stepping over some of the use cases of selectors, however.

One of the things I was thinking would be important is that the result for these IPLD URLs / operations should be either a new IPLD data model node, or a URL pointing to such a node. Would using extra wildcards have it return a list node?

Might be good to talk about it on the call.

[softwareplumber]

Yes, there is a crossover with selector use cases, but that's fine. What I'm basically saying is that building space in the URI spec so that it could eventually handle some of those use cases without having to be re-written can only be a good thing. ( :-J And, in the meantime, this maybe gives us a way to write down simple selectors that doesn't involve mind-destroying numbers of curly braces ) I agree that the question of exactly what a path with wildcards in it would return is a vexed one. The simple answer ('a list of nodes') may be wrong. But I think that's a design bridge that could be crossed if anyone ever wants to implement the feature. What I'm suggesting is more leaving space (literal namespace) for someone to implement it if they want to. What call? I'm just a newb.

------ Original Message ------ From: "RangerMauve" ***@***.***> To: "ipfs/specs" ***@***.***> Cc: "softwareplumber" ***@***.***>; "Mention" ***@***.***>

Sent: 8/1/2022 4:38:56 PM Subject: Re: [ipfs/specs] IPIP: Add IPLD Gateway Specs (PR #293) Hmm. Extra keywords in the path seem interesting, I wonder if it'd be stepping over some of the use cases of selectors <https://ipld.io/specs/selectors/fixtures/selector-fixtures-1/>, however. One of the things I was thinking would be important is that the result for these IPLD URLs / operations should be either a new IPLD data model node, or a URL pointing to such a node. Would using extra wildcards have it return a list node? Might be good to talk about it on the call. — Reply to this email directly, view it on GitHub <#293 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGAPFYYWWCMGCYZ6LXCSI2DVXAYWBANCNFSM5ZZKYRKA>. You are receiving this because you were mentioned.Message ID:

***@***.***>

…

[RangerMauve]

Recording from the discussion we had about this at the IPLD thing last month is up here: https://www.youtube.com/watch?v=_uXKIEmJh3g

[RangerMauve]

I got some initial ipld:// protocol support into js-ipfs-fetch@4.2.0 😁 So far it supports a basic GET/POST which can do coversion between codecs at the protocol handler layer. https://github.com/RangerMauve/js-ipfs-fetch/#await-fetchipldcidexample-method-get-headers-accept-applicationjson

I've also put together a JS library for parsing and serializing ipld:// URLs with the new matrix parameters syntax for adding extra signaling for path segments

https://github.com/RangerMauve/js-ipld-url

I'm also gonna release it in the Agregore Browser for desktop to make it a bit easier to mess around with.

I got POST more figured out, along with some uses for the ?format parameter.

Next up, I wanna look into sketching up what the schema parameter could mean for path segments.

I'll also update the gateway spec with these new changes as they come. 😁