fix(gateway): undesired conversions to dag-json and friends

[hacdias]

In the spirit of https://github.com/protocol/bifrost-infra/issues/2290 and ipfs/specs#328 (review), this PR removes implicit conversion from UnixFS/RAW to DAG-* when json is requested, and performs the conversion only when expliti dag-json is present in the request.

We also fixed handling of CID with json and cbor codecs + added more tests.

Weird, unrelated things, are happening in the CI. – tracked in #9570

[hacdias]

@lidel another option I see is to completely remove support for ?format=json|cbor but that will also raise the question of: should we then also support CIDs whose codec is JSON|CBOR?

[lidel]

Let's add a test that ensures we keep read-only support for CIDs with json and cbor codec: when someone requests /ipfs/bagaaierayeva7pcl4bchk3udf3i5qm3eeqlbj6tzcmidv64uonglmi5nebca (CID with json codec) without any parameters, we want gateway to return same payload as raw block response, but with content-type: application/json based on the codec from CID.

[hacdias]

@lidel we had the tests already 😃 (test_cmp_dag_get).

[lidel]

This is getting really hairy. I am writing more tests, as I've found some new papercuts and missing cases that we should have regression tests for:

• Requesting CID with json codec with Accept: aplication/json should work, but does not
• Requesting CID with dag-json codec with Accept: aplication/json should work (without conversion of bytes), but does not

I will prepare a fix and push together with tests.

[lidel]

Pushed two fixes and tests in c3ebe80
It should solve the problem from https://github.com/protocol/bifrost-infra/issues/2290 and be ready for 0.18.

@hacdias when you start on Friday, take a look at the above changes as a sanity check and lmk if anything is weird or needs to be better commented.

(interop/webui tests are failing due to unrelated reason – tracking in #9570)

[hacdias]

@lidel what you pushed works if a file has JSON content (or whatever content it has). However, it does not work for a CID that has the codec JSON (or CBOR for that matter). The fault in the tests lies in that FILE_JSON_CID is just a regular UnixFS/Raw CID and not a JSON CID.

I pushed a fix and another test... I'm also not very happy with this. It's becoming increasingly complex and I feel like there must be a way of simplifying this. I will give it some thought during today's morning and see if I can simplify it somehow.

[lidel]

Thanks for catching this @hacdias

Maybe we could simplify things a bit by adding CID codec check at the end of customResponseFormat func, and set explicit one when CID is json cbor dag-json or dag-cbor? at least that will happen it one place.

I will circle back to this PR after meetings, but don't want to block the release, we have test coverage now which is the most important part.

We may:

• merge it as-is to fix the JSON problem for 0.18, and refactor/simplify later (fill issue with ideas how to simplify it)
  • if we do this, we need to update https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.18.md to reflect changes in this PR
• revert entire JSON/CBOR support, and move it to Kubo 0.19

I feel this PR makes it safe to ship in 0.18, but lmk

[hacdias]

@lidel I think we should merge as-is and simplify later if possible.

[BigLep]

I'm good with us doing the followups in the next release (assuming we do the changelog update and have the followup issue for 0.19). Thanks for handling.

[lidel]

(writing updates to release notes section, will push and merge before Monday)