-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Choosing future-proof stack for gateway tests #2
Comments
Thanks for the input! All good points 🤝 I'm happy to engage more once I'm back; I just wanted to quickly share a few thoughts. I think declarative tests definition is what could really make the setup future-proof. It makes every other component - testing library, reporters, HTTP library, etc. - easily replaceable. Even replacing the language should be pretty straightforward. I think that as long as we do keep these components easily replaceable, it's OK to lean on libraries that help us go faster. Especially now, in these early stages, where we want to verify if the approach even works for what we're trying to achieve. |
Unfortunately, this rant of mine probably isn't as helpful as it could be.. but I'm going to "send it" anyway. tbh: a lot of the reason the ESM migration was painful was because of obfuscator "cheatsheet" packages like create react app (CRA). This bundles up a ton of config, obfuscating a lot of what's happening, trying to do magic for you, and not allowing configuration when more advanced things are needed. My advice here for anyone is to never let CRA sit in your package for more than a week. If you're getting feature requests, EJECT first. It's great for starting up, but a horrible horrible mess for anything of significance. Maybe that's a hot-take, but I've been familiar with getting into the weeds of webpack for a long time. As far as all the dependencies and the pain with webui, ipld-explorer-components, and all the js-ipfs stuff (even current libp2p, multiformats, etc..).. part of the problem there is that we wrote components which are supposed to be consumable/pluggable, but enforce their own dependencies. It's a plugin/module system where you can only plugin when all dependencies match, or are forced to override. So basically, we split up our packages into a lot of little packages that all have the same problem: explicit reliance on specific dependencies, and implicit reliance on functionalities of the collections of those dependencies. We should use a repository, or other abstraction layer, where the interface is what's agreed upon, and the details and implementations are interchangeable. I don't see that we've done that for a lot of our packages, but reasonably so in the JS landscape of old. Whether we choose the JS world or not, JS itself should not be allowed directly for anything we expect to live longer than a few months: TypeScript only moving forward. With abstraction layers, generics, and dependency injection of "loosely defined" interfaces. Define your consumptions as a relationship, and within that relationship, the provider and receiver, and finally the message structure, before you start using the code. One example i'm seeing now (trying to get kubo-rpc-client into webui): we're using ipld-explorer-components, which is it's own page (you two know) but that page accepts only one non-primitive input (1 callback, then 1 bool and 2 strings) <ExplorePage
runTour={toursEnabled}
joyrideCallback={handleJoyrideCallback}
gatewayUrl={availableGatewayUrl}
publicGateway={publicGateway}
/> This package has dependencies that it's trying to load up, that are no longer in sync with what webui has in my branch, and it's causing issues. That component should not have runtime dependencies on any of the following: and should instead accept an object of a particular interface that has the methods it needs to call to do its work and returns the data in the expected format. If all of our dependencies accepted an injected IPFS provider, consumers could override anything they need to match the latest and greatest or flex where they need to adjust for particular business needs. It would also remove the need for each of our packages to have different build artifacts for various platforms, only some object that looks like Inversion of Control(IOC) would help a lot with our dependency hell, because consumers would always have the right of way, instead of each package trying to enforce it. |
Thanks for raising these points, I share the js-ecosystem frustration as well @SgtPooki. I don't know the problem space enough to push for any option, but I would favor anything that lowers friction with eventual maintainers. A few questions about constraints/expectations:
|
Just some quick notes which I hope help.
|
Yes, that is why having static fixtures as CARs will simplify things a lot, and we now proactively ask for them in IPIPs. Example: _redirects IPIP includes fixtures that we use in existing sharness tests. Sadly, we have some technical debt in gateway tests for older things. @laurentsenta in case its useful, we may be in a better place if we pay this debt in two steps:
It will make migration way less stressful, knowing we are testing the same DAGs.
I believe so. Static fixtures, clear inputs and outputs. Makes writing assertions straightforward, and we should be able to return actionable error every time.
I agree with Robin, but also would say: prioritize future readers :) In gateway context, we write tests once, rarely modify them (ossification-by-design), but run and read results many times.
We will have people not familiar with JS tasked with running this suite, often outside PL. |
Thanks for the detailed feedback @darobin, @lidel; That _redirects IPIP got me quite excited, it looks like a good candidate to write conformance tests from the spec (which might be quite different from porting kubo sharness tests).
That makes sense, I'll look into this! A few notes, to summarize: The test suite:
It seems like we have 3 blocks:
We care the most about the test definitions. We expect thousands of tests, and we don't want to change these often. Other parts, like the provision script, are probably something we can rewrite in a couple of hours. This is what I have tried recently:I've been playing around with generating A test looks like this: gateway-conformance/test/kubo-t0118-car.spec.ts Lines 24 to 55 in 907eaac
I took a few screenshots of errors while fixing a test: So far, it seems to me that:
A few next steps: We started from @darobin and @galargh poc and the |
No worries at all, we're all busy. It took me a week to get to watch this :) This looks really nice, thank you! I love the builder pattern. The one (relatively small) thing I would flag is that you seem to be relying on string matching (equals/contains) but a lot of these values can be wilder (eg. case insensitive, there may be spaces, the value might have to be anchored at the start of the string, etc.). If your HTTP library normalises values, this probably helps a fair bit (worth checking), but it's like that quite a few checks will need to go to regex land. This isn't a big deal and not a problem for the framework, I'm just flagging it as a potential problem that jumped out, probably worth addressing before too many tests are written. Good stuff! |
(we had a quick sync today, but also had chat during Kubo standup, and wanted to drop info here, food for thought)
Quick feedback on building something that will last years and decades:
during the past 7+ years we had bad experiences with JS stack (NPM dependencies, to be specific), and having something leaner, ideally based on something future proof like stdlib or curl or libcurl would be easier to maintain and accept than JS libraries which will break in next 1-2 years and will suck time out of teams to fix/maintain (i know this sounds like a controversial take, but ask @SgtPooki how painful ipfs-webui update to the latest js-ipfs and ESM was :'( )
as some prior art in complexity reduction: we have some poc in https://github.com/ipfs/kubo/blob/master/test/cli/gateway_test.go where @guseggert ported sharness into go tests ( you can compare test: port gateway sharness tests to Go tests kubo#9505).
If built-in client is not enough, we can use curl/libcurl (been around for 30 years, not going anywhere).
avoid using third-party JS libs, they are cool and maintained until they are not (request) and you are stuck with technical debt to refactor into something else.
i think it boils down to picking stable foundations, not specific language, so this is just a prompt for discussion. @darobin may have some useful lessons learned and rules of thumb around maintenance of something like WPT (which is, iirc, a bit of everything, JavaScript, Go, Python)
The text was updated successfully, but these errors were encountered: