Spec Compliance Test App

[lidel]

Summary

• We need a conformance test suite that implementers and service providers can run against their service to confirm it works as expected.
• ideally this would be CLI tool that takes <access-token> and the URL with API <endpoint> and runs a set of tests
  • tests should be designed to run idempotently and reduce friction (e.g. remove all pins as the first step and confirm there are no pins as the second, and remove everything as the last step)

I'm available to review / feed edge cases is anyone wants to pick this up.

Implementation details

Create JS client library

• I believe the prerequisite here is to generate an official JS client library from the Open API YAML spec at https://ipfs.github.io/pinning-services-api-spec/
• Then publish it on NPM as @ipfs-shipyard/pinning-service-client or similar
  • Prior art (generated by someone from community): https://www.npmjs.com/package/js-pinning-service-http-client

CLI Compliance Test suite

The compliance test would be a separate package (@ipfs-shipyard/pinning-service-complicance-checks) that uses the client library to run tests and exits with code 0 if there were no hard errors:

$ npx ipfs-pinning-service-complicance-checks https://service.example.com secret-token
Checking compliance of Pinning Service API at  https://service.example.com:
  (output)
Done!

Web interface (nice to have)

Would be nice to have a static website with two inputs for <access-token> and <endpoint> and "Test" button, but this is lower priority than CLI tool (we want something that can run automatically on CI to constantly validate services we list in ipfs-webui – ipfs/ipfs-webui#1854 (comment))

Test scenarios

Below are things we want to test, in order:

MVP list

• Compliance Check: confirm Authentication is present ipfs-shipyard/pinning-service-compliance#3
• Compliance Check: List pin objects (GET /pins) in all states  ipfs-shipyard/pinning-service-compliance#4
• Compliance Check: Add a single pin  ipfs-shipyard/pinning-service-compliance#5
• Compliance check: Pagination ipfs-shipyard/pinning-service-compliance#6
• Compliance Check: (details TBD) test delete via DELETE /pins/{requestid}  ipfs-shipyard/pinning-service-compliance#7
• Compliance Check: (details TBD) test replacement  ipfs-shipyard/pinning-service-compliance#8
• Compliance Check: (details TBD) test name match  ipfs-shipyard/pinning-service-compliance#9
• Compliance Check: (details TBD) test metadata ipfs-shipyard/pinning-service-compliance#10
• Compliance check: (details TBD) test delegates  ipfs-shipyard/pinning-service-compliance#11
• Add Pinning service provider: Test ipfs cluster ipfs-shipyard/pinning-service-compliance#13
• Compliance checks: Extended / unsorted list (TBD) ipfs-shipyard/pinning-service-compliance#12

[SgtPooki]

I've moved all of the checklist items from this issue to https://github.com/ipfs-shipyard/pinning-service-compliance/issues. We should update any of those task items in the appropriate issues going forward.