Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug: Summary compliance check SUM is not consistent between service runs #29

Closed
SgtPooki opened this issue Apr 27, 2022 · 3 comments · Fixed by #54
Closed

Bug: Summary compliance check SUM is not consistent between service runs #29

SgtPooki opened this issue Apr 27, 2022 · 3 comments · Fixed by #54
Assignees
@SgtPooki
Labels
effort/hours exp/beginner good first issue Good for newcomers help wanted Extra attention is needed kind/architecture kind/bug need/analysis P2 released status/in-progress
Milestone
Initial release

Comments

@SgtPooki
Copy link
Member

Problem

Testing 4 different pinning services, we end up with the following summary lines:

  • Summary (21/25 successful)
  • Summary (5/15 successful)
  • Summary (9/20 successful)
  • Summary (18/26 successful)

Details

The total compliance checks ran should ALWAYS be the same. If one errors, it should show in the final results and summary.

Each of the above lines should show the same denominator; always

More output

> npm start -- -d -s $PINATA_API_ENDPOINT $PINATA_API_TOKEN
npm start -- -d -s $WEB3_API_ENDPOINT $WEB3_API_TOKEN
npm start -- -d -s $NFT_API_ENDPOINT $NFT_API_TOKEN
npm start -- -d -s $ESTUARY_API_ENDPOINT $ESTUARY_API_TOKEN

> @ipfs-shipyard/pinning-service-compliance@0.0.3 start

https://api.pinata.cloud/psa compliance:

Summary (21/25 successful)

  ✘ Request with no authentication token
  ✘ Request with invalid token
  ✓ Pins post of CID 'bafkqadjrgy2tcmbvgi3tcnjtgiya'
  ✘ Can create and then delete a new pin
  ✓ List pin objects (GET /pins) in all states
  ✓ Can create and replace a pin's CID
  ✓ Can create a pin with name='32c1af99-ff7e-402f-9846-c56e6f30cde5'
  ✓ Can retrieve pin with name '32c1af99-ff7e-402f-9846-c56e6f30cde5' via the 'exact' TextMatchingStrategy
  ✓ Can retrieve pin with name '32C1AF99-FF7E-402F-9846-C56E6F30CDE5' via the 'iexact' TextMatchingStrategy
  ✓ Can retrieve pin with name 'ff7e-402f-9846-c56' via the 'partial' TextMatchingStrategy
  ✓ Can retrieve pin with name 'FF7E-402F-9846-C56' via the 'ipartial' TextMatchingStrategy
  ✓ Get all pins
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgi3tqnrsguza'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgi3tsmjyhezq'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgi3tsnzugaza'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgi4damrqga3q'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgi4danrzgmya'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgi4dcmjzg4zq'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgi4dcnrzgq2a'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgi4demrqgyza'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgi4denrygqyq'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgi4dgmjyga4q'
  ✓ First page of pins
  ✓ Retrieve the next page of pins
  ✘ Can delete all pins


See the full report at /Users/sgtpooki/code/work/protocol.ai/ipfs/pinning-service-compliance/pinning-service-compliance/docs/api.pinata.cloud.md

> @ipfs-shipyard/pinning-service-compliance@0.0.3 start

...

https://api.web3.storage compliance:

Summary (5/15 successful)

  ✘ Request with no authentication token
  ✘ Request with invalid token
  ✘ Pins post of CID 'bafkqadjrgy2tcmbvgi4tgmzygi3q'
  ✘ Can create and then delete a new pin
  ✘ List pin objects (GET /pins) in all states
  ✓ Can create and replace a pin's CID
  ✓ Can create a pin with name='3596f170-5ed2-43e8-bd41-e1f21a6a22fa'
  ✘ Can retrieve pin with name '3596f170-5ed2-43e8-bd41-e1f21a6a22fa' via the 'exact' TextMatchingStrategy
  ✘ Can retrieve pin with name '3596F170-5ED2-43E8-BD41-E1F21A6A22FA' via the 'iexact' TextMatchingStrategy
  ✘ Can retrieve pin with name '5ed2-43e8-bd41-e1f' via the 'partial' TextMatchingStrategy
  ✘ Can retrieve pin with name '5ED2-43E8-BD41-E1F' via the 'ipartial' TextMatchingStrategy
  ✓ Get all pins
  ✓ First page of pins
  ✘ Retrieve the next page of pins
  ✓ Can delete all pins


See the full report at /Users/sgtpooki/code/work/protocol.ai/ipfs/pinning-service-compliance/pinning-service-compliance/docs/api.web3.storage.md

> @ipfs-shipyard/pinning-service-compliance@0.0.3 start

...

https://nft.storage/api compliance:

Summary (9/20 successful)

  ✘ Request with no authentication token
  ✘ Request with invalid token
  ✘ Pins post of CID 'bafkqadjrgy2tcmbvgmydambvha2q'
  ✓ Can create and then delete a new pin
  ✘ List pin objects (GET /pins) in all states
  ✓ Can create and replace a pin's CID
  ✓ Can create a pin with name='9e2ec2c7-25a5-4316-8285-2f33aa0ecebf'
  ✘ Can retrieve pin with name '9e2ec2c7-25a5-4316-8285-2f33aa0ecebf' via the 'exact' TextMatchingStrategy
  ✘ Can retrieve pin with name '9E2EC2C7-25A5-4316-8285-2F33AA0ECEBF' via the 'iexact' TextMatchingStrategy
  ✘ Can retrieve pin with name '25a5-4316-8285-2f3' via the 'partial' TextMatchingStrategy
  ✘ Can retrieve pin with name '25A5-4316-8285-2F3' via the 'ipartial' TextMatchingStrategy
  ✓ Get all pins
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmydgnztge2a'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmydimjvgm2q'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmydinjygq4q'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmydkmjsgazq'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmydknjxgq2a'
  ✘ First page of pins
  ✘ Retrieve the next page of pins
  ✘ Can delete all pins


See the full report at /Users/sgtpooki/code/work/protocol.ai/ipfs/pinning-service-compliance/pinning-service-compliance/docs/nft.storage.md

> @ipfs-shipyard/pinning-service-compliance@0.0.3 start

...

https://api.estuary.tech/pinning compliance:

Summary (18/26 successful)

  ✘ Request with no authentication token
  ✘ Request with invalid token
  ✘ Pins post of CID 'bafkqadjrgy2tcmbvgmydqmbqge4a'
  ✘ Can create and then delete a new pin
  ✘ List pin objects (GET /pins) in all states
  ✘ Can create and replace a pin's CID
  ✓ Can create a pin with name='16ecf9ed-a0b1-4841-a5d3-6225954f0943'
  ✓ Can retrieve pin with name '16ecf9ed-a0b1-4841-a5d3-6225954f0943' via the 'exact' TextMatchingStrategy
  ✓ Can retrieve pin with name '16ECF9ED-A0B1-4841-A5D3-6225954F0943' via the 'iexact' TextMatchingStrategy
  ✓ Can retrieve pin with name 'a0b1-4841-a5d3-622' via the 'partial' TextMatchingStrategy
  ✓ Can retrieve pin with name 'A0B1-4841-A5D3-622' via the 'ipartial' TextMatchingStrategy
  ✓ Get all pins
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmytanrxg42a'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmytaojzga3a'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmytcmzsge3q'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmytcnrvgu2a'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmytcojzha3a'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmytemzugi3q'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmytenrwg4ya'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmytgmbrha4q'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmytgmzvgiyq'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmytgnzqgiyq'
  ✓ Can create new pin for testing pagination cid='bafkqadjrgy2tcmbvgmytimbvge2q'
  ✘ First page of pins
  ✓ Retrieve the next page of pins
  ✘ Can delete all pins


See the full report at /Users/sgtpooki/code/work/protocol.ai/ipfs/pinning-service-compliance/pinning-service-compliance/docs/api.estuary.tech.md

@SgtPooki SgtPooki self-assigned this Apr 27, 2022
@SgtPooki SgtPooki pinned this issue Apr 27, 2022
@SgtPooki SgtPooki added this to the Initial release milestone May 9, 2022
@SgtPooki
Copy link
Member Author

ive been able to improve one area where web3.storage was failing silently and therefore not emitting the correct final counts, but it's doing it again. I have the rest of them working properly:

npm start -- -s $PINATA_API_ENDPOINT $PINATA_API_TOKEN | grep -A 3 'The total counts'
npm start -- -s $WEB3_API_ENDPOINT $WEB3_API_TOKEN | grep -A 3 'The total counts'
npm start -- -s $ESTUARY_API_ENDPOINT $ESTUARY_API_TOKEN | grep -A 3 'The total counts'
npm start -- -s $NFT_API_ENDPOINT $NFT_API_TOKEN | grep -A 3 'The total counts'
(node:60828) ExperimentalWarning: The Fetch API is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
2022-05-18T21:30:29.557Z [info] : The total counts for this run are:
    Total Expectations          68
    Total ApiCall instances     33
    Total FirstClass ApiCalls   28
(node:60907) ExperimentalWarning: The Fetch API is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:60938) ExperimentalWarning: The Fetch API is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
2022-05-18T21:31:13.772Z [info] : The total counts for this run are:
    Total Expectations          68
    Total ApiCall instances     33
    Total FirstClass ApiCalls   28
(node:60969) ExperimentalWarning: The Fetch API is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
2022-05-18T21:32:16.576Z [info] : The total counts for this run are:
    Total Expectations          68
    Total ApiCall instances     33
    Total FirstClass ApiCalls   28

This was referenced May 19, 2022
Open
Open
Closed
@SgtPooki SgtPooki linked a pull request May 20, 2022 that will close this issue
fix: Compliance check sum is consistent for all services #52
Closed
@SgtPooki SgtPooki mentioned this issue May 20, 2022
Merged
@SgtPooki
Copy link
Member Author

SgtPooki commented May 24, 2022 •
edited

summation is fixed with PR #54

npm run dev-start -- -s $PINATA_API_ENDPOINT $PINATA_API_TOKEN -d |& grep -A 3 'The total counts'
npm run dev-start -- -s $WEB3_API_ENDPOINT $WEB3_API_TOKEN -d |& grep -A 3 'The total counts'
npm run dev-start -- -s $ESTUARY_API_ENDPOINT $ESTUARY_API_TOKEN -d |& grep -A 3 'The total counts'
npm run dev-start -- -s $NFT_API_ENDPOINT $NFT_API_TOKEN -d |& grep -A 3 'The total counts'
2022-05-20T23:51:22.786Z [debug] : The total counts for this run are:
    Total Expectations          81
    Total ApiCall instances     47
    Total FirstClass ApiCalls   13
2022-05-20T23:52:37.827Z [debug] : The total counts for this run are:
    Total Expectations          81
    Total ApiCall instances     48
    Total FirstClass ApiCalls   13
2022-05-20T23:53:30.336Z [debug] : The total counts for this run are:
    Total Expectations          77
    Total ApiCall instances     45
    Total FirstClass ApiCalls   13
2022-05-20T23:54:29.167Z [debug] : The total counts for this run are:
    Total Expectations          62
    Total ApiCall instances     35
    Total FirstClass ApiCalls   13

@SgtPooki SgtPooki unpinned this issue Jun 1, 2022
@github-actions
Copy link

🎉 This issue has been resolved in version 1.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@SgtPooki SgtPooki

Labels
effort/hours exp/beginner good first issue Good for newcomers help wanted Extra attention is needed kind/architecture kind/bug need/analysis P2 released status/in-progress
Projects
None yet
Milestone
Initial release
1 participant
@SgtPooki