Feat: API Consumer Testing

[nelsonic]

As part of building our REST API #256 we want to adopt a Software Engineering-approach/mindset
and add automated testing to GitHub CI.

Note: the this is not the Elixir (ExUnit) tests which are part of building the feature.
Rather, what we want is testing from the perspective of the person consuming the API.

As noted by @LuchoTurtle's research/notes into API Testing: github.com/dwyl/learn-api-design#api-testing
There are several options to chose from in this space.

Requirements

1. Must be Open Source and capable of running on GitHub Actions/CI. @apardo04's locust-load-test looks like a good candidate (as noted by @LuchoTurtle 👌)
2. Should run after all the other steps (mix format, mix test, coveralls) but should fail the build if the API is not working as expected.
3. Should either run headless by booting the Phoenix App on GitHub CI and then running the tests against the server or should run against the deployed Review App.
4. Should NOT increase build time by more than 10 seconds; we want to test our API but don't want to slow down our velocity. We will need to setup
5. Should Record API Test Results. Initially it's fine for these to just be in the build logs. But ideally, a comment should be posted to the PR with a summary of the API test. And then we should also start recording them in a secondary format e.g. JSON on S3 so that we can aggregate the data and run stats to know if/when things slow down. 💭

Todo

• Document stetting up https://locust.io in API.md for our basic API [PR] Stage 1 - No auth, anonymous API for Timers and Items #263
• Create the locustfile.py with lots of comments to help a complete beginner understand it
• Create a PR that executes the locust-load-test GitHub Action against the unauthenticated API.
• Prepare for doing authenticated API Requests as part of "Stage 3" (but leave these as stubs we will return to a the end of the week). 👌

[nelsonic]

@LuchoTurtle please bas this PR off #263 which will be merged shortly. 👌

[LuchoTurtle]

Posting this so I don't forget.

https://github.com/hoppscotch/hoppscotch/tree/main/packages/hoppscotch-cli should be considered for running API definition tests and then Locust for load testing the API. Locust is just a stress loader. We should run assertions on common test scenarios for each request.

However, Hoppscotch CLI is still in alpha, which is quite concerning as breaking changes will occur rapidly. Perhaps using Postman (which is far more mature) is better for short/medium-term 🤔 . We fall under the "Free" pricing range of Postman quite easily and "wait" until Hoppscotch's CLI matures.

[nelsonic]

@LuchoTurtle indeed Hoppscotch looks like a near-clone of Postman.

What features does Postman offer (that we need) that aren't already in Hoppscotch?
My reading of the Hoppscotch docs suggests it already has everything we need.
We don't need "teams" we just need a collection that can be committed to GitHub.

Yes, the CLI https://docs.hoppscotch.io/cli is "alpha":

But there are enough "eyes" on it and it's based on the Go version: https://github.com/hoppscotch/hopp-cli which looked pretty feature-complete ... 💭

My biggest gripe with Postman is that it's nothing special, it's very Closed Source and they charge a bunch for it!
This kind of useful Dev tool should be Open Source. Even if they then charge Companies money for a Hosted version ...

As noted in: dwyl/technology-stack#117 (comment) if you need to use Postman to get your work done faster, go for it.
But if you have T2h to transfer your Postman knowledge and learn Hopscotch, that would benefit all of us. 💭

[LuchoTurtle]

I'm much more inclined to use Hoppscotch, it's FOSS and it's great. I just thought the CLI being in alpha could cause issues, that's all. If it doesn't bother you, I don't mind.
Even though it's super similar to Postman, Postman is more complete, the CI integration is seamless and matured and allows monitoring, which Hoppscotch doesn't (just simple collections and tests).

Again, I don't mind Hoppscotch and I will use it from now on, just thought that being "less-mature" would be an issue to you.

I'm going to create tests in Hoppscotch and try to integrate CI with their CLI.

[nelsonic]

Prefer Open Source to maturity in most cases. Look forward to seeing the tests. 👌

[LuchoTurtle]

An update on this:

• had some trouble installing https://docs.hoppscotch.io/cli. Was getting errors related to permissions but even running with sudo and --forced wasn't working. I found later this was because my npm install was corrupted. Reinstalling and restarting fixed this :|

• I've created tests on all possible scenarios in Hoppscotch.

• Running hopps test correctly runs the tests with the server running on localhost.

I'm now attempting to use https://github.com/nektos/act to run and test Github Actions locally. But it's failing trying to Setup elixir :/

[LuchoTurtle]

The problem seems to be related to erlef/setup-beam#179 and https://stackoverflow.com/questions/71040681/qemu-x86-64-could-not-open-lib64-ld-linux-x86-64-so-2-no-such-file-or-direc.

Producing ARM builds, due to using a M1 Mac, seems to be the culprit.

[LuchoTurtle]

Surpassed the issue by running with a --container-architecture linux/amd64 argument but I still get errors.

[nelsonic]

@LuchoTurtle please don't waste time with trying to run the GitHub Actions on your localhost.
While it's a super-nice-to-have, it's waaaay beyond the scope of this exercise.
If you can run the tests using the CLI on your localhost that's great.
Then let GitHub Actions run the rest.

I've never run the codecov action on my localhost, it's always run on GitHub.
That should be the same for the API testing.

[nelsonic]

Even if you have to do 100 commits to get this working and it takes 2 mins to run each one on GitHub Actions,
it's better than waiting for this hurdle to be overcome when you have a "brick wall" ("Doesn't work on M1 Mac ... 🙄)

[nelsonic]

Let GitHub Actions run the scripts and focus on the work that you can do on localhost. 🙏

[nelsonic]

According to the Hoppscotch Docs: https://github.com/hoppscotch/hoppscotch
the web app is "installable" as a PWA:

Review of: https://discussions.apple.com/thread/254319263?answerId=258100088022#258100088022

Indicates that this only works in Google Chrome and Microsoft Edge ...
while it's lame that Safari doesn't yet have PWA Support on Mac,
We do have Chrome and Edge [for dev reasons] on our Macs,
so that's the approach I'm taking to testing this out.

That will create what looks like a "Native" App on your Mac:

Which then opens full-screen an feels Native:

And you're off to the races!

Having the PWA will significantly increase your dev speed because you can easily Alt+Tab between your IDE and Hoppscotch and not have to hunt for a Tab in your Web Browser.

[nelsonic]

.hoppscotch directory is hidden from finder:

is this naming convention required for the CI tests to work?
could we just call the folder /api and have the same effect?

[nelsonic]

This is lame ...

[LuchoTurtle]

.hoppscotch was intentionally named that way to differentiate CI from project files.
Naming it api might generate confusion because people will think the API project files will reside there, instead of lib/app_web/api.
Using VSCode with this change also helps things being organized.

It doesn't show in your Finder because it's hiding hidden files.
https://www.pcmag.com/how-to/how-to-access-your-macs-hidden-files#:~:text=View%20Hidden%20Files%20in%20Terminal&text=Find%20Terminal%20under%20Launchpad%20%3E%20Other,AppleShowAllFiles%20true%20and%20press%20Enter

You just have to type Command + Shift + . (period) to show these files.
I did this on day one, it's easier to manage .git folders, as well.

[nelsonic]

Yeah, I know about enabling hidden files in Mac. don't worry.
My concern is adding extra steps for other people where they aren't required ... 🤷‍♂️

[LuchoTurtle]

In that case, I believe renaming to something along the lines of "_hoppscotch" is more intuitive, as it differentiates project files from CI files and shows in Finder.

[nelsonic]

Google Chrome install PWA:

Creates a native-looking App with an icon you can Alt+Tab to:

Hoppscotch empty:

Open existing collection:

Import from hoppscotch:

Collection imported:

If you see the following "environment not found":

You need to load the envs.json environment variables file....

[nelsonic]

Even after you have imported the environment configuration file, it's not automatically selected:

You need to manually select Localhost.
With the "Environments" tab selected, click the "Select environment" selector and chose "Localhost":

[nelsonic]

Via: https://github.com/hoppscotch/hoppscotch/wiki/Environments#accessing-environment-data

[nelsonic]

After reading through: hoppscotch/hoppscotch#1425 ⏳

[nelsonic]

@LuchoTurtle did you have to add the Hoppscotch Browser Extension to your Google Chrome to get this working on localhost? 🤷‍♂️

[nelsonic]

After importing the collection and environment, it still won't work ...

You will see the message:

Could not send request.

Unable to reach the API endpoint. Check your network
connection or select a different interceptor and try again.

These are the available options:

If you select "Browser extension" it will open the Chrome web store where you can install the extension.

Install the extension, and once installed add the the http://localhost:4000 origin:

Then the presence of the extension will be visible in the Hoppscotch window/UI:

Now the requests in the collection will work:

[nelsonic]

@LuchoTurtle judging from your screenshots, you appear to have the Hoppscotch Chrome Extension installed:

[nelsonic]

Having this documented in the API.md would have saved me a chunk of time. ⏳ 😢

[nelsonic]

This is how I eventually discovered that this was required: hoppscotch/hoppscotch#2256 (comment)

[nelsonic]

Tried to install the CLI:

npm i -g @hoppscotch/cli

Got the error:

npm ERR! code EACCES
npm ERR! syscall mkdir
npm ERR! path /usr/local/lib/node_modules/@hoppscotch
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/@hoppscotch'
npm ERR!  [Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/@hoppscotch'] {
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'mkdir',
npm ERR!   path: '/usr/local/lib/node_modules/@hoppscotch'
npm ERR! }
npm ERR!
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR!
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/n/.npm/_logs/2023-01-22T02_32_54_836Z-debug-0.log

[nelsonic]

As per: https://stackoverflow.com/a/51024493/1148249 ran:

sudo chown -R $USER /usr/local/lib/node_modules

Got a different error:

n@mbp16 mvp % npm i -g @hoppscotch/cli
npm ERR! code EACCES
npm ERR! syscall symlink
npm ERR! path ../lib/node_modules/@hoppscotch/cli/bin/hopp
npm ERR! dest /usr/local/bin/hopp
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, symlink '../lib/node_modules/@hoppscotch/cli/bin/hopp' -> '/usr/local/bin/hopp'
npm ERR!  [Error: EACCES: permission denied, symlink '../lib/node_modules/@hoppscotch/cli/bin/hopp' -> '/usr/local/bin/hopp'] {
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'symlink',
npm ERR!   path: '../lib/node_modules/@hoppscotch/cli/bin/hopp',
npm ERR!   dest: '/usr/local/bin/hopp'
npm ERR! }
npm ERR!
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR!
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/n/.npm/_logs/2023-01-22T02_38_22_574Z-debug-0.log

[nelsonic]

Ran:

sudo chown -R $USER /usr/local/bin

Followed by:

npm i -g @hoppscotch/cli

Then finally:

hopp test -e ./lib/api/localhost.json ./lib/api/MVP.json

[LuchoTurtle]

@nelsonic Thank you for the feedback on the PR and your issues, it is thoroughly appreciated!
I find the issues you ran into super weird because I had no issues installing hoppscotch nor had to manually install the extension in Chrome. Otherwise, I would certainly write it in the onboarding experience in api.md, it seems like a pain!

I have the extension installed, you are correct, but I did not manually do it. My localhost requests were working normally from the get-go, which is why I find it weird you had trouble getting up and running. Perhaps it was installed when I started using the PWA?

Regarding importing environments and actually changing them, it's my fault it wasn't clear enough. I've grown so accustomed to using Postman that I assume that someone would need to change environments to get the variables to work. That's my fault, should have documented that process more thoroughly/shoshinier (?).

Regarding the "three small pictures", it's totally my fault. I use the preview in VSCode to write the README and get instant feedback and so far it has been faithful to Github's markdown.

However, as you duly noted, it wasn't working. So I'm going to double-check if the README is proper on GitHub as well, instead of relying on VSCode's preview.

Also, the pictures to me looked correct side-by-side but I was working on a much bigger screen, should have thought that on laptops it wasn't discernable. Lesson learned! 😅

Again, thanks for the feedback and for adding the much-needed extra information that was lacking. Will surely pay more attention next time (especially with new "epic" sections like "API development").

[nelsonic]

@LuchoTurtle similarly, I'm working on a 4K screen.
The pictures are tiny and having them side-by-side forces the person to click to open them.
They open in a new tab which loses the contextual text and forces the person to manually close the tab.
Again. if our Manifesto says:

Your time should never be wasted.

Then this is breaking a promise.
Please think about the time of the person reading whatever you write.
If it's you're causing them unnecessary effort, you're stealing their time.

Furthermore, I don't care how good it looks on your screen or VSCode Preview.
"As a reader" of this documentation, I'm not going to use your screen to read the doc or VSCode, am I?
All I care about is what it will look like on GitHub:
https://github.com/dwyl/mvp/blob/b0d58f55d6a6268e607685c79f3f98be77c34ad2/API.md

It's not side-by-side it's scrolling and tiny.
GitHub markdown parser is garbage. But it's what we are stuck with, for now.

And when I click on one of the images:

There's so much whitespace in the image I'm drownging in it.
Constrain your screens before taking the screenshot.
The relevant information of this screenshot is:

That's an 80% reduction.
The Details Matter.
If nobody has told you this before; that's unfortunate. But I'm telling you now. Pay close attention to detail. 🔍
Or be confined to mediocrity. 🤷‍♂️

[nelsonic]

If things magically work for you without any extra effort that's great. 🎉
However the far more likely reality is that you are documenting things retrospectively
and thus skipping steps because you cannot remember them ... 💭
If you document things while you are learning/implementing them
you avoid all of this time-wasting. (My Time and Yours)

[nelsonic]

Hoppscotch testing added by @LuchoTurtle via: #276
Thanks again. 👌