A tool for automating acceptance tests at tiF!
Unit tests and integration tests are great, but they often do not test all the requirements. This is because those testing styles usually don’t test the software directly as a user would, including mocking many essential components.
A common style of addressing this is known as E2E testing, where you simulate button clicks on a live instance of your app. Such tests often look like this:
describe('The Login Page', () => {
beforeEach(() => {
// reset and seed the database prior to every test
cy.exec('npm run db:reset && npm run db:seed')
// seed a user in the DB that we can control from our tests
// assuming it generates a random password for us
cy.request('POST', '/test/seed/user', { username: 'jane.lane' })
.its('body')
.as('currentUser')
})
it('sets auth cookie when logging in via form submission', function () {
// destructuring assignment of the this.currentUser object
const { username, password } = this.currentUser
cy.visit('/login')
cy.get('input[name=username]').type(username)
// {enter} causes the form to submit
cy.get('input[name=password]').type(`${password}{enter}`)
// we should be redirected to /dashboard
cy.url().should('include', '/dashboard')
// our auth cookie should be present
cy.getCookie('your-session-cookie').should('exist')
// UI should reflect this user being logged in
cy.get('h1').should('contain', 'jane.lane')
})
})This tests the app as a user would, but it does not test that the business value is upheld. If a mobile app were to be developed alongside this website, the same test specification cannot be reused for testing the login functionality of the mobile app.
Acceptance testing tests business processes, not button clicks. A better and reusable test specification for the above looks like this:
New Test: Roswaal can Siiiiiiiign In
Abstract: Roswaal is a clown who wants to enter the ciiiiiiircus, he needs to enter with the secret paaaaaaaaassword!
Step 1: Roswaal is at the doooooooooor.
Requirement 1: Have Roswaal enter his naaaaaaaaaame.
Step 2: Roswaal is being blocked by securiiiiiiity.
Requirement 2: Have Roswaal enter his paaaaaaaaassword.
Step 3: Roswaal tries to access the ciiiiiiircus
Requirement 3: Verify that Roswaal signed in succeeeeeeeessfully.
This specification can be reused no matter how the app works. The user interface is not the app, but just an implementation detail of how it provides value.
As Roswaal, I turn this specification into code!
The tool works as a Slack Bot. This is how you can add a test!
- First, you use Slack, and compile a test like the one above. Simply use the
/add-testscommand! Make sure to include the test specification inside ``` brackets, or a code block! - Then, a PR is opened on the frontend reeeeeeeeeepo for you to review. This contains generated code that looks something like this. Over time, you code up the TODO functions by interacting with the app. This is usually done through detox.
// TestCase.test.ts
// Generated by Roswaal, do not touch.
import * as TestActions from "./TestActions"
import { launchApp } from "../Launch"
import { RoswaalTestCase } from "../TestCase"
import { roswaalClient } from "../Client"
test("Roswaal can Siiiiiiiign In", async () => {
const testCase = new RoswaalTestCase("Roswaal can Siiiiiiiign In", TestActions.beforeLaunch)
// Roswaal is at the doooooooooor.
testCase.appendAction(TestActions.haveRoswaalEnterHisNaaaaaaaaaame)
// Roswaal is being blocked by securiiiiiiity.
testCase.appendAction(TestActions.haveRoswaalEnterHisPaaaaaaaaassword)
// Roswaal tries to access the ciiiiiiircus
testCase.appendAction(TestActions.verifyThatRoswaalSignedInSucceeeeeeeessfully)
await roswaalClient.run(testCase)
})
// TestActions.ts
import { TestAppLaunchConfig } from "../Launch"
export const beforeLaunch = async (): Promise<TestAppLaunchConfig> => {
// Perform any setup work in here, (setting location, reseting device
// permissions, etc.)
return {}
}
export const haveRoswaalEnterHisNaaaaaaaaaame = async () => {
// Roswaal is at the doooooooooor.
throw new Error("TODO")
}
export const haveRoswaalEnterHisPaaaaaaaaassword = async () => {
// Roswaal is being blocked by securiiiiiiity.
throw new Error("TODO")
}
export const verifyThatRoswaalSignedInSucceeeeeeeessfully = async () => {
// Roswaal tries to access the ciiiiiiircus
throw new Error("TODO")
}- After approving the PR, and running the test, you can view its progress on Slack using the
/view-testscommand! - If you wish to remove the test, you can use
/remove-tests <test name>command. That will open another PR to remove the test!
Since our app heavily uses a map, the tool can automatically generate code to set the app in a certain location. When writing a test specification, use the Set Location: <location-name> command.
New Test: Set the Locaaaaaaation
Set Location: The Ciiiiiiiiircus
...
Make sure that the name of the location was added through the /add-locations command. That command will generate another PR to add the location to the list of known locations. For example, entering this into slack will create a new location PR that adds New York and Antarctica.
/add-locations
New York, 50.0, 50.0
Antacrtica, -12.1, -12.1
You can view all available locations using the /view-locations command!
Here are some examples of what a real test specification may look like.
New Test: Update Event
Abstract: Paul is planning to host an outdoor dance lesson in John McLaren Park. However, the latest weather forecast predicts heavy rain and possible flooding during the event. To ensure everyone's safety and comfort, Paul needs to change the time and location of the event and notify the attendees right away.
Set Location: John McLaren Park
Step 1: Paul creates outdoor dance lesson event in John McLaren Park next Sunday morning.
Step 2: Laura joins Paul's outdoor dance lesson event before it starts.
Step 3: Paul updates the time and location of the event to take place in Cayuga Park on Monday at noon.
Step 4: Laura is notified that the time and location of the event have changed.
Step 5: Laura views the updated event.
Requirement 1: Have Paul create the event "Outdoor Dance Lesson" in John McLaren Park next Sunday at 8:00 AM.
Requirement 2: Have Laura join the "Outdoor Dance Lesson" event.
Requirement 3: Have Paul update the time and location of the "Outdoor Dance Lesson" event to be in Cayuga Park on Monday at 12:00 PM.
Requirement 4: Verify that Laura was notified that "Outdoor Dance Lesson" was updated.
Requirement 5: Verify that Laura can view the updated event details from the notification.
This specification describes a real user scenario in which Paul schedules an event, and then changes his mind due to real-world external weather conditions. It does not describe how to implement the scenario, but by implementing the test case we ensure that the app can handle this scenario from a high-level.
To develop this tool, the first thing you’ll want to do after pulling is run ./setup.sh This pulls both the main frontend git repository, and a sandboxed git repository that is used during development and testing.
Afterwards, you’ll need to fill in a .env file with the variables shown in .env.example, this includes registering secrets to access the Slack and GitHub API respectively.
You can use the normal cargo commands for development/building/testing, but if you would like to cross compile the tool for linux, you can run cargo build --release --target=x86_64-unknown-linux-gnu. That command will cross-compile a release build for linux no matter what operating system you are using.
Functional Core, Imperative Shell is an act of separating “pure” code from impure code. In this case, it means separating side-effectless logic involving pure structs from the IO code. This allows easy reusability and testability of the complex logic, and it pushes the harder to test/control IO code out to integration tests.
Try to use this development style when writing code for this tool.
For the most part, this tool is ran as a web server using Axum, but the usage of Axum is only limited to the http folder. The code in the http folder calls directly into the code inside the operations folder, where the main logic happens.
Each main operation of this tool is represented as a separate file in the operations folder. Each operation returns a status enum, and that enum gets translated into an HTTP response code, or Slack Blocks depending on the operation. The operations orchestrate all the isolated modules of the system into the actual functionality provided by this tool.
The other folders are generally considered shared modules, and folders are organized by whatever is more convenient between features and technical concern.
Sqlite is used for storage as this is a simple internal tool, and the fact that Sqlite is incredibly easy to work with compared to larger RDBMS systems. Sqlx is used as the library to interact with the database, and particularly you will use the RoswaalSqlite struct which serializes every transaction against the database. This serialization helps to avoid SQLITE_BUSY errors.
The tool works with both Git, GitHub, and Slack to add tests. When a new test is compiled, its source code is generated and committed in a local repository using the git2 crate. The RoswaalGitRepository struct is used to interact with git, and particularly LibGit2RepositoryClient interacts with the git2 crate. LibGit2RepositoryClient runs a dedicated thread for all git repository actions. This is to avoid blocking IO on the tokio thread pool, and because git2 is not thread-safe.
All newly created git branches are unique by attaching a 10 character nano-id to the end of the branch name. This ensures that 2 branches do not clash with each other.
After generating the code for the test specification in the git repository, a pull request is opened using the Github API, and the local branch name opened by the repository is deleted. If the branch cannot be deleted, then a warning is posted in the slack output message.
When a new test is compiled, it is not considered in a “merged” state. Thus it is stored with its git branch name in the Sqlite database. After merging the PR opened by creating the test, the /merge endpoint is called to complete the merge, which will remove the branch name from the database record. This process allows us to have multiple tests with the same name on different branches if we need to decide between which one to merge. Likewise, we also invoke the /close endpoint if the PR is closed.
In the slack folder, you’ll find the application specific slack components for the tool, but if you look into the ui_lib subfolder, you’ll see a generic UI library for making slack views. The UI library takes a SwiftUI approach to making views, here’s an example:
pub struct MessageView<'v, Base: SlackView> {
base: &'v Base,
}
impl<'v, Base: SlackView> MessageView<'v, Base> {
pub fn new(base: &'v Base) -> Self {
Self { base }
}
}
impl<'v, Base: SlackView> SlackView for MessageView<'v, Base> {
fn slack_body(&self) -> impl SlackView {
If::is_true(
RoswaalEnvironement::current() == RoswaalEnvironement::Dev,
|| {
SlackSection::from_markdown(
"_This a test view!_",
)
},
)
.flat_chain_block_ref(self.base)
}
}We can see that like SwiftUI, we have a slack_body function which simply returns another view. This allows views to be written in a declarative style.
Any slack command that has to generate code, or edit the git repo is considered a long-running command. This is because those commands will take longer than the 3 seconds (due to all the network IO operations including pushing and pulling from the remote repository) allowed by Slack to return a response. For long running endpoints, a pending message is sent while the real work takes place in the background. Once the real work finishes, then the actual message is sent to Slack through the callback url given by the Slack request.
For non-long-running commands, no pending message is sent, and the slack command is processed normally.
You can find more by viewing the RoswaalSlackHandler trait.
We’ll repeatedly run the acceptance tests in isolation from this tool as they live on the frontend repo. After a test run, the frontend repo uploads the results to the /progress endpoint, and the progress gets stored in the database. You can use the /view-tests command on Slack to view the progress in an intelligent and formatted manner.