Merge pull request #103 from dialex/framework/karate

Framework review: karate

.spelling

@@ -86,6 +86,8 @@ png
 descope
 kanban
 onboarding
+whitespace
+standalone
  - docs/toolbox/note-taking.md
 udos
 deas

code/framework/README.md

@@ -2,39 +2,42 @@
 
 ## Review criteria
 
-| Category | Opinion | Score |
-| -------- | ------- | :---: |
-| _Use cases_        |  | ❓ |
-| _Learning curve_   |  | ❓ |
-| _Language_         |  | ❓ |
-| _Ecosystem_        |  | ❓ |
-| _Readability_      |  | ❓ |
-| _Extensibility_    |  | ❓ |
-| _Maintainability_  |  | ❓ |
-| _Documentation_    |  | ❓ |
-| **VERDICT**        |  | **_/5** |
+| Category          | Opinion |  Score   |
+| ----------------- | ------- | :------: |
+| _Use cases_       |         |    ❓    |
+| _Learning curve_  |         |    ❓    |
+| _Language_        |         |    ❓    |
+| _Ecosystem_       |         |    ❓    |
+| _Readability_     |         |    ❓    |
+| _Extensibility_   |         |    ❓    |
+| _Maintainability_ |         |    ❓    |
+| _Documentation_   |         |    ❓    |
+| **VERDICT**       |         | **\_/5** |
 
 ## Automation scope: [Todo list](http://todomvc.com/examples/react/#/)
 
-- **Sanity Tests**
-  - Run a test with an assertion
-- **Smoke Tests**
-  - Access homepage on browser
-- **Functional Tests**
-  - Add item to list
-  - Mark item as done
-  - Edit item text
-  - Remove item from list
+- [ ] **Smoke Tests**
+  - [ ] Access homepage on browser
+- [ ] **Functional Tests**
+  - [ ] Add item to list
+  - [ ] Mark item as done
+  - [ ] Edit item text
+  - [ ] Remove item from list
 
 ## Automation scope: [Amazon](https://amazon.com/)
 
-- **Sanity Tests**
-  - Run a test with an assertion
-- **Smoke Tests**
-  - Access homepage on browser
-- **Functional Tests**
-  - Search for book
-  - Add book to cart
+- [ ] **Smoke Tests**
+  - [ ] Access homepage on browser
+- [ ] **Functional Tests**
+  - [ ] Search for book
+  - [ ] Add book to cart
+
+## Automation scope: [Postman API](https://docs.postman-echo.com/)
+
+- [ ] Successful GET
+- [ ] Successful POST
+- [ ] Failed method
+- [ ] Assert response structure
 
 ## TODO
 

code/framework/cypress/Makefile

@@ -1,4 +1,5 @@
 install:
+	echo "⚠️  Assuming you have Node.JS and NPM installed..."
 	npm install cypress
 
 clean:

code/framework/karate/.gitignore

@@ -0,0 +1,2 @@
+target/
+karate.jar

code/framework/karate/Makefile

@@ -0,0 +1,9 @@
+install:
+	echo "⚠️  Assuming you have Java 8 or above installed..."
+	curl -L https://dl.bintray.com/ptrthomas/karate/karate-0.9.5.jar -o karate.jar
+
+clean:
+	rm -rf target/
+
+test: clean
+	java -jar karate.jar --threads=4 src/test/karate --clean

code/framework/karate/README.md

@@ -0,0 +1,25 @@
+# Karate
+
+Review [here](https://github.com/dialex/start-testing/tree/master/docs/toolbox/framework/karate.md).
+
+## Commands
+
+```sh
+make install  # Install dependencies
+make test     # Run tests
+```
+
+## Automation scope: [Postman API](https://docs.postman-echo.com/)
+
+- [x] Successful GET
+- [x] Successful POST
+- [x] Failed method
+- [x] Assert response structure
+
+## Automation scope: Amazon
+
+- [x] **Smoke Tests**
+  - [x] Access homepage on browser
+- [x] **Functional Tests**
+  - [x] Search for item
+  - [x] Add item to cart

code/framework/karate/src/test/fixtures/postman-post-payload.json

@@ -0,0 +1,12 @@
+{
+  "some-field": "some value",
+  "another-field": 2,
+  "list-of-stuff": [
+    {
+      "name": "stuff1"
+    },
+    {
+      "name": "stuff2"
+    }
+  ]
+}

code/framework/karate/src/test/karate/api-postman-contract.feature

@@ -0,0 +1,17 @@
+Feature: Postman Echo GET
+
+  Background:
+    * url 'https://postman-echo.com'
+
+  # Inspired by https://github.com/intuit/karate/blob/master/karate-junit4/src/test/java/com/intuit/karate/junit4/demos/schema-like.feature
+  Scenario: Contract of /get response
+
+    Given path "/get"
+    And param word1 = "Domo arigato"
+    And param word2 = "Sayonara"
+    And def schema =
+      """
+      { args: { word1: #string, word2: #string } }
+      """
+    When method get
+    Then match response contains schema

code/framework/karate/src/test/karate/api-postman-get.feature

@@ -0,0 +1,27 @@
+Feature: Postman Echo GET
+
+  Background:
+    * url 'https://postman-echo.com'
+    * def endpoint = "get"
+
+  Scenario: Endpoint /get exists
+
+    Given path endpoint
+    When method get
+    Then status 200
+
+  Scenario: Returns the parameters received
+
+    Given path endpoint
+    And param param1 = "Hajime"
+    And param arg2 = "Dozo"
+    When method get
+    Then status 200
+    And match response == "#notnull"
+    And match response.args.param1 == "Hajime"
+    # Assertion using JsonPath (see https://stackoverflow.com/questions/62280561/)
+    # Option 1: compare JsonPath result with an array (which is misleading)
+    And match $..arg2 == ["Dozo"]
+    # Option 2: store JsonPath result in tmp var (which is cumbersome)
+    * def tmpKarateSupportForJsonPathIsPooPoo = $..arg2
+    And match tmpKarateSupportForJsonPathIsPooPoo[0] == "Dozo"

code/framework/karate/src/test/karate/api-postman-post.feature

@@ -0,0 +1,33 @@
+Feature: Postman Echo POST
+
+  Background:
+    * url 'https://postman-echo.com'
+    * def endpoint = "post"
+
+  Scenario: Endpoint /post exists
+
+    Given path endpoint
+    And request {}
+    When method post
+    Then status 200
+
+  Scenario: Returns the payload received
+
+    Given path endpoint
+    And request read('../fixtures/postman-post-payload.json')
+    When method post
+    Then status 200
+    And match response.data.some-field == "some value"
+    And match response.data.list-of-stuff[1].name == "stuff2"
+
+  # this is actually the "get started" example from the docs...
+  #   you know, the kind of code you want to always work,
+  #   because it's your first interaction with the tool (ಠ _ ಠ)
+  Scenario: Failed request returns 500
+
+    Given url "http://myhost.com/v1/cats"
+    And request { name: "Billie"}
+    When method post
+    Then status 500
+# Then status 201
+# And match response == { id: "#notnull", name: "Billie" }

code/framework/karate/src/test/karate/ui-amazon.feature

@@ -0,0 +1,38 @@
+Feature: Amazon UI automation
+
+  Background:
+    * def baseUrl = 'https://www.amazon.com/'
+    * configure driver = { type: 'chrome' } # this is optional
+
+  Scenario: Access homepage on browser
+    Given driver baseUrl
+    Then driver.url == baseUrl
+    And assert locate("#twotabsearchtextbox").exists
+    And assert locate("{}Cart").exists
+
+  Scenario: Search for item
+    Given driver baseUrl
+    And def searchTerm = "Explore It"
+    And def searchTermEncoded = "Explore+It"
+    # This one-liner should work, but doesn't ¯\_(ツ)_/¯
+    # When input("#twotabsearchtextbox", [searchTerm, Key.ENTER])
+    When input("#twotabsearchtextbox", searchTerm)
+    # This should work, but doesn't
+    # And click("#nav-search-submit-text")
+    And mouse("#nav-search-submit-text").click()
+    # This should not be necessary, but it is (╯°□°）╯︵ ┻━┻
+    And waitFor('#nav-search-submit-text').click()
+    Then match driver.url contains "s?k=" + searchTermEncoded
+    And locate("div.s-result-list").exists
+    And locate("{span}Explore It!: Reduce Risk and Increase Confidence with Exploratory Testing").exists
+  # Read more about this issue at https://github.com/intuit/karate/issues/1169
+
+  # FIXME: I can't get this to work, I give up (╯°□°）╯︵ ┻━┻
+  # Scenario: Add item to cart
+  #   Given driver baseUrl
+  #   And def searchTerm = "Cypress"
+  #   And input("#twotabsearchtextbox", searchTerm)
+  #   And mouse('#nav-search-submit-text').click()
+  #   When mouse(".s-result-item .s-image").click()
+  #   And mouse('#add-to-cart-button').click()
+  #   Then match text("#nav-cart-count") == 1

code/framework/robot/Makefile

@@ -1,6 +1,5 @@
 install:
-	easy_install pip
-	brew install python2
+	echo "⚠️  Assuming you have Python, pip and brew installed..."
 	pip install robotframework
 	pip install robotframework-seleniumlibrary
 	pip install robotframework-requests

docs/_sidebar.md

@@ -55,6 +55,7 @@
   - ~~Staying up to date~~
 - **EXTRA: Test Frameworks**
   - [Cypress](/toolbox/framework/cypress.md)
+  - [Karate](/toolbox/framework/karate.md)
   - [Robot Framework](/toolbox/framework/robot.md)
 - **EXTRA: Notebook**
   - [User Story Mapping](/notebook/user-story-mapping.md)

docs/toolbox/framework/cypress.md

@@ -17,7 +17,7 @@ Example of automation at [GitHub](https://github.com/dialex/start-testing/tree/m
 | _Language_         | Tests are written in Javascript, thus some basic knowledge is required to code and understand the tests. You will only take full advantage of this powerful framework if you are proficient with Javascript though. | 🥈 |
 | _Ecosystem_        | Javascript. I used VS Code to write tests and (after a simple configuration) it's IntelliSense made me very productive. | 🥇 |
 | _Readability_      | A non-tech person can only understand the title of each test. E2E tests easily become hard to read even for devs. There are unofficial libs to support Gherkin. By default there's no test report file, only an ASCII output, but you can add any [Mocha](https://docs.cypress.io/guides/tooling/reporters.html#Custom-Reporters) test reporter. | 🥈 |
-| _Extensibility_    | You can create your own Cypress Commands (not picked up by IntelliSense) or regular Javascript functions. PageObjects is a nightmare (their [recommendation](https://docs.cypress.io/faq/questions/using-cypress-faq.html#Can-I-use-the-Page-Object-pattern) isn't better). There are several Cypress plugins (but they offer [functionality that should be built-in](https://github.com/cypress-io/cypress/issues/1865#issuecomment-484897559)). | 🥉 |
-| _Maintainability_  | It's Javascript, for better or worse. Debugging is average (pause, go back in time, DOM inspection), I longed for step-by-step. IntelliSense usually guides your coding. Cypress commands run asynchronously and that leads to issues (e.g. run conditions) and limitations (e.g. can't mix sync and async code). | 🥉 |
+| _Extensibility_    | You can create your own Cypress Commands (not picked up by IntelliSense) or regular Javascript functions. PageObjects is a nightmare (their [recommendation](https://docs.cypress.io/faq/questions/using-cypress-faq.html#Can-I-use-the-Page-Object-pattern) isn't better). There are several Cypress plugins (but they offer [functionality that should be built-in](https://github.com/cypress-io/cypress/issues/1865#issuecomment-484897559)). It's a pity that parallel test execution is a paid feature. | 🥉 |
+| _Maintainability_  | Debugging is good (pause, go back in time, DOM inspection), I longed for step-by-step. IntelliSense usually guides your coding. Cypress commands run asynchronously and that leads to issues (e.g. run conditions) and limitations (e.g. can't mix sync and async code). | 🥈 |
 | _Documentation_    | The online doc is abundant and comprehensive. It covers how to use Cypress, including some [recipes/examples](https://github.com/cypress-io/cypress-example-recipes#application-actions), but also explains some test concepts (e.g. when to use test mocks, anti-patterns, etc.). | 🥇 |
-| **VERDICT**        | Perfect for a Javascript project with an experienced team. | **4/5** |
+| **VERDICT**        | Perfect for a Javascript project with an experienced team. | **4.5/5** |

docs/toolbox/framework/karate.md

@@ -0,0 +1,21 @@
+# Karate
+
+> Karate is the only open-source tool to combine API test-automation, mocks, performance-testing and even UI automation into a single, unified framework. You don't have to compile (Java) code. Just write tests in a readable syntax.
+>
+> — [Official website](https://intuit.github.io/karate/)
+
+## Code
+
+Example of automation at [GitHub](https://github.com/dialex/start-testing/tree/master/code/framework/karate).
+
+| Category          | Opinion |  Score   |
+| ----------------- | ------- | :------: |
+| _Use cases_       | Automate API tests. It also supports performance and end-to-end (E2E) tests. Mobile testing is in the roadmap. |    🥇    |
+| _Learning curve_  | Steep. It's easy to understand how it works, but it lacks an official step-by-step tutorial where each main DSL keyword is introduced, one at a time. It feels like a framework made by and to "hardcore developers". The purpose of using Gherkin is to make the tools accessible to non-devs, yet I never felt productive or in control. |    🥈    |
+| _Language_      | Tests are written in [Karate's DSL](https://hackernoon.com/yes-karate-is-not-true-bdd-698bf4a9be39). It uses the syntax of Gherkin and it adds some custom keywords and operators. It also supports calls to Java and JavaScript, outside the Given-When-Then structure. |    🥇    |
+| _Ecosystem_       | The [VS Code extension](https://marketplace.visualstudio.com/items?itemName=kirkslota.karate-runner) didn't work at all (e.g. no autocomplete, no debugger, no run test button). In terms of community… there's not much since it's a recent tool. The author is an [enthusiastic](https://twitter.com/KarateDSL/status/1167533484560142336) salesperson which led to [some bad PR](https://twitter.com/jarbon/status/1136589061605416961), result of the "my tool is amazing, if you disagree you're against me" attitude. He's very active on Stack Overflow and GitHub though, replies within hours! |    🥉    |
+| _Readability_     | It's easier to read than to know what to write. Without autocomplete or good documentation it becomes hard to know how to use the Karate DSL to achieve what you want to do. Your tests use Gherkin's structure, but you have to mix programming syntax too (e.g. operators, selectors). This [sums the downsides](https://club.ministryoftesting.com/t/karate-for-test-automation-what-is-your-experience/39336/2) of natural (e.g. no refactoring) and programming (e.g. learning curve) languages. The CLI output is very verbose (without colours or whitespace) so it's hard. The HTML report doesn't clear results from previous test runs. Oh, and the DSL doesn't allow inline comments. |    😭    |
+| _Extensibility_   | You don't need plugins, since it follows the "batteries included" philosophy. Running tests in parallel is very easy. It includes Mocking, [code coverage](https://github.com/intuit/karate/tree/master/karate-demo#code-coverage-using-jacoco) and HTML reports (but it doesn't reset results between runs, even when using `--clean`, which makes the report useless). |    🥈    |
+| _Maintainability_ | The [*big difference*](https://intuit.github.io/karate/#cucumber-vs-karate) from Cucumber is that you *don't* need to write extra "step definitions" (but you end up mixing Gherkin with programming). You can't use PageObjects, but you can centralise selectors in a `locators.json` to achieve DRY.  I could not debug at all, zero of features [advertised here](https://twitter.com/KarateDSL/status/1167533484560142336) worked for me (Cypress, even [Selenium](https://hackernoon.com/the-world-needs-an-alternative-to-selenium-so-we-built-one-zrk3j3nyr), does a better job). On top of that, I got [incoherent test results on UI automation](https://stackoverflow.com/questions/62308044/karate-ui-automation-test-results-are-not-coherent). |    😭    |
+| _Documentation_   | Documentation is a single README file. It's verbose, exhaustive for beginners, hard to navigate and find what you are looking for. It feels like the official docs were written by the author but never revised from the perspective of a beginner (e.g. the provided quick start example fails; it can't be copied because it's an image; it was faster to skim blogs for examples and reverse-engineer the bits that I needed). |    🥉    |
+| **VERDICT**       | Useful for [API testing](https://docs.google.com/document/d/1ETTrdMVcBXaPjdKY-_67zCWBsi2Ctc5DIQUIfr02H7A/edit), because it uses a generic language and can be executed as a standalone. But don't be fooled, the target audience are developers and it's not an alternative for UI/E2E tests. | **3/5** |

docs/toolbox/tester-tools.md

@@ -219,7 +219,7 @@ A tool might not be listed if it is too expensive, closed-source, or not good en
 
 ## 🌍 LISTS OF RESOURCES
 
-- _[The "awesome" manifesto](https://github.com/sindresorhus/awesome/blob/master/awesome.md): enumerates the requirements for an awesomely curated list_
+- _[The "awesome" manifesto](https://github.com/sindresorhus/awesome/blob/main/readme.md): enumerates the requirements for an awesomely curated list_
 - ⭐️ [Awesome falsehood](https://github.com/kdeldycke/awesome-falsehood): biases that influence developers' decisions
 - [Awesome hacking](https://github.com/Hack-with-Github/Awesome-Hacking): repositories useful to hackers, pentesters and security researchers
 - 💯 [Awesome security](https://github.com/sbilly/awesome-security): tools and resources about security

package.json

@@ -1,6 +1,6 @@
 {
   "name": "start-testing",
-  "version": "0.3.0",
+  "version": "0.3.1",
   "description": "A crowdsourced testing course, written by testers for testers wannabes.",
   "scripts": {
     "bump:patch": "npm version --no-commit-hooks --no-git-tag-version patch",
@@ -24,7 +24,7 @@
     "lint:broken-links-internal": "remark -u validate-links docs --ignore-path .remarkignore --frail",
     "lint:broken-links-external": "remark -u remark-lint-no-dead-urls docs --frail",
     "pr:prepare": "npm-run-all fix lint site",
-    "pr:ready": "npm run bump:patch && npm run bump:time && git commit -m 'bump up version' .",
+    "pr:ready": "npm run bump:patch && npm run bump:time && git commit -m 'chore: bump up version' .",
     "preview": "echo \"Opening a preview on your browser now.\" && sleep 1 && open http://localhost:3000/#/",
     "site": "npm-run-all --parallel serve preview",
     "serve": "docsify serve docs",

time-counter.txt

@@ -28,3 +28,4 @@
 15: 535
 7: 800+80
 12: 1100
+102: 1000