Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

api is not coherent with spec / api newman/postman tests are failing #6

Open
pandorasNox opened this issue May 7, 2022 · 0 comments

Comments

@pandorasNox
Copy link

pandorasNox commented May 7, 2022

I created an k8s setup as an exercise and it works pretty well.

However I also added a setup for the newman/postman tests, which shows that the api is not coherent with the spec

┌─────────────────────────┬───────────────────┬──────────────────┐
│                         │          executed │           failed │
├─────────────────────────┼───────────────────┼──────────────────┤
│              iterations │                 1 │                0 │
├─────────────────────────┼───────────────────┼──────────────────┤
│                requests │                32 │                0 │
├─────────────────────────┼───────────────────┼──────────────────┤
│            test-scripts │                48 │                4 │
├─────────────────────────┼───────────────────┼──────────────────┤
│      prerequest-scripts │                18 │                0 │
├─────────────────────────┼───────────────────┼──────────────────┤
│              assertions │               182 │              119 │
├─────────────────────────┴───────────────────┴──────────────────┤
│ total run duration: 17.2s                                      │
├────────────────────────────────────────────────────────────────┤
│ total data received: 1.96kB (approx)                           │
├────────────────────────────────────────────────────────────────┤
│ average response time: 14ms [min: 3ms, max: 120ms, s.d.: 24ms] │
└────────────────────────────────────────────────────────────────┘

as there are a lot of tests failing I just want to give an example of the first one

Conduit

❏ Auth
↳ Register
  POST [http://app-api:80/api/v1/users](http://app-api/api/v1/users) [201 Created, 193B, 120ms]
  ┌
  | response:
  │ 'user:', {
  │   email: 'u1651955558@mail.com',
  │   username: 'u1651955558'
  │ }
  └
  ✓  Response contains "user" property
  ✓  User has "email" property
  ✓  User has "username" property
  1. User has "bio" property
  2. User has "image" property
  3. User has "token" property
 
 [...]
 
     #  failure         detail                                                                           
                                                                                                        
 001.  AssertionError  User has "bio" property                                                          
                       expected false to be truthy                                                      
                       at assertion:3 in test-script                                                    
                       inside "Auth / Register"                                                         
                                                                                                        
 002.  AssertionError  User has "image" property                                                        
                       expected false to be truthy                                                      
                       at assertion:4 in test-script                                                    
                       inside "Auth / Register"                                                         
                                                                                                        
 003.  AssertionError  User has "token" property                                                        
                       expected false to be truthy                                                      
                       at assertion:5 in test-script                                                    
                       inside "Auth / Register"   

the response json was custome added in the collection via "console.log(\"user:\", user);",

when I change the server/user.go beginning in line 69 to e.g.

		user := conduit.User{
			Email:    input.User.Email,
			Username: input.User.Username,
			Bio:      "none",
		}

then the corresponding assertion (bio) succeeds

here is the test k8s job for reference:

---
apiVersion: batch/v1
kind: Job
metadata:
  name: check-app-api-newman
spec:
  backoffLimit: 0
  template:
    metadata:
      labels:
        app: check-app-api-newman
    spec:
      initContainers:
        - name: tests
          image: tests-tilt-ref
          env:
            - name: APP_API_HOST
              value: app-api
            - name: APP_API_PORT
              value: "80"
            - name: API_PATH
              value: /api/v1
          command:
            - ash
            - -c
            - |
              set -o errexit
              set -o nounset
              # set -o xtrace

              if set +o | grep -F 'set +o pipefail' > /dev/null; then
                # shellcheck disable=SC3040
                set -o pipefail
              fi

              if set +o | grep -F 'set +o posix' > /dev/null; then
                # shellcheck disable=SC3040
                set -o posix
              fi

              export APIURL=http://${APP_API_HOST}:${APP_API_PORT}${API_PATH}

              func_curl_check_endpoint() {
                TEST_ENDPOINT=${1:-http://localhost:80/healthz}

                echo "test endpoint: ${TEST_ENDPOINT}"
                curl -v --fail --silent -o /dev/null -X 'GET' \
                  --connect-timeout 1 \
                  --max-time 5 \
                  --retry 60 \
                  --retry-delay 1 \
                  --retry-max-time 120 \
                  "${TEST_ENDPOINT}" \
                  -H 'accept: application/json'
              }

              func_curl_check_endpoint "${APIURL}/health"

              echo api reachable/healthy

              # === api create user ======================================================

              REGISTRATION_JSON=$(cat <<END_HEREDOC
              {
                "user":{
                  "username": "Jacob",
                  "email": "jake@jake.jake",
                  "password": "jakejake"
                }
              }
              END_HEREDOC
              )

              printf 'REGISTRATION_JSON:\n%s\n' "${REGISTRATION_JSON}"

              # allowed to fail (not perfect but atm easiest way for user-already-exist kind of check)
              curl -v --header "Content-Type: application/json" \
                --request POST \
                --data "${REGISTRATION_JSON}" \
                "${APIURL}/users"

              echo ""

              echo done;
      containers:
        - name: test-api-newman
          image: tests-api-newman-tilt-ref
          env:
            - name: APP_API_HOST
              value: app-api
            - name: APP_API_PORT
              value: "80"
            - name: API_PATH
              value: /api/v1
            # - name: USERNAME
            #   value: Jacob
            # - name: EMAIL
            #   value: jake@jake.jake
            # - name: PASSWORD
            #   value: jakejake
          command:
            - ash
            - -c
            - |
              set -o errexit
              set -o nounset
              # set -o xtrace

              if set +o | grep -F 'set +o pipefail' > /dev/null; then
                # shellcheck disable=SC3040
                set -o pipefail
              fi

              if set +o | grep -F 'set +o posix' > /dev/null; then
                # shellcheck disable=SC3040
                set -o posix
              fi

              export APIURL=http://${APP_API_HOST}:${APP_API_PORT}${API_PATH}

              cd /api && ./run-api-tests.sh

              echo done
      restartPolicy: Never

Dockerfile

FROM docker.io/alpine:3.15.4 AS curl-tests

RUN apk update && apk add curl jq

# =============================================================================

FROM docker.io/postman/newman:5.3.1-alpine AS api-newman-tests

COPY ./api /api
RUN chmod +x /api/run-api-tests.sh

complet setup can be checked here:
https://github.com/pandorasNox/go-realworld

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant