Skip to content

Commit

Permalink
Make 1.20 fast (#103)
Browse files Browse the repository at this point in the history
  • Loading branch information
andrerfcsantos committed Mar 23, 2023
1 parent 3926fbd commit 35f10f2
Show file tree
Hide file tree
Showing 10 changed files with 49 additions and 42 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
- name: Install Go
uses: actions/setup-go@4d34df0c2316fe8122ab82dc22947d607c0c91f9
with:
go-version: 1.19.x
go-version: 1.20.x
- name: Checkout code
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
- name: Run linters
Expand All @@ -27,7 +27,7 @@ jobs:
if: success()
uses: actions/setup-go@4d34df0c2316fe8122ab82dc22947d607c0c91f9
with:
go-version: 1.19.x
go-version: 1.20.x
- name: Checkout code
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
- name: Run tests
Expand All @@ -40,7 +40,7 @@ jobs:
if: success()
uses: actions/setup-go@4d34df0c2316fe8122ab82dc22947d607c0c91f9
with:
go-version: 1.19.x
go-version: 1.20.x
- name: Checkout code
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
- name: Calc coverage
Expand Down
29 changes: 18 additions & 11 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,23 +1,30 @@
FROM golang:1.19.7-alpine3.17
FROM golang:1.20.2-alpine3.17

# add addtional packages needed for the race detector to work
RUN apk add --update build-base make
# Add addtional packages needed for the race detector to work
RUN apk add --update build-base make

# add a non-root user to run our code as
RUN adduser --disabled-password --gecos "" appuser
# Add a non-root user to run our code as
RUN adduser --disabled-password appuser

WORKDIR /opt/test-runner
COPY . .
# Copy the source code into the container
# and make sure appuser owns all of it
COPY --chown=appuser:appuser . /opt/test-runner

# Build and run the testrunner with appuser
USER appuser

# This populates the build cache with the standard library
# and command packages so future compilations are faster
RUN go build std cmd

# Install external packages
WORKDIR /opt/test-runner/external-packages
RUN go mod download
# Populate the build cache with the external packages
RUN go build ./...

# Build the test runner
WORKDIR /opt/test-runner
RUN GOOS=linux GOARCH=amd64 go build -o /opt/test-runner/bin/test-runner /opt/test-runner

USER appuser
ENV GOCACHE=/tmp
RUN go build -o /opt/test-runner/bin/test-runner /opt/test-runner

ENTRYPOINT ["sh", "/opt/test-runner/bin/run.sh"]
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ This is [Exercism's test runner](https://github.com/exercism/v3-docs/tree/master
## Executing the Test Runner

The test runner requires 2 parameters:

- `input_dir`: the path containing the solution to test
- `output_dir`: the output path for the test results

Expand Down Expand Up @@ -59,7 +60,7 @@ For troubleshooting / debug you can name the container, run it in interactive mo
```bash
docker run --name exercism-go-test-runner -d -i --network none --read-only -v $(pwd)/testrunner/testdata/practice/gigasecond:/solution -v /tmp:/tmp exercism/go-test-runner gigasecond /solution /tmp
# You can then access the container as follows:
docker exec -it --user appuser $(docker ps -q --filter name=exercism-go-test-runner) /bin/sh
docker exec -it $(docker ps -q --filter name=exercism-go-test-runner) /bin/sh
```

### External Go packages
Expand All @@ -81,19 +82,19 @@ A student can have a `go.mod` file declaring only supported dependencies, but if

## Subtests

The test runner is responsible for [returning the `test_code` field](https://github.com/exercism/v3-docs/blob/master/anatomy/track-tooling/test-runners/interface.md#command), which should be a copy of the test code corresponding to each test result.
The test runner is responsible for [returning the `test_code` field](https://github.com/exercism/v3-docs/blob/master/anatomy/track-tooling/test-runners/interface.md#command), which should be a copy of the test code corresponding to each test result.

For top-level tests, the AST is used to return the function code directly. For [tests containing subtests](https://blog.golang.org/subtests), additional processing is required. To ease the burden of advanced AST processing on unstructured / non deterministic test code, subtests should adhere to the following specification. **If a test employs subtests, do not mix it with test or other code outside of the Run() call.**
For top-level tests, the AST is used to return the function code directly. For [tests containing subtests](https://blog.golang.org/subtests), additional processing is required. To ease the burden of advanced AST processing on unstructured / non deterministic test code, subtests should adhere to the following specification. **If a test employs subtests, do not mix it with test or other code outside of the Run() call.**

- Subtests not meeting the spec will be treated as top-level tests, with the entire test function code being returned for every subtest.
- Assertions/outputs made outside of the Run() call will not be included in the result JSON because the "parent" tests are removed from the results if subtests are present. (Parent test reports were confusing to students because they did not include any assertion or `fmt.Println` output.)

At some point, we may [implement a static analyzer](https://rauljordan.com/2020/11/01/custom-static-analysis-in-go-part-1.html) which warns the exercise submitter when they commit subtests not meeting the specification.


### Subtest Format Specification

The specification is annotated in the comments of the following example test:

```go
func TestParseCard(t *testing.T) {
// There can be additional code here, it will be shown for all subtests.
Expand Down Expand Up @@ -138,6 +139,7 @@ func TestParseCard(t *testing.T) {
```

The test code above will result in the following `test_code` field, corresponding to the test named `TestParseCard/parse_queen`:

```go
tt := struct {
name string
Expand All @@ -162,9 +164,7 @@ This is done via the `.meta/config.json` file of the exercise. See example below
{
// ...
"custom": {
"testingFlags": [
"-race"
]
"testingFlags": ["-race"]
}
}
```
Expand Down Expand Up @@ -234,4 +234,4 @@ Besides what is mentioned in the open issues, the test runner has the following
- Sub-tests need to follow a certain format, see details above.

[task-id]: https://exercism.org/docs/building/tooling/test-runners/interface#h-task-id
[task-id-comments-examples]: https://github.com/exercism/go-test-runner/tree/main/testrunner/testdata/concept/conditionals-with-task-ids
[task-id-comments-examples]: https://github.com/exercism/go-test-runner/tree/main/testrunner/testdata/concept/conditionals-with-task-ids
10 changes: 5 additions & 5 deletions testrunner/testdata/expected/auto_assigned_task_ids.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,35 +13,35 @@
"name": "TestSimpleSubtest/ parse ace",
"status": "pass",
"test_code": "func TestSimpleSubtest(t *testing.T) {\n\ttt := struct {\n\t\tname string\n\t\tcard string\n\t\twant int\n\t}{\n\t\tname: \"parse ace\",\n\t\tcard: \"ace\",\n\t\twant: 11,\n\t}\n\n\tif got := ParseCard(tt.card); got != tt.want {\n\t\tt.Errorf(\"ParseCard(%s) = %d, want %d\", tt.card, got, tt.want)\n\t}\n\n}",
"message": "\n=== RUN TestSimpleSubtest/parse_ace\n\n --- PASS: TestSimpleSubtest/parse_ace \n",
"message": "\n=== RUN TestSimpleSubtest/parse_ace\n\n--- PASS: TestSimpleSubtest/parse_ace \n",
"task_id": 2
},
{
"name": "TestParseCard/ parse two",
"status": "pass",
"test_code": "func TestParseCard(t *testing.T) {\n\ttt := struct {\n\t\tname string\n\t\tcard string\n\t\twant int\n\t}{\n\t\tname: \"parse two\",\n\t\tcard: \"two\",\n\t\twant: 2,\n\t}\n\n\tif got := ParseCard(tt.card); got != tt.want {\n\t\tt.Errorf(\"ParseCard(%s) = %d, want %d\", tt.card, got, tt.want)\n\t}\n\n}",
"message": "\n=== RUN TestParseCard/parse_two\n\n --- PASS: TestParseCard/parse_two \n",
"message": "\n=== RUN TestParseCard/parse_two\n\n--- PASS: TestParseCard/parse_two \n",
"task_id": 3
},
{
"name": "TestParseCard/ parse jack",
"status": "pass",
"test_code": "func TestParseCard(t *testing.T) {\n\ttt := struct {\n\t\tname string\n\t\tcard string\n\t\twant int\n\t}{\n\t\tname: \"parse jack\",\n\t\tcard: \"jack\",\n\t\twant: 10,\n\t}\n\n\tif got := ParseCard(tt.card); got != tt.want {\n\t\tt.Errorf(\"ParseCard(%s) = %d, want %d\", tt.card, got, tt.want)\n\t}\n\n}",
"message": "\n=== RUN TestParseCard/parse_jack\n\n --- PASS: TestParseCard/parse_jack \n",
"message": "\n=== RUN TestParseCard/parse_jack\n\n--- PASS: TestParseCard/parse_jack \n",
"task_id": 3
},
{
"name": "TestParseCard/ parse king",
"status": "pass",
"test_code": "func TestParseCard(t *testing.T) {\n\ttt := struct {\n\t\tname string\n\t\tcard string\n\t\twant int\n\t}{\n\t\tname: \"parse king\",\n\t\tcard: \"king\",\n\t\twant: 10,\n\t}\n\n\tif got := ParseCard(tt.card); got != tt.want {\n\t\tt.Errorf(\"ParseCard(%s) = %d, want %d\", tt.card, got, tt.want)\n\t}\n\n}",
"message": "\n=== RUN TestParseCard/parse_king\n\n --- PASS: TestParseCard/parse_king \n",
"message": "\n=== RUN TestParseCard/parse_king\n\n--- PASS: TestParseCard/parse_king \n",
"task_id": 3
},
{
"name": "TestBlackjack/ blackjack with ten (ace first)",
"status": "fail",
"test_code": "func TestBlackjack(t *testing.T) {\n\tsomeAssignment := \"test\"\n\tfmt.Println(someAssignment)\n\n\ttype hand struct {\n\t\tcard1, card2 string\n\t}\n\ttt := struct {\n\t\tname string\n\t\thand hand\n\t\twant bool\n\t}{\n\t\tname: \"blackjack with ten (ace first)\",\n\t\thand: hand{card1: \"ace\", card2: \"ten\"},\n\t\twant: true,\n\t}\n\n\t_ = \"literally anything\"\n\n\tgot := IsBlackjack(tt.hand.card1, tt.hand.card2)\n\tif got != tt.want {\n\t\tt.Errorf(\"IsBlackjack(%s, %s) = %t, want %t\", tt.hand.card1, tt.hand.card2, got, tt.want)\n\t}\n\n\t// Additional statements should be included\n\tfmt.Println(\"the whole block\")\n\tfmt.Println(\"should be returned\")\n}",
"message": "\n=== RUN TestBlackjack/blackjack_with_ten_(ace_first)\n\n --- FAIL: TestBlackjack/blackjack_with_ten_(ace_first) \n\npanic: Please implement the IsBlackjack function [recovered]\n\n\tpanic: Please implement the IsBlackjack function\n\n\n\ngoroutine x [running]:\n\ntesting.tRunner.func1.2({, })\n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n\ntesting.tRunner.func1()\n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n\npanic({, })\n\n\tPATH_PLACEHOLDER/src/runtime/panic.go \n\nconditionals.IsBlackjack(...)\n\n\tPATH_PLACEHOLDER/testrunner/testdata/concept/auto_assigned_task_ids/conditionals.go\n\nconditionals.TestBlackjack.func1?)\n\n\tPATH_PLACEHOLDER/testrunner/testdata/concept/auto_assigned_task_ids/conditionals_test.go \n\ntesting.tRunner, \n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n\ncreated by testing.(*T).Run\n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n",
"message": "\n=== RUN TestBlackjack/blackjack_with_ten_(ace_first)\n\n--- FAIL: TestBlackjack/blackjack_with_ten_(ace_first) \n\npanic: Please implement the IsBlackjack function [recovered]\n\n\tpanic: Please implement the IsBlackjack function\n\n\n\ngoroutine x [running]:\n\ntesting.tRunner.func1.2({, })\n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n\ntesting.tRunner.func1()\n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n\npanic({, })\n\n\tPATH_PLACEHOLDER/src/runtime/panic.go \n\nconditionals.IsBlackjack(...)\n\n\tPATH_PLACEHOLDER/testrunner/testdata/concept/auto_assigned_task_ids/conditionals.go\n\nconditionals.TestBlackjack.func1?)\n\n\tPATH_PLACEHOLDER/testrunner/testdata/concept/auto_assigned_task_ids/conditionals_test.go \n\ntesting.tRunner, \n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n\ncreated by testing.(*T).Run\n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n",
"task_id": 4
}
]
Expand Down
2 changes: 1 addition & 1 deletion testrunner/testdata/expected/broken.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"status": "error",
"version": 3,
"message": "# gigasecond [gigasecond.test]\n./broken.go: undefined: unknownVar\n./broken.go: undefined: UnknownFunction\nFAIL\tgigasecond [build failed]\n'PATH_PLACEHOLDER test --short --json .' returned exit code 2: exit status 2",
"message": "# gigasecond [gigasecond.test]\n./broken.go: undefined: unknownVar\n./broken.go: undefined: UnknownFunction\nFAIL\tgigasecond [build failed]\n'PATH_PLACEHOLDER test --short --json .' returned exit code 1: exit status 1",
"tests": null
}
12 changes: 6 additions & 6 deletions testrunner/testdata/expected/explicit_task_ids.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,42 +13,42 @@
"name": "TestSimpleSubtest/ parse ace",
"status": "pass",
"test_code": "// Some other comment\n// testRunnerTaskID=2\nfunc TestSimpleSubtest(t *testing.T) {\n\ttt := struct {\n\t\tname string\n\t\tcard string\n\t\twant int\n\t}{\n\t\tname: \"parse ace\",\n\t\tcard: \"ace\",\n\t\twant: 11,\n\t}\n\n\tif got := ParseCard(tt.card); got != tt.want {\n\t\tt.Errorf(\"ParseCard(%s) = %d, want %d\", tt.card, got, tt.want)\n\t}\n\n}",
"message": "\n=== RUN TestSimpleSubtest/parse_ace\n\n --- PASS: TestSimpleSubtest/parse_ace \n",
"message": "\n=== RUN TestSimpleSubtest/parse_ace\n\n--- PASS: TestSimpleSubtest/parse_ace \n",
"task_id": 2
},
{
"name": "TestSimpleSubtest2/ parse ace",
"status": "pass",
"test_code": "// testRunnerTaskID=2 More text here\nfunc TestSimpleSubtest2(t *testing.T) {\n\ttt := struct {\n\t\tname string\n\t\tcard string\n\t\twant int\n\t}{\n\t\tname: \"parse ace\",\n\t\tcard: \"ace\",\n\t\twant: 11,\n\t}\n\n\tif got := ParseCard(tt.card); got != tt.want {\n\t\tt.Errorf(\"ParseCard(%s) = %d, want %d\", tt.card, got, tt.want)\n\t}\n\n}",
"message": "\n=== RUN TestSimpleSubtest2/parse_ace\n\n --- PASS: TestSimpleSubtest2/parse_ace \n",
"message": "\n=== RUN TestSimpleSubtest2/parse_ace\n\n--- PASS: TestSimpleSubtest2/parse_ace \n",
"task_id": 2
},
{
"name": "TestParseCard/ parse two",
"status": "pass",
"test_code": "// testRunnerTaskID=1\n// Some other comment\nfunc TestParseCard(t *testing.T) {\n\ttt := struct {\n\t\tname string\n\t\tcard string\n\t\twant int\n\t}{\n\t\tname: \"parse two\",\n\t\tcard: \"two\",\n\t\twant: 2,\n\t}\n\n\tif got := ParseCard(tt.card); got != tt.want {\n\t\tt.Errorf(\"ParseCard(%s) = %d, want %d\", tt.card, got, tt.want)\n\t}\n\n}",
"message": "\n=== RUN TestParseCard/parse_two\n\n --- PASS: TestParseCard/parse_two \n",
"message": "\n=== RUN TestParseCard/parse_two\n\n--- PASS: TestParseCard/parse_two \n",
"task_id": 1
},
{
"name": "TestParseCard/ parse jack",
"status": "pass",
"test_code": "// testRunnerTaskID=1\n// Some other comment\nfunc TestParseCard(t *testing.T) {\n\ttt := struct {\n\t\tname string\n\t\tcard string\n\t\twant int\n\t}{\n\t\tname: \"parse jack\",\n\t\tcard: \"jack\",\n\t\twant: 10,\n\t}\n\n\tif got := ParseCard(tt.card); got != tt.want {\n\t\tt.Errorf(\"ParseCard(%s) = %d, want %d\", tt.card, got, tt.want)\n\t}\n\n}",
"message": "\n=== RUN TestParseCard/parse_jack\n\n --- PASS: TestParseCard/parse_jack \n",
"message": "\n=== RUN TestParseCard/parse_jack\n\n--- PASS: TestParseCard/parse_jack \n",
"task_id": 1
},
{
"name": "TestParseCard/ parse king",
"status": "pass",
"test_code": "// testRunnerTaskID=1\n// Some other comment\nfunc TestParseCard(t *testing.T) {\n\ttt := struct {\n\t\tname string\n\t\tcard string\n\t\twant int\n\t}{\n\t\tname: \"parse king\",\n\t\tcard: \"king\",\n\t\twant: 10,\n\t}\n\n\tif got := ParseCard(tt.card); got != tt.want {\n\t\tt.Errorf(\"ParseCard(%s) = %d, want %d\", tt.card, got, tt.want)\n\t}\n\n}",
"message": "\n=== RUN TestParseCard/parse_king\n\n --- PASS: TestParseCard/parse_king \n",
"message": "\n=== RUN TestParseCard/parse_king\n\n--- PASS: TestParseCard/parse_king \n",
"task_id": 1
},
{
"name": "TestBlackjack/ blackjack with ten (ace first)",
"status": "fail",
"test_code": "// testRunnerTaskID=3\nfunc TestBlackjack(t *testing.T) {\n\tsomeAssignment := \"test\"\n\tfmt.Println(someAssignment)\n\n\ttype hand struct {\n\t\tcard1, card2 string\n\t}\n\ttt := struct {\n\t\tname string\n\t\thand hand\n\t\twant bool\n\t}{\n\t\tname: \"blackjack with ten (ace first)\",\n\t\thand: hand{card1: \"ace\", card2: \"ten\"},\n\t\twant: true,\n\t}\n\n\t_ = \"literally anything\"\n\n\tgot := IsBlackjack(tt.hand.card1, tt.hand.card2)\n\tif got != tt.want {\n\t\tt.Errorf(\"IsBlackjack(%s, %s) = %t, want %t\", tt.hand.card1, tt.hand.card2, got, tt.want)\n\t}\n\n\t// Additional statements should be included\n\tfmt.Println(\"the whole block\")\n\tfmt.Println(\"should be returned\")\n}",
"message": "\n=== RUN TestBlackjack/blackjack_with_ten_(ace_first)\n\n --- FAIL: TestBlackjack/blackjack_with_ten_(ace_first) \n\npanic: Please implement the IsBlackjack function [recovered]\n\n\tpanic: Please implement the IsBlackjack function\n\n\n\ngoroutine x [running]:\n\ntesting.tRunner.func1.2({, })\n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n\ntesting.tRunner.func1()\n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n\npanic({, })\n\n\tPATH_PLACEHOLDER/src/runtime/panic.go \n\nconditionals.IsBlackjack(...)\n\n\tPATH_PLACEHOLDER/testrunner/testdata/concept/explicit_task_ids/conditionals.go\n\nconditionals.TestBlackjack.func1?)\n\n\tPATH_PLACEHOLDER/testrunner/testdata/concept/explicit_task_ids/conditionals_test.go \n\ntesting.tRunner, \n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n\ncreated by testing.(*T).Run\n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n",
"message": "\n=== RUN TestBlackjack/blackjack_with_ten_(ace_first)\n\n--- FAIL: TestBlackjack/blackjack_with_ten_(ace_first) \n\npanic: Please implement the IsBlackjack function [recovered]\n\n\tpanic: Please implement the IsBlackjack function\n\n\n\ngoroutine x [running]:\n\ntesting.tRunner.func1.2({, })\n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n\ntesting.tRunner.func1()\n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n\npanic({, })\n\n\tPATH_PLACEHOLDER/src/runtime/panic.go \n\nconditionals.IsBlackjack(...)\n\n\tPATH_PLACEHOLDER/testrunner/testdata/concept/explicit_task_ids/conditionals.go\n\nconditionals.TestBlackjack.func1?)\n\n\tPATH_PLACEHOLDER/testrunner/testdata/concept/explicit_task_ids/conditionals_test.go \n\ntesting.tRunner, \n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n\ncreated by testing.(*T).Run\n\n\tPATH_PLACEHOLDER/src/testing/testing.go \n",
"task_id": 3
}
]
Expand Down
2 changes: 1 addition & 1 deletion testrunner/testdata/expected/missing_func.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"status": "error",
"version": 3,
"message": "# gigasecond [gigasecond.test]\n./missing_func_test.go: undefined: AddGigasecond\n./missing_func_test.go: undefined: AddGigasecond\nFAIL\tgigasecond [build failed]\n'PATH_PLACEHOLDER test --short --json .' returned exit code 2: exit status 2",
"message": "# gigasecond [gigasecond.test]\n./missing_func_test.go: undefined: AddGigasecond\n./missing_func_test.go: undefined: AddGigasecond\nFAIL\tgigasecond [build failed]\n'PATH_PLACEHOLDER test --short --json .' returned exit code 1: exit status 1",
"tests": null
}
2 changes: 1 addition & 1 deletion testrunner/testdata/expected/missing_task_ids.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"name": "TestSimpleSubtest/ parse ace",
"status": "pass",
"test_code": "// testRunnerTaskID=1\nfunc TestSimpleSubtest(t *testing.T) {\n\ttt := struct {\n\t\tname string\n\t\tcard string\n\t\twant int\n\t}{\n\t\tname: \"parse ace\",\n\t\tcard: \"ace\",\n\t\twant: 11,\n\t}\n\n\tif got := ParseCard(tt.card); got != tt.want {\n\t\tt.Errorf(\"ParseCard(%s) = %d, want %d\", tt.card, got, tt.want)\n\t}\n\n}",
"message": "\n=== RUN TestSimpleSubtest/parse_ace\n\n --- PASS: TestSimpleSubtest/parse_ace \n"
"message": "\n=== RUN TestSimpleSubtest/parse_ace\n\n--- PASS: TestSimpleSubtest/parse_ace \n"
}
]
}

0 comments on commit 35f10f2

Please sign in to comment.