Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test strategy #12

Closed
dschaefer opened this issue Nov 13, 2018 · 17 comments
Closed

Test strategy #12

dschaefer opened this issue Nov 13, 2018 · 17 comments
Assignees
@dschaefer

Comments

@dschaefer
Copy link
Contributor

Time to put together a test strategy for the adapter. We want to be able to confidently make changes without breaking the clients.

We have experience with mocha but I hear that it doesn't have good Jenkins reporting which we need for the CI machine.

Anyone else have any ideas?
@dschaefer
Copy link
Contributor Author

There's probably two levels of testing here. Unit and integration. Unit tests the APIs of the components, where integration would actually test against a running gdb and fake out a client.

@simark
Copy link
Contributor

simark commented Nov 14, 2018

For the integration test using a fake client, this seems nice, part of the vscode-debugadapter repo:

https://github.com/Microsoft/vscode-debugadapter-node/tree/master/testSupport

I am planning to start writing some tests with it, any objections?

What kind of test output does your Jenkins job want? It is possible to write new reporters for mocha, so I assume it's possible to output what you need.

For the unit tests, mocha + chai is a combo that works well, that's what we use in Theia.

@dschaefer
Copy link
Contributor Author

For the integration test using a fake client, this seems nice, part of the vscode-debugadapter repo:

https://github.com/Microsoft/vscode-debugadapter-node/tree/master/testSupport

I am planning to start writing some tests with it, any objections?

That would be awesome! Go ahead.

What kind of test output does your Jenkins job want? It is possible to write new reporters for mocha, so I assume it's possible to output what you need.

For the unit tests, mocha + chai is a combo that works well, that's what we use in Theia.

Yeah, that's what we use for our web projects here at QNX. The biggest issue we've had is with finding good reporters that present the results nicely. I'll have to see what the current state of all that is.

@dschaefer
Copy link
Contributor Author

Actually we have a reporter we're using internally. I should be able to open it.

@dschaefer dschaefer self-assigned this Nov 16, 2018
simark pushed a commit to simark/cdt-gdb-adapter that referenced this issue Nov 20, 2018
Add first, simple launch test
8c798d1 
Add an integration test that sets a breakpoint, runs, then expect to hit
that breakpoint.

If you forget a step (forget to build the package or the test examples),
the errors are not very descriptive nor helpful, that's something I'd
like to improve in the future.

Refs eclipse-cdt-cloud#12

Signed-off-by: Simon Marchi <simon.marchi@ericsson.com>
simark pushed a commit to simark/cdt-gdb-adapter that referenced this issue Nov 20, 2018
Add first, simple launch test
1349d40 
Add an integration test that sets a breakpoint, runs, then expect to hit
that breakpoint.

If you forget a step (forget to build the package or the test examples),
the errors are not very descriptive nor helpful, that's something I'd
like to improve in the future.

Refs eclipse-cdt-cloud#12

Signed-off-by: Simon Marchi <simon.marchi@ericsson.com>
simark pushed a commit to simark/cdt-gdb-adapter that referenced this issue Nov 21, 2018
Add first, simple launch test
0275e50 
Add an integration test that sets a breakpoint, runs, then expect to hit
that breakpoint.

If you forget a step (forget to build the package or the test examples),
the errors are not very descriptive nor helpful, that's something I'd
like to improve in the future.

Refs eclipse-cdt-cloud#12

Signed-off-by: Simon Marchi <simon.marchi@ericsson.com>
dschaefer pushed a commit that referenced this issue Nov 26, 2018
@dschaefer
Add first, simple launch test
764ce26 
Add an integration test that sets a breakpoint, runs, then expect to hit
that breakpoint.

If you forget a step (forget to build the package or the test examples),
the errors are not very descriptive nor helpful, that's something I'd
like to improve in the future.

Refs #12

Signed-off-by: Simon Marchi <simon.marchi@ericsson.com>
@dschaefer
Copy link
Contributor Author

We have the integration story in place. Thanks @simark!
Be good to have a unit test strategy where we mock out gdb as well.

@jonahgraham
Copy link
Contributor

@thegecko Please comment here and I will try to figure it out.

@jonahgraham jonahgraham mentioned this issue Sep 26, 2019
Merged
@jonahgraham
Copy link
Contributor

In #136 @thegecko mentions issues with MacOS. I didn't know that tests didn't work on MacOS. I am in the process of bringing up a machine to add to the CI to regtest on Windows, but I don't have any immediate plans on Mac - but I could do the same there at some point.

@thegecko the best would be that tests "just worked" on Mac. What is the issue there? Is it the getting a GDB/GCC/toolchain that works easily, or something else?

@thegecko
Copy link
Contributor

I see lots of gdb errors, so it's likely that needs to be configured properly. e.g.

Uncaught Error: spawn gdbserver ENOENT

Error: Unable to find Mach task port for process-id 58698: (os/kern) failure (0x5).
 (please check gdb is codesigned - see taskgated(8))

Error: gdbserver has hit error Error: spawn gdbserver ENOENT

I wonder if a docker container would make sense to encapsulate the test environment?

@jonahgraham
Copy link
Contributor

I wonder if a docker container would make sense to encapsulate the test environment?

There is already one based on cdt-infra, you can do something like this if you are in the root of the project. (Or just run bash in the container and do things manually)

docker run --rm -it -v $PWD:/work -w /work  quay.io/eclipse-cdt/cdt-infra-eclipse-full:ubuntu-16.04-a6ea855 /bin/bash -c 'yarn install && yarn build && yarn test'

For ease of use a "latest" tag would probably make sense - perhaps also one that does not start UI/VNC environment.

@simark
Copy link
Contributor

simark commented Sep 26, 2019

The state of GDB on macOS isn't so great, so there is probably nothing else to do other than fix GDB itself.

@thegecko
Copy link
Contributor

thegecko commented Nov 6, 2019

@jonahgraham I see you updated instructions in #144

On my machine this errors unfortunately:

Robs-iMac:cdt-gdb-adapter thegecko$ docker run --rm -it -v $(git rev-parse --show-toplevel):/work -w /work/$(git rev-parse --show-prefix) --cap-add=SYS_PTRACE --security-opt seccomp=unconfined quay.io/eclipse-cdt/cdt-infra-eclipse-full:latest yarn test
Unable to find image 'quay.io/eclipse-cdt/cdt-infra-eclipse-full:latest' locally
latest: Pulling from eclipse-cdt/cdt-infra-eclipse-full
5667fdb72017: Pull complete 
d83811f270d5: Pull complete 
ee671aafb583: Pull complete 
7fc152dfb3a6: Pull complete 
eeb59844c88b: Pull complete 
f43df22e4711: Pull complete 
d07816dc3ca8: Pull complete 
e9fccb9b0829: Pull complete 
a004456d8e7b: Pull complete 
4d6366bbe6a4: Pull complete 
458bfadf248e: Pull complete 
257e47fc66b2: Pull complete 
27457488a691: Pull complete 
4161bba5b45e: Pull complete 
6ed5400d93c5: Pull complete 
e8fb540c4f6c: Pull complete 
513ebb995671: Pull complete 
aa2458f1d031: Pull complete 
ec88e9f7f81d: Pull complete 
e01b96631b9f: Pull complete 
0bb7b11f7d58: Pull complete 
6b4ad39fb5ee: Pull complete 
2a422eb69cc2: Pull complete 
0dde373ae04b: Pull complete 
450ca14a3dbf: Pull complete 
bad1ea8640db: Pull complete 
f56a5e03017c: Pull complete 
b83b56f4437f: Pull complete 
e9d995492e40: Pull complete 
3e5a80d8f1bf: Pull complete 
Digest: sha256:c7409b802f7d70c936bd851efee25951e548d935615cb2b7f5959bc3aab2f9cc
Status: Downloaded newer image for quay.io/eclipse-cdt/cdt-infra-eclipse-full:latest
whoami: cannot find name for user ID 1000
yarn run v1.19.1
$ yarn test:integration && yarn test:pty && yarn test:integration-run-in-terminal && yarn test:integration-remote-target && yarn test:integration-remote-target-run-in-terminal
$ cross-env JUNIT_REPORT_PATH=test-reports/integration.xml JUNIT_REPORT_STACK=1 JUNIT_REPORT_PACKAGES=1 mocha --reporter mocha-jenkins-reporter dist/integration-tests/*.spec.js
/bin/sh: 1: cross-env: not found
error Command failed with exit code 127.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 127.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

@jonahgraham
Copy link
Contributor

I think you need to do the yarn install & yarn build first for the container, cross-env comes in there.

@thegecko
Copy link
Contributor

thegecko commented Nov 6, 2019

you need to do the yarn install & yarn build first

Yup, that fixed the run.

Unfortunately 8 of the 19 tests failed :(

@jonahgraham
Copy link
Contributor

Does qemu work on Macos? Because so much of the focus is on embedded dev, I thought it would be good if we had a "true" cross platform / remote dev environment. That may be easier?

@jonahgraham
Copy link
Contributor

I suspect that the 8 out of 19 failed is because the elf files were not rebuilt in the container? I can reproduce something similar. I have added to the documentation on running tests to say that you should be fully clean when changing platforms (the change is implicit when going from host to container run).

@jonahgraham
Copy link
Contributor

See PR #146

@thegecko thegecko mentioned this issue Nov 13, 2019
Merged
@jonahgraham
Copy link
Contributor

We have fairly decent tests now - some more work to be done in #160

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dschaefer dschaefer

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@thegecko @dschaefer @jonahgraham @simark