Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

write results to file #37

Open
peshay opened this issue Dec 14, 2018 · 4 comments
Open

write results to file #37

peshay opened this issue Dec 14, 2018 · 4 comments

Comments

@peshay
Copy link

peshay commented Dec 14, 2018

I like to get the test_results written to a file, so a build-server can later pick it up.
When I used your role from ansible-galaxy, I could add in your main.yml

- name: write report
  local_action: copy content="{{ goss_output.stdout }}" dest="{{ goss_report_file }}"
  become: false
  when: goss_report_file is defined

I just wanted to make this a pull requests, but see that you changed a lot here! ;)
Still I would file a pull request for that feature, but maybe you can already tell me what you think about this idea?

@naftulikay
Copy link
Owner

@peshay I'm stoked to have users! 🙌 🌞

I was actually considering doing something similar to this, but I wasn't sure how to make it ergonomic and flexible enough for users. The current use-case that is supported is:

  • Set degoss_clean_on_failure to false, and when failures occur, grab {{ degoss_tmp_root }}/result.json and examine it to understand the failures.

The problem with this current approach is that it's not at all easy to get degoss_tmp_root on failure. On success, the next play should see the degoss_tmp_root fact, but if it fails, currently it just aborts.

I guess I'm just asking what the exact use case is here. Do you want the role to never fail, even on test failure? Would adding degoss_result_file and copying the result there, regardless of pass/fail, satisfy your use-case?

@peshay
Copy link
Author

peshay commented Dec 17, 2018

My use case is to have my role development checked by a build server. So it's totally fine, that if something fails in the goss checks, that the role fails and stops further execution and that the build shows as failed. But in any case I want the report output in a file, so the jenkins tap plugin can grab the report and I could see which checks failed and so on.

@naftulikay
Copy link
Owner

@peshay thanks for getting back.

So the new refactor only and exclusively uses the JSON output so as to make handling failure cases easier. There is an outstanding bug (goss-org/goss#317) that Goss returns 0 on failure at times, so the only reliable way for us to detect failure is to use the JSON output and try to deserialize the output into JSON. If deserialization fails, we know that Goss encountered a critical error, otherwise we report success and failure.

So presently, the only output format supported going forward will be JSON. It should be possible to translate the JSON into Jenkins tap format, but yeah, I don't know if we can still support different output formats :(

If I added degoss_result_file to store the JSON in, would this still be helpful?

I had considered executing with JSON output and then subsequently again with the desired other output format, but this would require all tests to be idempotent, which is really error-prone.

@naftulikay
Copy link
Owner

Oh, also, there may be a workaround to make it easy to catch events. See callback_plugins/goss_format.py here in the repository for an example of how to do this. Catch any result from a degoss task and you'll be able to store or otherwise work with the JSON.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants