POC/DRAFT: Add transforms

[aelsabbahy]

This is to showcase some of the changes I have in mind for #538 to do a major enhancement on the goss/gomega language. Check out the examples folder for an idea of where I'm going with this.

This is horrible code that needs to be cleaned up, just tried to get a quick braindump out there since the code explains it a lot better than my ticket.

For usage examples take a look at this:
https://github.com/aelsabbahy/goss/tree/add_transforms/examples

[paskal]

Hi there, I'm interested in having a fix for #483 and I would strongly prefer contributing to upstream istead of mainating my own fork. If I understand correctly, this MR is a potential fix: is there any way I could assist you in finishing it?

[aelsabbahy]

This PR is a potential fix in that it allows direct string matching of the command output. Rather than only limiting it to the current pattern matcher logic.

Here's an example with some work from this PR:

command:
  test_exec:
    exec: 'echo -e "This is an example output of a command\nGoss will print this in full when using the new string matcher\nas opposed to using the pattern matcher"'
    exit-status: 0
    stdout:
    - This is an example output of a command, broken
    - Goss will print this in full when using the new string matcher
    - as opposed to using the pattern matcher
    stderr: []
    timeout: 10000
  test_exec2:
    exec: 'echo -e "This is an example output of a command\nGoss will print this in full when using the new string matcher\nas opposed to using the pattern matcher"'
    exit-status: 0
    stdout: |
      This is an example output of a command, broken
      Goss will print this in full when using the new string matcher
      as opposed to using the pattern matcher
    stderr: []
    timeout: 10000

notice test_exec2 is testing stdout against a string rather than an array. The output of running this in the new version of goss will be:

.F..F.

Failures/Skipped:

Command: test_exec: stdout:
Expected
    {} # not implemented yet, will probably say "io.Reader{}", but it will not contain the command output
to contain patterns
    ["This is an example output of a command, broken","Goss will print this in full when using the new string matcher","as opposed to using the pattern matcher"]
the missing elements were
    ["This is an example output of a command, broken"]

Command: test_exec2: stdout:
Expected
    "This is an example output of a command\nGoss will print this in full when using the new string matcher\nas opposed to using the pattern matcher\n"
to equal
    "This is an example output of a command, broken\nGoss will print this in full when using the new string matcher\nas opposed to using the pattern matcher\n"

Total Duration: 0.002s
Count: 6, Failed: 2, Skipped: 0

This PR will be a pretty major refactor of how goss handles pattern matching and how results are output. It is a breaking change so it will be in goss v0.4.0.. and since v0.4.0 will be a breaking change, there's a few other breaking changes that I would like to have in there prior to release.

Here's my todo.md (and progress) for v0.4.0:

# TODO:

[X] Find way to get private unexported values from gomega matchers
[X] Implement gomega matcher for validate contains
[X] Remove old validatecontains code
[X] implement gjson transform
[X] Add equal matcher
[X] Use benumerically as defualt numbers
[X] Add toNumeric transformer
[-] create new testresult object
[x] Move all matcher data outside of matchers and into testresult object
[ ] Move all formatting related to testresult into outputs/
[-] Implement clean short failure messages for matchers
[ ] Check have-key message with matcher
[ ] Write docs
[ ] github.com/sanity-io/litter?
[ ] And matcher indent is funky/incorrect
[ ] get rid of have-key-with-value or dry out have-key-with-value?

# optional
[ ] catch .(string) type errors in gomega.go (if input is not controlled)
[ ] Implement have-patterns for array
[ ] all matcher potentially
[ ] investigate why string diff does not trigger for equal matcher
[ ] register matchers using init?

# Other v0.4.0 stories, since they potentially introduce breaking changes
[ ] psutil vs go-ps, possibly switch to the former
[ ] process, comm vs args. Can we match on either or with the psutil library?
[ ] rpm expand full version
[ ] rpm/deb version matchers? create generic generic version of semver.Range that can be used with a version interface. Create wrappers around 3rd party rpm/deb comparison libraries to honor version interface

Let me know if there's something in there that you're interested in helping out with. If I had to guess I'm about ~60% done with this PR. However, I haven't even started on any of the other v0.4.0 work.

[paskal]

I'll look into these early next week and will either do something or ask more questions then. Thank you for the detailed answer!

[aelsabbahy]

Awesome, please reach out to me before putting in work/time, I want to avoid a situation where you do work I already completed.

Also, is the string matcher change what you're looking for with regards to #483?

The reason it can't be done with patterns is because goss reads a file line by line. This is apparent if you run goss v on a GB text file or command output, the memory usage stays very low. Using string matcher would read the results into memory, but will have the benefit of displaying it.

[paskal]

I don't know yet how I would approach as the only thing I know for now is that delivering more information to the affected asset owner is a necessity. If I understand you correctly and it will work with text match but not with regexp, I am fine with it as it's infinitely better than the current situation which doesn't seem to have a workaround (at least I'm not aware of it).

I think I'll start with looking into this PR and figuring out if I can help with it, and then going trough the list and pinging you in PRs\issues.

[paskal]

My apologies for the delay, I don't know if I'll have time to look into this as I got a lot of other stuff on hands. @nloginov is interested in having this in place as well and will likely try to help you at some point in time.

[aelsabbahy]

@petemounce Tagging you on here since you are creating a bunch of PRs currently.

This is a major work stream that I'm working on for goss v4. I would love your feedback on it.

The manual.md changes and this file explain the differences:
examples/goss_awesome_gomega.yaml

I will most likely create a v4 branch, this + a few other PRs will be merged into it and hopefully I can get v4 out the door within a month. This is a fairly large re-write and may impact some of the PRs you have out there.

[petemounce]

@petemounce Tagging you on here since you are creating a bunch of PRs currently.

This is a major work stream that I'm working on for goss v4. I would love your feedback on it.

The manual.md changes and this file explain the differences:
examples/goss_awesome_gomega.yaml

I will most likely create a v4 branch, this + a few other PRs will be merged into it and hopefully I can get v4 out the door within a month. This is a fairly large re-write and may impact some of the PRs you have out there.

I've reviewed just the docs in this pass; this is a pretty massive PR to feed back on. Is there any possibility of slicing it up into smaller chunks?

What kinds of feedback are you after? UX/requirements/design/implementation/coding style/tests/something else?

If I've understood the intent - this is so that inputs can be converted into the appropriate types to allow assertion with a richer set of matchers. Is that right?

[petemounce]

Suggested change

[aelsabbahy]

Was this a typo on your end? github UI isn't clear on what this change actually is. Looks to be adding a -?

[petemounce]

Yes, it's removing a newline - sorry, nit-pick on my part! I have pre-commit set up to automatically do things like that.

[petemounce]

As a user, I would find this easier to interpret if

• the front of the messaging included the input value and type
• the end of the messaging showed the desired output type

So, sketching something like this?

Suggested change

	KernelParam: net.core.somaxconn: value:
	Expected
	128
	to be >
	200
	the transform chain was
	[{"to-numeric":{}}]
	Total Duration: 0.001s
	Count: 1, Failed: 1, Skipped: 0
	KernelParam: net.core.somaxconn: value:
	Expected
	(string) 128
	to be >
	(numeric) 200
	the transform chain was
	[{"to-numeric":{ input: "128" }}]
	Total Duration: 0.001s
	Count: 1, Failed: 0, Skipped: 0, Error: 1

I put the types ahead of the values since the width of the type-string-names is predictable, and then the lines can be padded out to the longest one so that the values are all going to appear starting from the same column, making them easier to eyeball.

I added Error: 1 because it feels to me that a failure to transform is an internal error as opposed to a failed assertion. That struck me as something I'd like to know separately, since it means there's something wrong in how I've defined my assertions, rather than a for-sure assertion-failure.

[petemounce]

This suggestion accidentally also (I think?) removes the need for the -o include_raw, since it includes the same information with no extra output lines needed.

[aelsabbahy]

Gonna defer this conversation to regular github comments to make it easier to track. I'll detail the different use-cases there.

[petemounce]

This means the tests would need to be re-run to get that extra information. As a user, I would prefer that if a test fails, all of the possible debugging information were given to me so I can have it up-front. Not doing that

• in the best case, costs me the time to re-run the tests
  • but this is harder on machines that are remote
• in the worst case, costs me a lot of time to re-run the tests multiple times in the event of chasing an intermittent failure (much more likely in machine state testing, and more likely again if the test touches an external dependency like addr, http etc).

[petemounce]

This works because the default type in YAML is a string, right?

[aelsabbahy]

you mean have-prefix: pre vs have-prefix: "pre" if so, yes.. that's yaml-ism.

[petemounce]

Suggested change

[aelsabbahy]

again, this looks like - being suggested as an addition.. is that correct?

[petemounce]

Is it to late to rename consist-of to satisfies?

[aelsabbahy]

consist-of has been around since 2016. Maybe as part of 4 or a 4.X release I can add other aliases, but I prefer not breaking the old interface.

The verbiage was taken from the gomega project which goss relies on. https://onsi.github.io/gomega/#consistofelement-interface

[petemounce]

Ah, gotcha. Yeah, definitely not worth a breaking change. If it's possible easily to alias, that might be nice, but ... not that nice.

[petemounce]

I haven't, from this and the example below, figured out where I would use contain-element vs consist-of. I find it difficult to understand partly because both of them allow matchers to be present.

I can think of wanting to:

• assert that an array contains particular elements/matches (without caring about ordering)
• assert that an array contains a sequence of particular elements/matches (e.g. contains a then any-single-element then b, but that sequence can be anywhere in the array)
  • same but a then not-b then b
• assert that an array contents matches sequence x, y, z exactly

I think I'm struggling partly because the doc prose is more of the reference style (which I find quite abstract, though of course that's highly subjective!) rather than how-to/use-case style. I think if it contained more worked examples I'd have an easier time.

[aelsabbahy]

contain-element is there since 2016. This PR adds contain-elements which is really just a subset check.

For your scenarios:

assert that an array contains particular elements/matches (without caring about ordering)

contain-element / contain-elements / ["a", "b"] (this is an alias for contain-elements) will satisfy this

assert that an array contents matches sequence x, y, z exactly

equal

assert that an array contains a sequence of particular elements/matches (e.g. contains a then any-single-element then b, but that sequence can be anywhere in the array)
same but a then not-b then b

I don't think this is supported today, sounds like a subset of matchers where order matters. I'm curious about the real-world use-case of this.. can be a new matcher that's added in 4.X

I think if it contained more worked examples I'd have an easier time.

I love this suggestion, I wonder if it should be:

1. A new section in manual.md
2. A new doc that's linked to from manual.md (examples.md?)
3. Just an example folder(s) of yaml files. Could use either the matching or maybe real world examples?

Thoughts?

[petemounce]

I think a new section of the manual.md would work well.

Would you be interested in a PR that adds https://www.mkdocs.org/ -> github pages for documentation?

[aelsabbahy]

I like the input suggestion. The problem that I've been having is striking a balance between noisy default output and useful default outputs.

$ cat goss.yaml
matching:
  example:
    content: '{"foo": "bar", "moo" {"nested": "cow"}, "count": "15"}'
    matches:
      gjson:
        moo.nested: cow
        foo: {have-prefix: b}
        count: {le: 2}  # this will fail
        '@this': {have-key: "foo"}
        moo:
          and:
            - {have-key: "nested"}
            - {not: {have-key: "nested2"}}

And the different executions:

$ goss v
F

Failures/Skipped:

Matching: example: matches:
Expected
    15
to be <=
    2
the transform chain was
    [{"gjson":{"Path":"count"}},{"to-numeric":{}}]

Total Duration: 0.001s
Count: 1, Failed: 1, Skipped: 0
$ goss v -o include_raw
F

Failures/Skipped:

Matching: example: matches:
Expected
    15
to be <=
    2
the transform chain was
    [{"gjson":{"Path":"count"}},{"to-numeric":{}}]
the raw value was
    "{\"foo\": \"bar\", \"moo\" {\"nested\": \"cow\"}, \"count\": \"15\"}"

Total Duration: 0.001s
Count: 1, Failed: 1, Skipped: 0
$ goss v --format tap
1..1
not ok 1 - Matching: example: matches: Expected "15" to be <= 2 the transform chain was [{"gjson":{"Path":"count"}},{"to-numeric":{}}]
$ goss v --format tap -o include_raw
1..1
not ok 1 - Matching: example: matches: Expected "15" to be <= 2 the transform chain was [{"gjson":{"Path":"count"}},{"to-numeric":{}}] the raw value was "{\"foo\": \"bar\", \"moo\" {\"nested\": \"cow\"}, \"count\": \"15\"}"

-o include_raw vs -o exclude_raw is really more about what gets set as the default. I initially had raw included by default, but got the opposite feedback. Need a good way to poll goss users on topics like this, any suggestions?

Errors are caught like other goss errors are. That said, I think the tansforms need more clearer error messages and to explain which transform encountered it, since this isn't very clear.

$ goss v
F

Failures/Skipped:

example: matches: Error: strconv.ParseFloat: parsing "15x": invalid syntax

Total Duration: 0.000s
Count: 1, Failed: 1, Skipped: 0

[petemounce]

I like the input suggestion. The problem that I've been having is striking a balance between noisy default output and useful default outputs.

My preference is for an error to tell me all of the inputs and all of the outputs, so that I don't need to run the test again to find them.

I think that for machine-state assertions, that's more-than-normally valuable as a default due to the risk of intermittent failures. I think that because machine-state is not an isolated system; it's fraught with external influencers, dependencies, and those squishy meatbags changing things on what would otherwise be shining clean examples of computational perfection. :)

-o include_raw vs -o exclude_raw is really more about what gets set as the default. I initially had raw included by default, but got the opposite feedback. Need a good way to poll goss users on topics like this, any suggestions?

Maybe a gitter.im or community slack? I don't have any good suggestions, sorry.

[aelsabbahy]

@ripienaar Adding you here since you're pretty active with major goss changes as of late. This is probably the defining feature of v0.4.0, would love your input. The major change is the ability to do transforms and compare "15" <= 15, or use gjson to extract JSON values from a string (this allows for a much more powerful command/http tests).

At this point, I think it's just a discussion about UI and sane defaults, but feel free to review any part of this PR.

For example, using tap output and the example from the comment above #576 (comment)

currently the PR defaults to a "quiet" output

$ goss v --format tap
1..1
not ok 1 - Matching: example: matches: Expected {"count":"15","foo":"bar","moo":{"nested":"cow"}} to have key matching "foox" the transform chain was [{"gjson":{"Path":"@this"}}]

This can be changed by using -o include_raw to make it more verbose and include the "Raw" input:

$ goss v --format tap -o include_raw
1..1
not ok 1 - Matching: example: matches: Expected {"count":"15","foo":"bar","moo":{"nested":"cow"}} to have key matching "foox" the transform chain was [{"gjson":{"Path":"@this"}}] the raw value was "{\"foo\": \"bar\", \"moo\" {\"nested\": \"cow\"}, \"count\": \"15\"}"

@petemounce is proposing removing the include_raw and just having the transform chain include the "input" as part of it.

$ goss v --format tap                                                                                                                                                                          
1..1                                                                                                                                                                                           
not ok 1 - Matching: example: matches: Expected {"count":"15","foo":"bar","moo":{"nested":"cow"}} to have key matching "foox" the transform chain was [{"gjson":{"Path":"@this","Input":"{\"foo\": \"bar\", \"moo\" {\"nested\": \"cow\"}"}}] 

[aelsabbahy]

@petemounce This is very close now, I think..

The failure is in the windows test:

...F.F..
Failures/Skipped:
Command: wrap a powershell with quotes - expect 0 because travis does not restrict anonymous logins: stdout:
Error
    HavePatterns matcher expects patterns to be a string. got: 
        <int>: 0
Command: wrap a powershell - expect 0 because travis does not restrict anonymous logins: stdout:
Error
    HavePatterns matcher expects patterns to be a string. got: 
        <int>: 0
Total Duration: 0.848s
Count: 8, Failed: 2, Skipped: 0

Any objection to this change/fix? ac0e1ae