probes/yamlfilecontent: Bump yaml-filter, extend the schema and probe to be able to work with a set of values in maps #1552

evgenyz · 2020-06-25T21:51:30Z

This introduces YAML Path selection $.blah['key1','key2'] syntax and
enriches yamlfilecontent test and schema with EntityStateRecordType and
EntityItemRecordType result elements for collecting and checking
complex objects in YAML/JSON documents.

This change also depends on removal of the 'only lower case' restriction
from the name attribute of the field element (EntityXxxFieldType).

evgenyz · 2020-06-25T21:57:32Z

Example of a check for the "Upgradable" problem:

    <ind-def:yamlfilecontent_object version="1" id="oval:0:obj:5">
      <ind-def:path>/tmp</ind-def:path>
      <ind-def:filename>openshift-logging.yaml</ind-def:filename>
      <ind-def:yamlpath>.status.conditions[:]['status','type']</ind-def:yamlpath>
    </ind-def:yamlfilecontent_object>

    <ind-def:yamlfilecontent_state version="1" id="oval:0:ste:5">
      <ind-def:value datatype="record">
        <field name="status" datatype="string">True</field>
        <field name="type" datatype="string">Upgradeable</field>
      </ind-def:value>
    </ind-def:yamlfilecontent_state>

More details in tests/probes/yamlfilecontent/test_probes_yamlfilecontent_key.xml.

CC @JAORMX

evgenyz · 2020-06-25T21:59:26Z

Problems so far: this thing

<field name="value_of" datatype="string" var_ref="oval:0:var:2" var_check="at least one" entity_check="at least one"/>

is not working, no idea why. (CC @jan-cerny)

Edit: fixed.

evgenyz · 2020-06-25T22:02:21Z

As a bonus side of this change – we can now address any map, given that there is no values with complex objects (selection syntax could help if that's not the case).

evgenyz · 2020-06-26T06:25:29Z

This is a possible fix for ComplianceAsCode/content#5822.

yuumasato · 2020-06-30T09:24:04Z

Problems so far: this thing
<field name="value_of" datatype="string" var_ref="oval:0:var:2" var_check="at least one" entity_check="at least one"/>
is not working, no idea why. (CC @jan-cerny)

It could be because record data type does not allow variables.
https://github.com/OVALProject/Language/blob/master/docs/oval-common-schema.md#---complexdatatypeenumeration---

evgenyz · 2020-06-30T09:27:27Z

Problems so far: this thing
<field name="value_of" datatype="string" var_ref="oval:0:var:2" var_check="at least one" entity_check="at least one"/>
is not working, no idea why. (CC @jan-cerny)
It could be because record data type does not allow variables.
https://github.com/OVALProject/Language/blob/master/docs/oval-common-schema.md#---complexdatatypeenumeration---

Yeah, but it's a field (child of it, not record itself). I'm not sure here, but I understood that should be possible for field.

matejak

I have provided some comments about the OVAL.

Would it be possible to split the C code into more functions that would make it more obvious what is going on there? The code is starting to be complex, and it is said that function names document better than comments that tend to become obsolete very quickly.

schemas/oval/5.11.3/independent-definitions-schema.xsd

tests/probes/yamlfilecontent/test_probes_yamlfilecontent_array.xml

yuumasato

The idea to use record datatype is interesting, and makes sense to me.
There are a few caveats that should be considered though:

~~No variables~~
~~Only equals operator, no pattern matching~~
name should be all lower case

Edit: the remarks on the standard are about the record not the field.

tests/probes/yamlfilecontent/test_probes_yamlfilecontent_key.xml

schemas/oval/5.11.3/independent-system-characteristics-schema.xsd

schemas/oval/5.11.3/independent-definitions-schema.xsd

yuumasato · 2020-06-30T10:17:47Z

Yeah, but it's a field (child of it, not record itself). I'm not sure here, but I understood that should be possible for field.

Well noted, it makes sense that records cannot be compared to variables, and there is nothing related to fields comparison with variables.

schemas/oval/5.11.3/independent-definitions-schema.xsd

schemas/oval/5.11.3/oval-definitions-schema.xsd

tests/probes/yamlfilecontent/test_probes_yamlfilecontent_key.xml

evgenyz · 2020-06-30T12:15:47Z

Would it be possible to split the C code into more functions that would make it more obvious what is going on there? The code is starting to be complex, and it is said that function names document better than comments that tend to become obsolete very quickly.

I will brush it up during the final implementation of all the details we are currently discussing (like having value_of entry together with result and so on). This version was made with the least amount of code changes in mind to show what is actually going on.

evgenyz · 2020-07-15T13:01:02Z

Problems so far: this thing
<field name="value_of" datatype="string" var_ref="oval:0:var:2" var_check="at least one" entity_check="at least one"/>
is not working, no idea why. (CC @jan-cerny)
It could be because record data type does not allow variables.
https://github.com/OVALProject/Language/blob/master/docs/oval-common-schema.md#---complexdatatypeenumeration---
Yeah, but it's a field (child of it, not record itself). I'm not sure here, but I understood that should be possible for field.

So, it just wasn't implemented. I fixed it.

yuumasato

I'd go with value.
Changes in the scanner look good to me.

src/OVAL/results/oval_cmp_impl.h

tests/probes/yamlfilecontent/test_probes_yamlfilecontent_key.xml

tests/probes/yamlfilecontent/test_probes_yamlfilecontent_types.sh

evgenyz · 2020-07-31T06:45:23Z

All current issues were addressed, waiting for reaction in OVAL-Community/OVAL#91.

Add proper evaluation implementation for operation, var_check and var_ref attributes in the field entry of a record.

be able to work with a set of values in maps This introduces YAML Path selection ($.blah['key1','key2']) syntax and enriches yamlfilecontent test and schema with EntityStateRecordType and EntityItemRecordType result elements for collecting and checking complex objects in YAML/JSON documents. This change also depend on removal of the 'only lower case' restriction from the name attribute of the field element (EntityXxxFieldType).

And adjust probe implementation and tests for the workaround (lowercase and escape capital letters in field names with ^, use # instead of empty field names for scalars).

schemas/oval/5.11.3/independent-system-characteristics-schema.xsd

src/OVAL/probes/independent/yamlfilecontent_probe.c

jan-cerny · 2020-09-29T11:37:33Z

looks great

jan-cerny · 2020-10-01T13:49:51Z

Let's merge it now, any other improvements can be opened separately.

evgenyz force-pushed the improve-yaml-probe branch from 5941773 to a648dc9 Compare June 25, 2020 22:15

evgenyz requested review from jan-cerny and yuumasato June 25, 2020 22:17

evgenyz added this to the 1.3.4 milestone Jun 25, 2020

evgenyz force-pushed the improve-yaml-probe branch from a648dc9 to c4f3537 Compare June 26, 2020 05:42

matejak reviewed Jun 30, 2020

View reviewed changes

schemas/oval/5.11.3/independent-definitions-schema.xsd Outdated Show resolved Hide resolved

tests/probes/yamlfilecontent/test_probes_yamlfilecontent_array.xml Outdated Show resolved Hide resolved

yuumasato reviewed Jun 30, 2020

View reviewed changes

tests/probes/yamlfilecontent/test_probes_yamlfilecontent_key.xml Outdated Show resolved Hide resolved

schemas/oval/5.11.3/independent-system-characteristics-schema.xsd Outdated Show resolved Hide resolved

schemas/oval/5.11.3/independent-definitions-schema.xsd Outdated Show resolved Hide resolved

jan-cerny reviewed Jun 30, 2020

View reviewed changes

evgenyz force-pushed the improve-yaml-probe branch from 396cfd4 to bc37fce Compare July 17, 2020 07:18

yuumasato reviewed Jul 21, 2020

View reviewed changes

src/OVAL/results/oval_cmp_impl.h Show resolved Hide resolved

evgenyz force-pushed the improve-yaml-probe branch 2 times, most recently from 37f4fb6 to be38435 Compare July 23, 2020 05:32

evgenyz marked this pull request as ready for review July 23, 2020 05:32

matusmarhefka reviewed Jul 23, 2020

View reviewed changes

tests/probes/yamlfilecontent/test_probes_yamlfilecontent_key.xml Show resolved Hide resolved

jan-cerny reviewed Jul 27, 2020

View reviewed changes

tests/probes/yamlfilecontent/test_probes_yamlfilecontent_types.sh Outdated Show resolved Hide resolved

evgenyz force-pushed the improve-yaml-probe branch 2 times, most recently from 571d179 to 46cf732 Compare September 28, 2020 21:00

evgenyz added 2 commits September 28, 2020 23:15

OVAL: Fix evaluation of field entries in records

b8dcde2

Add proper evaluation implementation for operation, var_check and var_ref attributes in the field entry of a record.

evgenyz added 4 commits September 28, 2020 23:15

OVAL: Fix record field evaluation

4f129b7

probes/yamlfilecontent: Avoid collecting empty records

277548a

Revert changes in base OVAL schema

8d64db6

And adjust probe implementation and tests for the workaround (lowercase and escape capital letters in field names with ^, use # instead of empty field names for scalars).

Bump yaml-filter

5f9b9b6

evgenyz force-pushed the improve-yaml-probe branch from 46cf732 to 5f9b9b6 Compare September 28, 2020 21:15

JAORMX approved these changes Sep 29, 2020

View reviewed changes

jan-cerny reviewed Sep 29, 2020

View reviewed changes

schemas/oval/5.11.3/independent-system-characteristics-schema.xsd Outdated Show resolved Hide resolved

jan-cerny reviewed Sep 29, 2020

View reviewed changes

src/OVAL/probes/independent/yamlfilecontent_probe.c Show resolved Hide resolved

evgenyz added 2 commits September 29, 2020 15:31

Update schemas

183e351

probes/yamlfilecontent: Add missed SEXP_free

227cce0

evgenyz force-pushed the improve-yaml-probe branch from dde5fb8 to 227cce0 Compare September 29, 2020 13:32

jan-cerny self-assigned this Oct 1, 2020

jan-cerny merged commit 0693cb3 into OpenSCAP:maint-1.3 Oct 1, 2020

evgenyz deleted the improve-yaml-probe branch July 9, 2021 07:32

probes/yamlfilecontent: Bump yaml-filter, extend the schema and probe to be able to work with a set of values in maps #1552

probes/yamlfilecontent: Bump yaml-filter, extend the schema and probe to be able to work with a set of values in maps #1552

Uh oh!

Conversation

evgenyz commented Jun 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

evgenyz commented Jun 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

evgenyz commented Jun 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

evgenyz commented Jun 25, 2020

Uh oh!

evgenyz commented Jun 26, 2020

Uh oh!

yuumasato commented Jun 30, 2020

Uh oh!

evgenyz commented Jun 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matejak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yuumasato left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yuumasato commented Jun 30, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

evgenyz commented Jun 30, 2020

Uh oh!

evgenyz commented Jul 15, 2020

Uh oh!

yuumasato left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

evgenyz commented Jul 31, 2020

Uh oh!

Uh oh!

Uh oh!

jan-cerny commented Sep 29, 2020

Uh oh!

jan-cerny commented Oct 1, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

evgenyz commented Jun 25, 2020 •

edited

Loading

evgenyz commented Jun 25, 2020 •

edited

Loading

evgenyz commented Jun 25, 2020 •

edited

Loading

evgenyz commented Jun 30, 2020 •

edited

Loading

yuumasato left a comment •

edited

Loading