Abstract relationships as top-level constructs rather than embedded within other constructs

[terrymacdonald]

The current way relationships are conveyed (only in objects) has two issues.

Firstly, is that I cannot currently 'agree' with a relationship that someone has asserted between objects they have published. In some cases it is extremely valuable for someone to say they also think that object A and Object B are related. Currently I would need to produce my own object and assert the relationship from my own object to that object. Its for this reason that I would like to propose that Relationships should be able to be sent independently of the objects they refer to. In that case a third-party will be able to send their agreement that a relationship is valid. A subsequent consumer can then easily see that 14 others agree the suggested relationship is valid and only 2 say its not - therefore the relationship is more likely to be valid.

e.g. mycompany agrees completely with the assertion that thirdparty made that there is a relationship between the two objects that thirdparty published.

<stix:Relationships>
    <stix:Relationship id="mycompany:relationship-2c560b47-7b11-48a1-bcda-968dfbe0ceba" timestamp="2015-03-11T01:01:01+00:00" xsi:type='relationship:RelationshipType'>
        <stixCommon:Confidence>
            <stixCommon:Value xsi:type="stixVocabs:HighMediumLowVocab-1.0">High</stixCommon:Value>
        </stixCommon:Confidence>
        <stixCommon:Information_Source>
            <stixCommon:Identity id="mycompany:Identity-50790476-b684-11e3-9149-0800271e87d2">
                <stixCommon:Name>MyCompany, Inc.</stixCommon:Name>
            </stixCommon:Identity>
        </stixCommon:Information_Source>
        <relationship:Object>
            <indicator:Indicator id="thirdparty:Indicator-33fe3b22-0201-47cf-85d0-97c02164528d" xsi:type='indicator:IndicatorType' timestamp="2015-02-20T09:00:00.000000Z">
        </relationship:Object>
        <stixCommon:Relationship xsi:type="stixVocabs:AssertionAgreementVocab-1.0">Agrees</stixCommon:Relationship>
        <relationship:Related_Object>
            <indicator:Indicator id="thirdparty:Indicator-8cc8061e-2bed-496b-882e-2c4ad1b30f23" xsi:type='indicator:IndicatorType' timestamp="2015-02-27T13:00:00.000000Z">
        </relationship:Related_Object>
    </stix:Relationship>

^ That's effectively a [+1] from mycompany!.

The second use of the Top-level relationship object is to agree with the object that someone else sent. Note this is different from agreeing with the relationship between objects (the edges) that the previous example was describing. This use of the relationship object is agreeing with the object itself (the nodes in knowledge-graph-speak).

e.g. mycompany agrees completely that the assertion that thirdparty made in their info is correct.

<stix:Relationships>
    <stix:Relationship id="mycompany:relationship-2c560b47-7b11-48a1-bcda-968dfbe0ceba" timestamp="2015-03-11T01:01:01+00:00" xsi:type='relationship:RelationshipType'>
        <stixCommon:Confidence>
            <stixCommon:Value xsi:type="stixVocabs:HighMediumLowVocab-1.0">High</stixCommon:Value>
        </stixCommon:Confidence>
        <stixCommon:Information_Source>
            <stixCommon:Identity id="mycompany:Identity-50790476-b684-11e3-9149-0800271e87d2">
                <stixCommon:Name>MyCompany, Inc.</stixCommon:Name>
            </stixCommon:Identity>
        </stixCommon:Information_Source>
        <relationship:Object>
            <indicator:Indicator id="thirdparty:Indicator-33fe3b22-0201-47cf-85d0-97c02164528d" xsi:type='indicator:IndicatorType' timestamp="2015-02-20T09:00:00.000000Z">
        </relationship:Object>
        <stixCommon:Relationship xsi:type="stixVocabs:AssertionAgreementVocab-1.0">Agrees</stixCommon:Relationship>
    </stix:Relationship>
</stix:Relationships>

I think we need this functionality to crowdsource agreement from communities.

[johnwunder]

This is a duplicate of #70 but given that this write-up is better I'll close that one.

[jordan2175]

I agree completely with Terry. This could add so many needed and interesting ways of expressing data. It goes without saying.....

1. The ID field or IDREF field here should be required
2. There should be NO inline content in these, just like the report object.
3. These should be first-class citizens and you should be able to send them by themselves in a STIX Package

[terrymacdonald]

oops. wrong button.

[MarkDavidson]

Here's a notional structure:

   <stix:Relationship 
                      id="example:Relationship-some-uuid-5555"
                      from_idref="example:Indicator-some-uuid-1234" 
                      to_idref="example:Campaign-some-other-uuid-0987">
       <stix:Relationship_Type>indicates</stix:Relationship_Type> <!-- Could be a [1,n] construct to permit multiple relationships -->
       <stix:Confidence>...</stix:Confidence>
       <stix:Information_Source>...</stix:Information_Source>
       <!-- Is other stuff necessary? -->
   </stix:Relationship>

I think there are some positives here:

• There are currently a bunch (17? It's been a while since I looked) of locations in the schema where relationships can exist. This notional structure could help reduce that to a single location (provided it's a top level construct) and would hopefully simplify processing.
• In STIX 1.1.1, modifications to relationships within the language require structural changes to the schema (e.g., adding/removing elements/attributes). With this notional structure, adding/removing relationships could be as straightforward as updating a controlled vocabulary (this is still a schema change, but it's not structural).
• Relationships can be made without modification to the objects they relate. In STIX 1.1.1, adding a "Related_Campaign" (for instance) requires modifying the Indicator itself (aka versioning). A top level relationship construct would permit relationship modifications that don't require any modification to the related objects (e.g., Indicators).
• My sense is that this can be implemented in a backward compatible manner, provided all existing relationships could be mapped into whatever form the "new" construct might take.

I think this idea hits all requirements from @jordan2175.

As for @terrymacdonald, you have an interesting problem statement:

[...] I cannot currently 'agree' with a relationship [...]

What about a "relationship assertion" where you specify the relationship you want to make an assertion about, plus an assertion? Something like

<stix:Relationship_Assertion
       id="example:assertion-uuid-9999"
       idref="example:Relationship-some-uuid-5555">
   <stix:Assertion_Type>+1</stix:Assertion_Type>
   <stix:Confidence>...</stix:Confidence>
   <stix:Information_Source>...</stix:Information_Source>
   <!-- Is other stuff necessary? -->
</stix:Relationship_Assertion>

I also toyed with the idea that a Relationship Assertion would just be a special type of relationship where the from_idref would be required (the idref refers to the relationship you are making the assertion about), the to_idref would be prohibited, and the Relationship_Type would contain the assertion. I wasn't sure which was better, but I wanted to concretely put a single one forward.

Thoughts?

-Mark

[jordan2175]

I really like those ideas @MarkDavidson, brilliant. The +1 or -1 also GREATLY simplifies one of the major long term use cases. This would be so easy to implement in a Facebook/Instagram like APP for STIX. I also like the idea of getting rid of all of the other relationship gunk all throughout SITX. This would greatly clean up the data model and speed up processing and code logic. @terrymacdonald thanks for push for this. This is really going to make things better.

[terrymacdonald]

Yeah I like that a lot Mark. We need to do a lot of thinking about having a separate Relationship_Assertion object though. I went with the combined Relationship_Assertion and Relationship object, mainly because it would simplify the concept for new people learning the model. The idea was that there was one object to send relationships, and the rest are the 'data nodes'.

That said, on second thoughts there is merit in having a separate Assertion object....

Its not just relationships we want people to be able to assert. Its also the data in an object. We want others to be able to say yes that object is right, but the relationships with this other object is wrong. So how about making the relationship_assertion object into a generic assertion object? e.g.

<stix:Assertion
        id="example:assertion-uuid-9999"
        idref="example:Relationship-some-uuid-5555">
    <stix:Assertion_Type>+1</stix:Assertion_Type>
    <stix:Confidence>...</stix:Confidence>
    <stix:Information_Source>...</stix:Information_Source>
    <!-- Is other stuff necessary? -->
 </stix:Assertion>

That would cover the relationship case, but also the data node case.

Comments?
Cheers
Terry MacDonald

[jordan2175]

Very interesting.... And I like where this is going. This would greatly simplify a lot of weirdness we currently have and allow much more interesting use cases to be solved.

[MarkDavidson]

@terrymacdonald, I'd like to discuss one of your paragraphs:

Its not just relationships we want people to be able to assert. Its also the data in an object. We want others to be able to say yes that object is right, but the relationships with this other object is wrong. So how about making the relationship_assertion object into a generic assertion object?

If we operate under the assumption that a relationship is a first class object, then a relationship wouldn't count as data in an object. You could disagree with a relationship using the assertion construct from above.

However, I think there is still a need to make assertions about data in objects. I reflected for a bit on what assertions you might want to make about data in an object, and I came to the conclusion that an assertion about data in an object is probably a sighting. (e.g., "I've seen that IP also!") Said another way, a sighting is a type of assertion.

Building on the assertion idea, there is certainly overlap across relationship/assertion concepts. I can think of three things that are relevant to this discussion:

• Relationship - Assert that two objects have a specified relationship
• Assertion - Convey an "agree", "disagree", "false positive" or similar assertion (sorry, circular definition) about an object
• Sighting - Assert that an object has been seen by someone who didn't create the object.

I got tired of typing the XML, so I just went with a list of properties to discuss a notional assertion object (it's python color-coded, but don't read too much into that)

class Assertion:
    id_ # The ID of the assertion itself
    from_idref  # The ID of the "from" object
    to_idref  # The ID of the "to" object
    assertion_type  # The type of assertion: Relationship, Sighting, Assertion
    assertion  # The assertion being made: Indicates, +1, Disagree
    information_source  # Info source of this assertion
    confidence  # A confidence assertion of this assertion_type / assertion
    # Is other information needed? Rationale for the assertion might be useful

A relationship could look like this:

Assertion(
      id_ = 'Assertion-1',
      from_idref='Indicator-1',
      to_idref='Campaign-1',
      assertion_type='Relationship'
      assertion='Indicates'
      information_source='Threat Analyst 1',
      confidence='High',
      rationale='I saw this during my analysis')

A sighting could look like this. Note that sighting specific components of an object would get more complex. A possible tearline here is that you can only "sight" things that have IDs in order to keep the Sighting Assertion simple.

Assertion(
      id_ = 'Assertion-2',
      from_idref='Indicator-1',
      # to_idref  # Not populated in this scenario
      assertion_type='Sighting'
      # assertion  # May or may not need to be populated.
      information_source='Threat Analyst 2',
      # confidence # Is confidence irrelevant for a sighting?
      rationale='This was seen by our IDS'
     )

Lastly, an assertion could look like this:

Assertion(
      id_ = 'Assertion-3',
      from_idref='Assertion-1',
      # to_idref  # Not used for assertion_type == Assertion
      assertion_type='Assertion'
      assertion='+1'
      information_source='Threat Analyst 3',
      confidence='High',
      rationale='I confirmed the relationship during independent analysis'
     )

I think there's a potential discussion point for splitting Relationships (which have both 'from' and 'to' properties) and Assertions (which would only have a from property), but I'll end my message here.

-Mark

[jordan2175]

Lets figure out 10-20 use cases and make sure this solves all the problems. The idea of doing relationships here and not inside of data objects, is very attractive.

[terrymacdonald]

Hi Mark,

Thanks for this. I'm not a fan of conflating these together. I believe all
we will be doing is making it more difficult for people to understand.
Technically we may be able to combine these all together, but we've seen
the confusing that can cause before (Observable patterns and instances).

When it comes down to it, the entire STIX package we generate is an
assertion. We could probably force all objects into the Assertion object if
we wanted, but all that would do is confuse people.

I prefer to keep them separated (https://www.youtube.com/watch?v=XN32lLUOBzQ
).

I believe the three concepts we are trying to allow STIX to convey from
producer to consumer are:

1. "These two things are related"
2. "I agree"/"I do not agree"
3. "I've seen that too"

In order to keep this simple to understand for new players, I still
maintain we need to keep the separation of the above concepts into
individual objects.

As mentioned on the STIX mailing list I believe the tweaked Sighting Object
I've proposed here (#306) is
NOT just "I've seen that too" (number 3), but is instead intended to simply
convey "I've seen this interesting thing". This is a subtle but important
difference. We need to separate the "stuff I've seen" from the "stuff I'm
looking for", and recognize that the "stuff I've seen" is a superset of
"I've seen that too".

So to describe each concept individually and how I propose we tackle them...

1. "These two things are related"

I believe that this should still be described using the relationship object
Mark first proposed. It is simple, is easy to understand and works.

<stix:Relationship
id="example:Relationship-some-uuid-5555"
from_idref="example:Indicator-some-uuid-1234"
to_idref="example:Campaign-some-other-uuid-0987">
stix:Relationship_Typeindicates/stix:Relationship_Type
stix:Confidence.../stix:Confidence
stix:Information_Source.../stix:Information_Source

/stix:Relationship

2. "I agree"/"I do not agree"

Assertion might not be the right word for the object we need to use here.
Maybe attestation is a better word? Or Opinion?

After reading your text I have to disagree that the Assertion object (I'll
keep calling it that) is likely only related to Sightings. I also see
people sending out [+1]'s for Indicators (saying its a good or bad pattern)
and potentially Courses of Action as well. So there is at least worth in
having the ability to [+1] a node.

Here's Mark's object with the Attestation name instead (just because I like
it)

<stix:Attestation
id="example:assertion-uuid-9999"
idref="example:Indicator-some-uuid-5555">
stix:Attestation_Typeaggree/stix:Attestation_Type
stix:Confidence.../stix:Confidence
stix:Information_Source.../stix:Information_Source

/stix:Attestation

3. "I've seen that too"

I think that this could be covered within the tweaked Sighting Object that
I proposed I've proposed here (
#306). IMHO we need to
separate the "stuff I've seen" (Sightings) from the "stuff I'm looking for"
(Indicators). "I've seen that too" is covered by the 'stuff I've seen''
functionality that I think should be covered by the Sightings object.

The "stuff I've seen" idea can be conveyed with something like this. In
this example we have detected an instance of something that we think is
worth capturing for later. We can add it to our knowledge base, but it
doesn't necessarily need to have any relationships at present, as we may be
able to form them in the future when get more information. At this stage,
we are just recording it because its a bit odd:

<stix:Sighting id="example:Sighting-0756a255-6623-4226-8356-015396918b38"
timestamp="2014-08-22T20:17:38.959000+00:00"
xsi:type='Sighting:SightingType' negate="false" version="2.1.1">
Sighting:Title192.168.1.1/Sighting:Title
Sighting:Detected_Time2014-01-31T12:48:50Z/Sighting:Detected_Time
<Sighting:Type xsi:type="stixVocabs:SightingTypeVocab-1.1">IPv4
Address/Sighting:Type
<Sighting:Observable
id="example:Observable-c46356e0-9d93-408b-ad2b-3aa2b561cfe9">
<cybox:Object id="example:Address-d1482a44-e95b-4ebc-abba-0486e0df7ea9">
<cybox:Properties xsi:type="AddressObj:AddressObjectType"
category="ipv4-addr">
<AddressObj:Address_Value
condition="Equals">192.168.1.1/AddressObj:Address_Value
/cybox:Properties
/cybox:Object
/Sighting:Observable
/stix:Sighting

The "I've seen that too" can be covered by this SIghting object as well. In
this example we have detected an instance of a pattern that another Org
shared through an Indicator at an earlier time:

<stix:Sighting id="example:Sighting-0756a255-6623-4226-8356-015396918b38"
timestamp="2014-08-22T20:17:38.959000+00:00"
xsi:type='Sighting:SightingType' negate="false" version="2.1.1">
Sighting:Title192.168.1.1/Sighting:Title
Sighting:Detected_Time2014-01-31T12:48:50Z/Sighting:Detected_Time
<Sighting:Type xsi:type="stixVocabs:SightingTypeVocab-1.1">IPv4
Address/Sighting:Type
<Sighting:Observable
id="example:Observable-c46356e0-9d93-408b-ad2b-3aa2b561cfe9">
<cybox:Object id="example:Address-d1482a44-e95b-4ebc-abba-0486e0df7ea9">
<cybox:Properties xsi:type="AddressObj:AddressObjectType"
category="ipv4-addr">
<AddressObj:Address_Value
condition="Equals">192.168.1.1/AddressObj:Address_Value
/cybox:Properties
/cybox:Object
/Sighting:Observable
Sighting:Related_Indicators
<stixCommon:Indicator
idref="yourorg:indicator-d6b60324-a1b8-4d51-8c26-3aef6351371d"
xsi:type='indicator:IndicatorType' version="1.1.1"/>
/Sighting:Related_Indicators
/stix:Sighting

I believe that they three objects as described above cover the scenarios
that I've described, and give us the flexibility to cover any scenarios I
can think of. It keeps things simple to understand which will help when new
players come onboard, and won't require huge amounts of documentation to
describe.

Thoughts?

Cheers
Terry MacDonald

On 7 April 2015 at 22:32, MarkDavidson notifications@github.com wrote:

@terrymacdonald https://github.com/terrymacdonald, I'd like to discuss
one of your paragraphs:

Its not just relationships we want people to be able to assert. Its also
the data in an object. We want others to be able to say yes that object is
right, but the relationships with this other object is wrong. So how about
making the relationship_assertion object into a generic assertion object?

If we operate under the assumption that a relationship is a first class
object, then a relationship wouldn't count as data in an object. You could
disagree with a relationship using the assertion construct from above.

However, I think there is still a need to make assertions about data in
objects. I reflected for a bit on what assertions you might want to make
about data in an object, and I came to the conclusion that an assertion
about data in an object is probably a sighting. (e.g., "I've seen that IP
also!") Said another way, a sighting is a type of assertion.

Building on the assertion idea, there is certainly overlap across
relationship/assertion concepts. I can think of three things that are
relevant to this discussion:

• Relationship - Assert that two objects have a specified relationship
• Assertion - Convey an "agree", "disagree", "false positive" or
  similar assertion (sorry, circular definition) about an object
• Sighting - Assert that an object has been seen by someone who didn't
  create the object.

I got tired of typing the XML, so I just went with a list of properties to
discuss a notional assertion object (it's python color-coded, but don't
read too much into that)

class Assertion:
id_ # The ID of the assertion itself
from_idref # The ID of the "from" object
to_idref # The ID of the "to" object
assertion_type # The type of assertion: Relationship, Sighting, Assertion
assertion # The assertion being made: Indicates, +1, Disagree
information_source # Info source of this assertion
confidence # A confidence assertion of this assertion_type / assertion
# Is other information needed? Rationale for the assertion might be useful

A relationship could look like this:

Assertion(
id_ = 'Assertion-1',
from_idref='Indicator-1',
to_idref='Campaign-1',
assertion_type='Relationship'
assertion='Indicates'
information_source='Threat Analyst 1',
confidence='High',
rationale='I saw this during my analysis')

A sighting could look like this. Note that sighting specific components
of an object would get more complex. A possible tearline here is that you
can only "sight" things that have IDs in order to keep the Sighting
Assertion simple.

Assertion(
id_ = 'Assertion-2',
from_idref='Indicator-1',
# to_idref # Not populated in this scenario
assertion_type='Sighting'
# assertion # May or may not need to be populated.
information_source='Threat Analyst 2',
# confidence # Is confidence irrelevant for a sighting?
rationale='This was seen by our IDS'
)

Lastly, an assertion could look like this:

Assertion(
id_ = 'Assertion-3',
from_idref='Assertion-1',
# to_idref # Not used for assertion_type == Assertion
assertion_type='Assertion'
assertion='+1'
information_source='Threat Analyst 3',
confidence='High',
rationale='I confirmed the relationship during independent analysis'
)

I think there's a potential discussion point for splitting Relationships
(which have both 'from' and 'to' properties) and Assertions (which would
only have a from property), but I'll end my message here.

-Mark

—
Reply to this email directly or view it on GitHub
#291 (comment).

[jordan2175]

For the I have seen this too, I am thinking that it would be best to reference the original ID instead of creating a new ID.

[athiasjerome]

I agree with Terry (KISS)

[athiasjerome]

An --example-- of Relationship data model could be found in
http://csrc.nist.gov/publications/nistir/ir7693/NISTIR-7693.pdf
6.4.1 Relationship Data Model (Page 21)

You could note, again just for example, the 'scope' (inclusive/exclusive)

[athiasjerome]

An interesting (easy to understand) description of the Relationship construct could be found in the "Entity-Relationship" chapter here
http://www.adaptiveobjectmodel.com/WICSA3/ArchitectureOfAOMsWICSA3.pdf

[athiasjerome]

I suggest to name this new construct RelationshipStatement
this will allow use of algorithms to handle multiple statements regarding a relationship