Skip to content

FS ISAC Proposal: Add Report Object as Minor Release

Jump to bottom Edit New page
johnwunder edited this page Sep 11, 2014 · 1 revision

FS-ISAC has proposed that a new STIX "Report Object" be added in the next minor release of STIX. This release would be 1.2 and would include that addition as well as other backwards-compatible fixes.

Background

In STIX 1.1.1 the STIX_Package element is used to convey a set of STIX component content (Indicators, Campaigns, etc.) and may optionally include a characterization of any context associating that content together (e.g. what type of content is it, is it related to a particular campaign, threat actor, TTP, etc.). As such, the common STIX_Package structure supports:

  • a role as an "envelope" to contain content (Indicators, Campaigns, etc.)
  • a role as a "report" to give context to some set of content (which may be defined elsewhere)
  • a role as both the set of content (envelope) and the context for it ("report").

The role a STIX_Package plays is determined based on the presence or absence of the contextual characterization fields. If the fields are present then the package is being used as a "report", if they are absent it's being used as an "envelope" without a report.

For example:

  • a simple "envelope" use could be an open source organization that publishes indicators daily without any associating context other than that a given set was published by them on a given day.
  • a simple "report" use could be a producer conveying that a particular set of indicators and a set of TTPs that were previously shared with independent contexts defined are now believed to also share a new context as part of an ongoing campaign. This would mean the associated content within the STIX_Package would be references to the previously shared content rather than new content defined in the STIX_Package.
  • a combined "envelope"/"report" use could be something as simple as a producer wanting to share a set of new indicators and explicitly specifying as part of the STIX_Package that the contained content is "IP Watchlist Indicators" to give the consumer a clear picture what to expect when processing the STIX_Package. This simple combined use is the most common typical use described by the STIX community to date.

Thus, STIX_Package is sometimes used to convey some kind of contextual grouping while other times is not. FS-ISAC asserts that this makes STIX_Package easy to use the wrong way (as a report to convey context, which is legal in the spec but incorrect per FS-ISAC's proposed model) and increases the chances of producer error. FS-ISAC further asserts that these explicit dual purposes increases the complexity of the STIX data model and the code required to process it.

Along those same lines, to support the varying uses described above STIX_Package currently allows both embedded content (the primary mechanism in use today) as well as @idrefs to externally-defined content.

To sum up the current state of affairs in STIX 1.1.1:

  • STIX_Package is currently used as an "envelope" for content and to convey associating context ("reports")
  • STIX_Packages used as reports (i.e. conveying some context) may either embed content or reference content

The FS-ISAC is proposing that a STIX 1.2 release be defined as follows:

  • That STIX_Package is only used as an envelope for content, and that STIX_Package is the only envelope for content in STIX.
  • That there is no explicit or implicit relationship between STIX components by virtue of being conveyed in the same envelope, i.e. STIX_Package.
  • That a new Report object is introduced and is used to explicitly express context for or affinity groupings between content.
  • That a STIX_Package always conveys content and therefore content is always embedded inside of it, and is never referenced.
  • That the new Report always relates existing content and therefore content is always referenced from a report, and is never embedded.

Proposed Changes

As explained above, the overall goal of this FS-ISAC proposal is to get to a situation where STIX_Package may only be used as an envelope to convey content (not context, such as package intent or title) and a new Report component is added and used to convey context when necessary.

Functionally, STIX_Package would continue to be the root element used to convey STIX content as it is today. Unlike today, the descriptive fields inside (Title, Package_Intent, Description, etc.) would be removed and therefore the default top-level STIX_Package could not be used to give a common title, package intent, etc. to a group of content. Instead, in order to convey that context the producer would include one or more explicit Report constructs that reference the content that was already conveyed and add the context that STIX_Package previously conveyed.

Note that this proposal does not expand capabilities currently in STIX. STIX already has the ability to convey reports via the use of the descriptive fields in STIX_Package. Instead, it adds a new component called Report to represent that context and reduces the roles of STIX_Package from conveying content plus context to simply being a raw envelope conveying content. In other words, it constrains the STIX model to prohibit the same construct from being used to both convey content and to express context around that content.

This change is implemented as follows. Note that when fields are "removed", if this change is implemented as a minor release they will instead be deprecated.

Changes to STIXType (STIX_Package)

STIXType becomes a raw container for content and will no longer convey context. Thus, the following fields would be removed:

  • @idref - Removed because it does not make sense to reference a simple wrapper around content
  • @timestamp - Removed because @timestamp is only used for versioning and as a simple content wrapper STIX_Package would not be versioned
  • STIX_Header (not removed itself, but the following fields in STIXHeaderType would be removed)
    • Title - Used to convey context so not necessary
    • Description - Used to convey context so not necessary
    • Short_Description - Used to convey context so not necessary
    • Package_Intent - Used to convey context so not necessary
  • Related_Packages - Does not really make sense for a wrapper to be related to other wrappers

Additionally, the ability to reference content from the top-level containers in STIX_Package would be removed. Content would only be embedded in these top-level containers STIX_Package in keeping with the purpose of STIX_Package as envelope-only.

The following fields would remain on STIX_Package:

  • @id, which could be used for tracking but could not be referenced in content from STIX (because it's a raw envelope)
  • STIX_Header/Profiles, which would continue to be used to specify the set of profiles that the content within complies with.
  • STIX_Header/Information_Source, which could be used to specify the source of the information conveyed in the package.
  • STIX_Header/Handling, which would continue to be used to specify handling for included content

Note that for Handling and Information_Source, as it is now it would be the responsibility of the consumer to apply the information source and handling to the included content as appropriate.

Addition of ReportType

A new type called ReportType would be created in a new report namespace. The type would be a clone of the STIX 1.1.1 version of STIXType with the following changes:

  • STIX_Header/Profiles would be removed and exist only on STIX_Package. This is because the profiles that content complies with are best conveyed on the "envelope" for that content, not from a Report.
  • @version would remain, but it would characterize the ReportObject version (1.0) rather than the STIX version (1.2) to align with the @version attribute on all other components.
  • STIX_Header would be renamed to Header and all fields would apply only to the report. More specifically:
    • On STIX_Package, Handling can be applied to any included content. On Report, the handling scope would be restricted to the report itself.
    • Similarly, on STIX_Package the Information_Source is applied to all included content. On Report, content is not included and thus the Information_Source applies only to the report itself.
  • Related_Packages would be renamed to Related_Reports

Other Changes

  • Currently, most content can reference a STIX_Package via Associated_Packages. That element would be removed and reports would always reference out to content.

Minor Release vs. Major Release

The changes above could be made in either a minor release or a major release. If made in a minor release, the fields would be deprecated but not removed. Suggested practices would be to not use them and while content that did would be valid it would not be conformant with STIX suggested practices and would not be optimized for the tooling that MITRE supports.

In a major release the elements would just be removed.

FS-ISAC is proposing this change as a minor release. Thus, the elements would be deprecated in STIX 1.2 and removed in the next major version.

Updated schemas are available for download here. A diff of the schemas and samples can be viewed here.

Enough talk. What does this look like?

Here's a few examples of content before and after the changes. These samples validate against the 1.1.1 production version and the proposed minor release (in Github on the add-report-object branch), respectively.

The 1.1.1 samples will validate against the online version of the 1.1.1 schemas. The samples with the proposed report object changes can be placed into the "samples" directory of the updated schemas.

Simple Report (Convey a simple report and the content for that report)

Simple Watchlist (Unrelated indicators, do not convey any reports)

Long Report with Content (Convey a report plus the content for that report)

Report, No Content (Convey a report without any content)

Several Reports, Same Content (Convey a set of reports describing some content, content is external)

FS-ISAC's Analysis of Implications

  1. Limiting the scope of STIX_Package (to an envelope) makes it easier to use by both consumers and producers of content.
  2. STIX_Package will be easier to understand by both humans and computers alike. Computers won't have to deal with ambiguity regarding inferred relationships and humans will be less likely to use STIX_Package the wrong way.
  3. The overall storage of STIX data within databases will be less complex due to object storage outside of the envelope (STIX_Package). Once removed from their envelopes, database stored STIX objects can use relationships to link up with their associated reports.
  4. TAXII will have more granular control over the size of multi-part TAXII messages. Today, the smallest message a TAXII server can send is limited by the size of a STIX_Package. By allowing TAXII to use STIX_Package as an envelope only, TAXII can decide which objects will be placed within the package to more effectively control the sizing of multi-part TAXII content.

MITRE's Analysis of Implications

As a Major or Minor Release

  1. The role of STIX_Package will be restricted to a single use: it NEVER explicitly specifies or implies a logical grouping of content. The newly introduced Report component would ALWAYS explicitly create a logic grouping of content. Non-Report content is ONLY conveyed outside the context of a Report.

  2. The use of contextual information (reports) in published content could potentially drop, as content creators would not by default need to create one and if context is desired, would need to take the extra step of creating a separate report. Reports would likely only be used for formal reports and existing watchlist-type feeds would probably not include them (in particular the Title and Package_Intent fields) unless consumers pressured producers to do so. In situations where Reports were not included, consumers would have less context of what they are receiving and would have to process and interpret the content themselves in order to extrapolate inferred context.

  • See the Simple Watchlist example above
  1. Content that is conveyed only for use in a single report will be somewhat larger and would require de-referencing relationships. Additionally, content in general will be larger as information source information will need to be duplicated among the individual components rather than being specified once at the package-level.
  • See the Long Report with Content example above
  1. Content referenced via external reports is equivalent to how it is now except the name of the construct used changes from STIX_Package to Report.
  • See the Report, No Content example above
  1. Content can be referenced by zero or more reports as it is now. No reports can contain content, however, so all reports are on equal footing with regards to the relationship between them and their content. Currently, reports (packages) can both contain and reference content.
  • For example, here is a case where STIX 1.1.1 has an embedded report, plus a related report that references into content.

  1. Schema (and resulting instance documents) size and complexity increases slightly as several new types will be added. Complexity of processing instance documents could arguably either increase or decrease depending on how you as an implementor feel about adding a new type to separate the dual purposes of STIX_Package. Some feel that adding new types is more complex, others feel that having one type serve dual purposes is more complex.

  2. All consumers will be forced to de-reference content coming from reports. A report can never be conveyed with the content included directly inside of it.

  • This is best illustrated by the Simple Report example

  1. The new meaning of STIX_Package as a raw envelope means that for all practical purposes consumers would need to deconstruct the package into the individual components and handle them separately. There would no longer be a place in the STIX data model for content to be linked back to the package that it was received in, meaning that maintaining provenance to the original source of the information would have to happen either on a per-component basis or happen outside of the STIX data model.

  2. For practical usage it would require that all top-level components have IDs, though this is already suggested practice.

As a Minor Release

In addition to the above considerations, there are a few more implications if this is undertaken as a minor release rather than a major release. These exist due to the fact that the existing fields in STIX_Package intended to convey context and the Related_Packages structures in each top-level component cannot be removed in a minor release due to our versioning policy.

  1. Although at a schema level this is a minor release (because it it technically backwards compatible), the processing model for STIX and overall structure of STIX documents changes significantly. Most current content uses fields that would be deprecated and many current suggested practices also suggest including these deprecated fields (specifically, Title and Package_Intent in STIX_Package). Thus, most current content would be valid against the schema but would not be forward compatible with the field deprecations.

  2. During the current major release (i.e. while STIX is at 1.x) there will be significant duplication in capabilities between STIX_Package and Report. Though the fields in STIX_Package will be deprecated they will still be there and therefore have the potential to cause confusion with multiple ways to convey the exact same information context. Additionally, consumers wishing to accept 1.2-2.0 content would potentially need to be able to process those deprecated fields as well as the new ones in order to convert them to something useful.

Discussion Questions

  1. Is this change, in general, a good idea?

  2. If it is a good idea, is the suggestion of implementing it as a minor release a good idea? Alternatively, should we just wait for a major release and do it then?

  3. Regardless of whether it's a major or minor release, is it something that should be pursued immediately or should we hold off on new releases while tools are developed to support the current version?

  4. Are any of the alternatives better? See: https://github.com/STIXProject/schemas/wiki/Report-Object-Options

Add a custom footer
Add a custom sidebar
Clone this wiki locally