python-stix-v1.1.* isn't backward-compatible with earlier versions of STIX (ie, 1.0)

[treyka]

Shy of bundling multiple releases of python-stix with my application and iteratively loading them in reverse chronological order via try/except blocks (the "Hail Mary" of dependency management) what do y'all suggest?

[bworrell]

Hi @treyka, thanks for the question! Each version of python-stix is designed to work with a particular version of the STIX Language (i.g., python-stix 1.1.x.y works with STIX v1.1.x, python-stix v1.0.x.y works with STIX v1.0.x, etc.).

I do not recommend this at all, but you can force python-stix v1.1.x.y to ignore version checking on parse by doing the following:

from stix.utils.parser import EntityParser

stix_document = your_stix_document_retrieval_function_not_found_in_python_stix()
parser = EntityParser()
stix_package = parser.parse_xml(stix_document, check_version=False)

That could potentially lead to all kinds of problems, so proceed with that at your own risk! :)

Otherwise, I would go with what you suggested--packaging multiple versions of python-stix with your application if you are going to work with multiple versions of STIX. You could peek at the version of the document (inspect the version attribute on the STIX_Package and then pass the document to the appropriate version of python-stix).

Definitely a tricky problem! Let us know if you have any other questions.

Thanks,
Bryan Worrell

[treyka]

Hey, Bryan -

Thanks for the quick feedback. Given the situation, it seems wisest to sniff the document header, see what version of STIX we're looking at, and load the appropriate libraries.

Given that python-stix depends on python-cybox and you could have a CybOX 2.0 payload in a STIX 1.1 document, should I also look further into the document to determine the CybOX payload version, load the version of python-cybox that corresponds to the payload, then load the version of python-stix that corresponds to the document as a whole?

This could be more elegant. :-/

[bworrell]

Hey @treyka,

STIX v1.1 imports the CybOX v2.1 schemas, so even though it is possible for a CybOX 2.0 document to live in a STIX document (i.e., the CybOX content is defined by the "http://cybox.mitre.org/cybox-2" namespace), the content is expected to be valid CybOX 2.1. This is reflected in the APIs, as python-stix v1.1.x.y depends on python-cybox v2.1.x.y.

Provided that your CybOX content is valid against CybOX 2.1, the underlying python-cybox v2.1.x.y API should be able to handle it without needing to wire up a bunch of dynamic parsers.

You are definitely attempting to do something tricky with the APIs as they are all designed to be used for the creation/parsing/manipulation of a particular version of their respective language. I'm eager to hear about your design/workflow/architecture as you continue to work on this.

Thanks,
Bryan Worrell

[treyka]

Hey, @bworrell -

python-stix has two major use cases:

1. creating content

2. parsing content

3. In terms of creating content, the user has control. An organization (say, a security vendor or an ISAC) can arbitrarily standardize on particular schema version(s) for all content they produce. The day that STIX 2.0 comes out, they can say, "Hey, downstream consumers, as of 90 days from now all our content will be in STIX 2.0 so you'd better get ready."

4. If you're parsing data generated by others, you have less control. I can, say, arbitrarily choose to reject STIX input < version 1.1. But what about going forward? Presumably these standards are going to be around for a while. The issue I'm looking at today with managing multiple versions of python-{stix,cybox} libs in order to handle backward compatibility is exactly the same issue I'll face the next time a major schema revision gets released. As these standards gain traction, there's going to be a lot more data sloshing around out there keyed off different schema versions. (See what I mean?)

The obvious solution would be for the python-{stix,cybox} libs to (eventually) be rearchitected to support a range of schema revisions.

[treyka]

Was just working on a pull request related to another issue and forked the branch head, which happens to be for STIX 1.1.1. Tried importing one of your (MITRE's) sample indicators as follows:

from stix.core import STIXPackage
my_stix_package = STIXPackage.from_xml('fireeye-pivy-report-with-indicators.xml')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "./python-stix/stix/core/stix_package.py", line 429, in from_xml
    return parser.parse_xml(xml_file)
  File "./python-stix/stix/utils/parser.py", line 99, in parse_xml
    self._check_version(tree)
  File "./python-stix/stix/utils/parser.py", line 37, in _check_version
    raise UnsupportedVersionError("Your python-stix library supports STIX %s. Document version was %s" % (supported_stix_version, document_version))
stix.utils.parser.UnsupportedVersionError: Your python-stix library supports STIX 1.1.1. Document version was 1.1

Seriously?! python-stix 1.1.1 isn't compatible with STIX 1.1?! Guys, this is getting ridiculous!!! It's just a point release!!!

[bworrell]

Hi Trey,

Sorry to hear you are having difficulties parsing STIX v1.1 content! STIX v1.1.1 included some changes that were not backwards compatible with STIX v1.1. For example, the stixCommon:StatementType/Source field was changed from ControlledVocabularyStringType to InformationSourceType. As such, STIX v1.1 content should not be parsed by python-stix v1.1.1.*.

The fireeye-pivy-report-with-indicators.xml file found here (https://github.com/STIXProject/schemas/blob/master/samples/Poison%20Ivy/fireeye-pivy-report-with-indicators.xml) is versioned as 1.1.1. Have you updated your fork with the current master content?

I should note that I have tried to parse this document but receive and error stating, cybox.utils.nsparser.UnknownObjectType: CodeObjectType does not have a specified API class because python-cybox does not have the Code Object implemented yet.

As mentioned in a previous message, you can disable the version checking and attempt to parse your content anyway, though I do not recommend this at all.

from stix.utils.parser import EntityParser

stix_document = your_stix_document_retrieval_function_not_found_in_python_stix()
parser = EntityParser()
stix_package = parser.parse_xml(stix_document, check_version=False)

You can also try changing the version attribute value on the STIX_Package to 1.1.1 and python-stix should attempt to parse your content, though it could just result in more issues popping up (so I don't recommend doing this).

I hope that helps!

[treyka]

Hi, Bryan -

I'm using lxml to sniff the document header, modifying sys.path accordingly, and loading the appropriate versions of python-{stix,cybox} . This works but it's bloody inelegant and means that I have to bundle a bunch of different versions of the python-{stix,cybox} libraries and track multiple branch heads for bug fixes.

Imagine a scenario in which I have a STIX file with idrefs pointing to previously ingested and serialized observable data that's was encoded in multiple versions of CybOX at different points in time. In order to interact with this, I'll be unloading and reloading libraries like mad.

The point isn't that I'm "having difficulties parsing STIX v1.1 content". Please step back from the trees and look at the forest!

[bworrell]

Hey Trey,

Thanks for all the feedback! We hear you about the complexities of working with multiple versions of STIX/CybOX in python-stix/python-cybox and want to let you know that we are thinking this through and discussing ways that we might be able to help. I hope to get back to you soon so we can continue this discussion!

Thanks,
Bryan Worrell

[treyka]

Bryan -

Cf. the Code Object in python-cybox not being implemented yet, I opened this issue a while back.

[bworrell]

@treyka,

Sorry for the delay here, but we've made some progress in the realm of backwards compatibility! We released v1.0.0-alpha.3 of stix-ramrod just yesterday, which provides APIs and scripts for updating STIX and CybOX content.

Because stix-ramrod is still in alpha, it hasn't been integrated into python-stix yet, but we're currently thinking through the best way of getting this done. For the time being, I have documented a method for working with stix-ramrod and python-stix for updating and parsing STIX content: http://stix-ramrod.readthedocs.org/en/latest/api/examples.html#working-with-python-stix. I hope to add more documentation that focuses on more complex upgrade scenarios in the near future.

Hopefully this is helpful to you and others who want to work with older STIX content in python-stix!

Thanks,
Bryan Worrell

[packet-rat]

Excellent news Bryan! I was just starting to try to figure out this morning how to update 100'a of STIX packages I use for testing/development. I know of others who are faced with the same challenge so thee timing couldn't be better!

Patrick Maroney
(609)841-5104

On Oct 9, 2014, at 11:50 AM, Bryan Worrell notifications@github.com wrote:

@treyka,

Sorry for the delay here, but we've made some progress in the realm of backwards compatibility! We released v1.0.0-alpha.3 of stix-ramrod just yesterday, which provides APIs and scripts for updating STIX and CybOX content.

Because stix-ramrod is still in alpha, it hasn't been integrated into python-stix yet, but we're currently thinking through the best way of getting this done. For the time being, I have documented a method for working with stix-ramrod and python-stix for updating and parsing STIX content: http://stix-ramrod.readthedocs.org/en/latest/api/examples.html#working-with-python-stix. I hope to add more documentation that focuses on more complex upgrade scenarios in the near future.

Hopefully this is helpful to you and others who want to work with older STIX content in python-stix!

Thanks,
Bryan Worrell

—
Reply to this email directly or view it on GitHub.=

[sarus]

Just started looking at STIX parsing and came across this thread. Was quite concerned by this comment:

STIX v1.1.1 included some changes that were not backwards compatible with STIX v1.1

The versioning scheme being used by the project suggests the use of semantic versioning but is not following the semver conventions:

http://semver.org/

Namely backwards compatibility must be maintained across patch version (i.e., v1.1.0 -> v1.1.1). In fact, in semver backwards compatibility must be maintained across the same major version. Have you given any thought to adopting semver to prevent this type of confusion in the future?

Apologies if you've already moved in this direction. I'm still very new to this project. Thanks!

[bworrell]

Hi @sarus! Thanks for the comment! The STIX language defines it's versioning policy here: https://stix.mitre.org/language/versioning_policy.html which is a quite a bit different than semantic versioning (as you noticed, heh). Check it out if you get a chance.

Several of our utilities and APIs follow semver (e.g., stix-ramrod). The data model APIs (python-stix/python-cybox/python-maec) orient themselves around the version of the language, which is definitely confusing at times because it tries to reconcile software versioning with the parent language/schema versioning, though the two aren't always super compatible :-\

Let us know if you have any thoughts on how to apply semantic versioning (or other versioning scheme) concepts to the STIX language! I also encourage you to bring these ideas up on our discussion lists :)

[guitarmanvt]

I agree with @sarus: the versioning difference between STIX and semver.org is jarring. However, keeping python-stix in lock-step with the STIX version is probably less confusing overall than trying to maintain two separate versioning schemes.

That said, consider this potential solution:

1. python-stix version numbers remain in lock-step with STIX version numbers.
2. A separate project implements a "wrapper" around multiple versions of python-stix, insulating projects against newer STIX library explosions on data that worked fine with older STIX libraries. This new project would use semantic versioning ala semver.org.

I thought about suggesting that the "sweet wrapper" project be named pixy-stix (https://en.wikipedia.org/wiki/Pixy_Stix). Unfortunately, pixy is already taken (http://www.cmucam.org/projects/cmucam5/wiki).

IMHO STIX itself should also use Semantic versioning.