Pandas 0.23.0 - json_normalize - 'STIXdatetime' object has no attribute 'nanosecond'

[Cyb3rWard0g]

Good evening Team,

I hope you guys are having a good day. I have been playing with the ATT&CK STIX content for the past week and I wanted to report an issue that I am not sure if this is an issue with Pandas or the STIX library. However, I figured it would be good to share it here first just in case I am missing something and also if anyone is having an issue when using pandas 0.23.0 (Latest Version) with ATT&CK STIX content via TAXII.

I tested ATT&CK STIX content with Pandas 0.21.0 and 0.22.0 and everything was working fine. I was getting everything fine like this:

{'contributors': 'NA',
 'data sources': [u'File monitoring',
                  u'Process monitoring',
                  u'Process command-line parameters'],
 'defense bypassed': 'NA',
 'description': u'Collected data is staged in a central location or directory prior to Exfiltration. Data may be kept in separate files or combined into one file through techniques such as Data Compressed or Data Encrypted.\n\nInteractive command shells may be used, and common functionality within cmd and bash may be used to copy data into a staging location.\n\nDetection: Processes that appear to be reading files from disparate locations and writing them to the same directory or file may be an indication of data being staged, especially if they are suspected of performing encryption or compression on the files.\n\nMonitor processes and command-line arguments for actions that could be taken to collect and combine files. Remote access tools with built-in features may interact directly with the Windows API to gather and copy to a location. Data may also be acquired and staged through Windows system management tools such as Windows Management Instrumentation and PowerShell.\n\nPlatforms: Linux, macOS, Windows\n\nData Sources: File monitoring, Process monitoring, Process command-line parameters',
 'detectable': 'NA',
 'detectable description': 'NA',
 'difficulty': 'NA',
 'difficulty description': 'NA',
 'effective permissions': 'NA',
 'matrix': u'mitre-attack',
 'network requirements': 'NA',
 'object created': '2017-05-31T21:30:58.938Z',
 'object created by ref': 'identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5',
 'object id': u'attack-pattern--7dd95ff6-712e-4056-9626-312ea4ab4c5e',
 'object marking refs': ['marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168'],
 'object modified': '2018-04-18T17:59:24.739Z',
 'object type': u'attack-pattern',
 'permission required': 'NA',
 'platforms': [u'Linux', u'macOS', u'Windows'],
 'references': 'NA',
 'remote support': 'NA',
 'system requirements': 'NA',
 'tactic': [u'collection'],
 'tactic type': 'NA',
 'technique': u'Data Staged',
 'technique id': u'T1074',
 'url': u'https://attack.mitre.org/wiki/Technique/T1074'}

However, when I tested it with Pandas 0.23.0, I got the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/Library/Python/3.6/lib/python/site-packages/pandas/core/dtypes/cast.py in try_datetime(v)
    913                                         require_iso8601=True,
--> 914                                         errors='raise')
    915         except ValueError:

pandas/_libs/tslib.pyx in pandas._libs.tslib.array_to_datetime()

pandas/_libs/tslib.pyx in pandas._libs.tslib.array_to_datetime()

ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
<ipython-input-5-e6f5fde53c7b> in <module>()
      3 print(" ")
      4 df = all_attack['techniques']
----> 5 df = json_normalize(df)
      6 df.reindex(['matrix', 'object created','tactic', 'technique', 'technique id', 'data sources'], axis=1)[0:5]

~/Library/Python/3.6/lib/python/site-packages/pandas/io/json/normalize.py in json_normalize(data, record_path, meta, meta_prefix, record_prefix, errors, sep)
    202             #       reasonably
    203             data = nested_to_record(data, sep=sep)
--> 204         return DataFrame(data)
    205     elif not isinstance(record_path, list):
    206         record_path = [record_path]

~/Library/Python/3.6/lib/python/site-packages/pandas/core/frame.py in __init__(self, data, index, columns, dtype, copy)
    385                     if is_named_tuple(data[0]) and columns is None:
    386                         columns = data[0]._fields
--> 387                     arrays, columns = _to_arrays(data, columns, dtype=dtype)
    388                     columns = _ensure_index(columns)
    389 

~/Library/Python/3.6/lib/python/site-packages/pandas/core/frame.py in _to_arrays(data, columns, coerce_float, dtype)
   7435     elif isinstance(data[0], collections.Mapping):
   7436         return _list_of_dict_to_arrays(data, columns,
-> 7437                                        coerce_float=coerce_float, dtype=dtype)
   7438     elif isinstance(data[0], Series):
   7439         return _list_of_series_to_arrays(data, columns,

~/Library/Python/3.6/lib/python/site-packages/pandas/core/frame.py in _list_of_dict_to_arrays(data, columns, coerce_float, dtype)
   7558     content = list(lib.dicts_to_array(data, list(columns)).T)
   7559     return _convert_object_array(content, columns, dtype=dtype,
-> 7560                                  coerce_float=coerce_float)
   7561 
   7562 

~/Library/Python/3.6/lib/python/site-packages/pandas/core/frame.py in _convert_object_array(content, columns, coerce_float, dtype)
   7578         return arr
   7579 
-> 7580     arrays = [convert(arr) for arr in content]
   7581 
   7582     return arrays, columns

~/Library/Python/3.6/lib/python/site-packages/pandas/core/frame.py in <listcomp>(.0)
   7578         return arr
   7579 
-> 7580     arrays = [convert(arr) for arr in content]
   7581 
   7582     return arrays, columns

~/Library/Python/3.6/lib/python/site-packages/pandas/core/frame.py in convert(arr)
   7575         if dtype != object and dtype != np.object:
   7576             arr = lib.maybe_convert_objects(arr, try_float=coerce_float)
-> 7577             arr = maybe_cast_to_datetime(arr, dtype)
   7578         return arr
   7579 

~/Library/Python/3.6/lib/python/site-packages/pandas/core/dtypes/cast.py in maybe_cast_to_datetime(value, dtype, errors)
   1086         elif not (is_array and not (issubclass(value.dtype.type, np.integer) or
   1087                                     value.dtype == np.object_)):
-> 1088             value = maybe_infer_to_datetimelike(value)
   1089 
   1090     return value

~/Library/Python/3.6/lib/python/site-packages/pandas/core/dtypes/cast.py in maybe_infer_to_datetimelike(value, convert_dates)
    948         value = try_datetime(v)
    949     elif inferred_type == 'datetime':
--> 950         value = try_datetime(v)
    951     elif inferred_type == 'timedelta':
    952         value = try_timedelta(v)

~/Library/Python/3.6/lib/python/site-packages/pandas/core/dtypes/cast.py in try_datetime(v)
    922                 from pandas import DatetimeIndex
    923 
--> 924                 values, tz = conversion.datetime_to_datetime64(v)
    925                 return DatetimeIndex(values).tz_localize(
    926                     'UTC').tz_convert(tz=tz)

pandas/_libs/tslibs/conversion.pyx in pandas._libs.tslibs.conversion.datetime_to_datetime64()

pandas/_libs/tslibs/conversion.pyx in pandas._libs.tslibs.conversion.convert_datetime_to_tsobject()

AttributeError: 'STIXdatetime' object has no attribute 'nanosecond'

The reason why at the beginning I thought it was STIX library was due to the following error message at the end:

AttributeError: 'STIXdatetime' object has no attribute 'nanosecond'

This is a very specific error in version 0.23.0 so I checked the changes to that specific definitions in pandas:

~/Library/Python/3.6/lib/python/site-packages/pandas/core/dtypes/cast.py in try_datetime(v)
    922                 from pandas import DatetimeIndex
    923 
--> 924                 values, tz = conversion.datetime_to_datetime64(v)
    925                 return DatetimeIndex(values).tz_localize(
    926                     'UTC').tz_convert(tz=tz)

** Pandas Version 0.22.0:**
https://github.com/pandas-dev/pandas/blob/0.22.x/pandas/core/dtypes/cast.py#L879

** Pandas Version 0.23.0:**
https://github.com/pandas-dev/pandas/blob/0.23.x/pandas/core/dtypes/cast.py#L908

So they added the following in version 0.23.0:
https://github.com/pandas-dev/pandas/blob/0.23.x/pandas/core/dtypes/cast.py#L920

# we might have a sequence of the same-datetimes with tz's
            # if so coerce to a DatetimeIndex; if they are not the same,
            # then these stay as object dtype, xref GH19671
            try:
                from pandas._libs.tslibs import conversion
                from pandas import DatetimeIndex

                values, tz = conversion.datetime_to_datetime64(v)
                return DatetimeIndex(values).tz_localize(
                    'UTC').tz_convert(tz=tz)
            except (ValueError, TypeError):
                pass

I checked the STIXdateTime class arguments and I dont see nanoseconds as an option
https://github.com/oasis-open/cti-python-stix2/blob/master/stix2/utils.py#L24

I am not sure if there is anything that needs to be done on the STIX library side.

I downgraded the Python3 Pandas package to 0.22.0 and it worked fine. I didnt want to start an issue in Pandas before asking you guys if this makes sense and if it is possible that nanoseconds needs to be defined as an argument for the STIXdatetime class.

I hope you all have a great weekend! No rush at all on this one. I will keep working with Pandas 0.22.0 for now. I dont need to use pandas to collect or filter the data initially. I use it for a better representation of the results after collecting everything via STIX and TAXII libraries. Therefore, if you want to close this issue since I am using an external library, I would understand. It is just that the STIXdatetime error message caught my attention and I wasnt sure if nanosecond is an standard or anything that needs to be defined on the STIX side. If not, then this issue can be close 😄

Once again guys, great job and thank you for all your help!! I hope you all have a great weekend!!!

[jburns12]

Hey @Cyb3rWard0g...I'm going to pass this along to the folks working on the python-stix2 lib to get their feedback on this. As always, thanks so much for bringing this to our attention!

[Cyb3rWard0g]

Thank you @jburns12 👍 it is probably nothing, but I figured I would let you guys know first just in case 😄 . Np. Thank you guys!

[emmanvg]

Hi @Cyb3rWard0g, it sounds like you might be working with stix2 objects rather than JSON when you reach line 5 in this section of the code. Since I cannot see at which step in your code this happened, my suggestion is to call serialize() on those objects.

<ipython-input-5-e6f5fde53c7b> in <module>()
      3 print(" ")
      4 df = all_attack['techniques']
----> 5 df = json_normalize(df)
      6 df.reindex(['matrix', 'object created','tactic', 'technique', 'technique id', 'data sources'], axis=1)[0:5]

For example, assuming all_attack['techniques'] is a stix2 object you can perform all_attack['techniques'].serialize(). Let me know if that helped!

[Cyb3rWard0g]

Hey @emmanvg Thank you for the suggestion and for taking a look at this issue. When I work with the data from ATT&CK content in STIX, I first parse every field to rename some of the columns and put everything in dictionaries. Those dictionaries then get aggregated in a list. STIX objects do not get passed to the list. I checked every object in techniques with a for loop :

techniques = all_attack['techniques']
for t in techniques:
  print(type(t)

<class 'dict'>
<class 'dict'>
<class 'dict'>
<class 'dict'>
<class 'dict'>
<class 'dict'>
<class 'dict'>
<class 'dict'>
<class 'dict'>
<class 'dict'>
<class 'dict'>
<class 'dict'>
<class 'dict'>
..
....

Now, I dont know how to check if those dictionaries are also considered STIX objects. When I pass serialize() to the objects it does not work because I am now working with lists and dictionaries. I loop through STIX objects and then pass them to dictionaries and then to a long list to manage the fields better and keep it consistent across all objects. I am releasing this first beta version of a script I am working on today. I hope it helps to provide more context to the question. 👍

[emmanvg]

Hi @Cyb3rWard0g, the behavior you are experiencing may be a bug that needs to be addressed in stix2. I noticed that using dict() on a stix2 object will not correctly serialize datetime properties. While we figure this out, you can perform the following workaround (look at the attached image). In this step

rename some of the columns and put everything in dictionaries.

Don't put them in dictionaries and instead serialize them and load the strings back to have a JSON object that you can pass to the pandas library. I hope this helps!

[Cyb3rWard0g]

Hey @emmanvg ! Thank you very much for the information. I will then use that method to convert the STIX objects to Json first. The reason why I rename most of the fields is because when you want to have mappings going matrix -> tactic -> technique -> technique id -> group -> group id -> software -> software id , I cannot use the field "name" or simply "id" since it will create conflicts. I will work on the serialize update. thank you again!!! 👍

[emmanvg]

@Cyb3rWard0g, I took a quick look at your project. Based on your usage basics notebook I noticed what you meant by modifying the pandas matrix and how you manipulate the STIX data (I am not a pandas expert). Though, when looking at your code this method creates new dictionaries based on direct interaction with stix2 objects. Hence why obj['created'] (and any other property that uses STIXdatetime) will return the STIXdatetime object instead. It may be possible that older versions of pandas performed additional work to serialize those datetime objects (since STIXdatetime subclasses datetime.datetime). Note that calling str(obj['created']) produces a string with similar format to the ones in your usage documentation. Ultimately, I will leave the desired workaround up to you, but you may have found a legitimate bug in cti-python-stix2.

[Cyb3rWard0g]

I really appreciate the help and thank you for providing all those details that make perfect sense!!! 👍 I will try the str(obj['created']) then and it seemed that will solve the issue since it will not be returning STIXdatetime object instead. This is why I wanted to share it with you guys first before opening all kinds of issues in Pandas without much information on how cti-python-stix2 was handling those functions. I am glad I could help 👍 Thank you again. I will share some updates of the changes to the script in the next couple of days.

[chisholm]

This is easy to reproduce without python-stix2. If you subclass python's datetime.datetime class, pandas blows up (this is with python 3.6):

import datetime
import pandas as pd

class DTTest(datetime.datetime):
    pass

our_dt = DTTest(2012,1,2)
pd.to_datetime(our_dt)

produces:

Traceback (most recent call last):
  File "...", line 8, in <module>
    pd.to_datetime(our_dt)
  File "...\lib\site-packages\pandas\core\tools\datetimes.py", line 469, in to_datetime
    result = _convert_listlike(np.array([arg]), box, format)[0]
  File "...\lib\site-packages\pandas\core\tools\datetimes.py", line 368, in _convert_listlike
    require_iso8601=require_iso8601
  File "pandas\_libs\tslib.pyx", line 492, in pandas._libs.tslib.array_to_datetime
  File "pandas\_libs\tslib.pyx", line 544, in pandas._libs.tslib.array_to_datetime
AttributeError: 'DTTest' object has no attribute 'nanosecond'

The stacktrace is a little different (I called a different function, to try to simplify), but I bet it's the same underlying issue. There is no documented "nanosecond" field on datetime. Instance attributes are documented here. It works fine with a plain datetime object. Pandas might be doing some voodoo it shouldn't be doing. I'd ask what's going on in a pandas help forum. Give them a simple test like this and ask why pandas is misbehaving.

In the meantime, I guess you need to avoid passing STIXdatetime instances into pandas!

[chisholm]

Actually, this existing pandas issue is probably the cause. Opened only 11 days ago, and is still open.

[Cyb3rWard0g]

Hey @chisholm ! very interesting. so when you say

In the meantime, I guess you need to avoid passing STIXdatetime instances into pandas!

You mean to follow the str(obj['created']) example to any object returning STIXdatetime correct?

[Cyb3rWard0g]

@chisholm thank you very much for providing more details. It is a 0.23.0 version issue since I am not having the same issues with Pandas 0.22.0 in Python2.7 and 3.6. I mentioned that they added this to their Pandas version 0.23.0 https://github.com/pandas-dev/pandas/blob/0.23.x/pandas/core/dtypes/cast.py#L920

# we might have a sequence of the same-datetimes with tz's
            # if so coerce to a DatetimeIndex; if they are not the same,
            # then these stay as object dtype, xref GH19671
            try:
                from pandas._libs.tslibs import conversion
                from pandas import DatetimeIndex

                values, tz = conversion.datetime_to_datetime64(v)
                return DatetimeIndex(values).tz_localize(
                    'UTC').tz_convert(tz=tz)
            except (ValueError, TypeError):
                pass

I tracked the error I had back to that part of the library.

[chisholm]

Hey chisholm ! very interesting. so when you say

In the meantime, I guess you need to avoid passing STIXdatetime instances into pandas!

You mean to follow the str(obj['created']) example to any object returning STIXdatetime correct?

If obj['created'] would be passed into pandas, yeah. Or convert to a datetime.datetime instance (i.e. from STIXdatetime to its superclass). Anything but passing through STIXdatetime's :)

I mentioned that they added this to their Pandas version 0.22.0

Actually, both of our stacktraces track further back than cast.py (or datetimes.py in my case). They track back to a "tslib.pyx" file. I suspect that's where the problem is, and is probably not Python code. My stacktrace references an array_to_datetime function. That is probably this. Looks like a funny combination of C and Python. I'm not familiar with how those languages integrate, but I suspect the actual bug/problem location is there. Oh yeah, my stacktrace specifically references line 544, which is here, and there is the nanosecond reference. From the pandas issue and that code comment in tslib.pyx (# i.e. a Timestamp object), it seems like they are trying to distinguish between a pandas Timestamp instance and a plain datetime.datetime instance, and aren't doing it correctly.

[Cyb3rWard0g]

niceee makes sense. Thank you @chisholm 😄

[Cyb3rWard0g]

Good evening @chisholm . I wanted to provide an update on how I am managing the STIXdatetime object type error. I noticed that even without using Pandas and simply "json.dumps(stix_object)", I get the same error. I pass the STIX object to a dictionary to apply a standard naming convention to all the data I retrieve from TAXII. for example when I run the function from my library, it returns a dictionary with STIX object types.

>>> group_name = lift.get_group_by_alias('Cozy Bear')
>>> group_name
{'created_by_ref': 'identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5', 'group_aliases': [u'APT29', u'The Dukes', u'Cozy Bear', u'CozyDuke'], 'grou_description': u'APT29 is threat group that has been attributed to the Russian government and has operated since at least 2008. (Citation: F-Secure The Dukes) (Citation: GRIZZLY STEPPE JAR) This group reportedly compromised the Democratic National Committee starting in the summer of 2015. (Citation: Crowdstrike DNC June 2016)', 'group_references': None, 'id': u'intrusion-set--899ce53f-13a0-479b-a0e4-67d46e241542', 'group': u'APT29', 'matrix': u'mitre-attack', 'created': '2017-05-31T21:31:52.748Z', 'url': u'https://attack.mitre.org/wiki/Group/G0016', 'modified': '2018-04-18T17:59:24.739Z', 'group_id': u'G0016', 'type': u'intrusion-set'}

I can confirm that the created field is a STIXdatetime:

<class 'stix2.utils.STIXdatetime'>

Then I just simply:

>>> import json
>>> json.dumps(group_name)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: '2017-05-31T21:31:52.748Z' is not JSON serializable

So I was wondering if it is also related to what @emmanvg mentioned earlier in this thread

Hi @Cyb3rWard0g, the behavior you are experiencing may be a bug that needs to be addressed in stix2. I noticed that using dict() on a stix2 object will not correctly serialize datetime properties. While we figure this out, you can perform the following workaround (look at the attached image). In this step

Anyways. I am just using the workaround passing the STIXdatetime object type to a str.

str(object['created'])

when I do that I get the following date and it can be passed as a string instead of STIXdatetime:

"2017-05-31 21:31:52.748000+00:00"

Just sharing some updates on how I am approaching this error 👍 Thank you for all your help an time ! Have a great weekend!

[emmanvg]

@Cyb3rWard0g, you should not call json.dumps(stix_obj) because stix2 objects implement a custom JSONEncoder found here. You should instead use stix_obj.serialize() and it will return a string of the serialized object. You could then turn it back into a dictionary by using json.loads(). Note that datetime properties will be serialized with a format like this one 2017-05-31T21:31:52.748Z. If str(object['created']) works best for you, I'd say continue down that path. 👍

[chisholm]

I pass the STIX object to a dictionary ...

I think you must mean the same bit of code emmanvg linked to earlier in the thread. Yeah, STIXdatetime is not JSON serializable via the default encoder. It's not a bug. Python's built-in encoder doesn't know anything about stix2 types. The types supported by the built-in encoder are listed here. As he notes, people who stick with the plain stix2 objects can call the serialize() method to obtain JSON. If you've got stix2 types embedded in a different data structure and want to serialize it all to JSON, you could try the encoder provided by the library, which he linked to. E.g. json.dumps(group_name, cls=STIXJSONEncoder).

[Cyb3rWard0g]

Thank you so much @emmanvg and @chisholm . It now makes more sense. I appreciate the time you guys took to answer my questions and all the details. I think this can be closed then correct?

[emmanvg]

No problem @Cyb3rWard0g! I think it is OK to close. If you have stix2 library problems in the future I would recommend opening them there to better address it.

[chisholm]

Yep. The main problem in this issue was a pandas bug, not a stix2 bug.

[gtback]

Thanks for the detailed report, @Cyb3rWard0g . And thanks for the helpful follow-ups, @emmanvg and @chisholm . Based on what everyone has said, this is:

• definitely not an issue in the mitre/cti STIX content.
• almost definitely related to a bug in pandas (linked above).
• likely not a bug we need/want to fix in python-stix2. The correct way to convert a stix2 object to a string is with serialize(). If you want to serialize stix2 objects in part of a larger data structure, you can use STIXJSONEncoder as @chisholm suggests, or create your own encoder class that handles datetimes, STIX objects, and anything else that's needed.

There's not currently a way to convert a stix2 object to a "clean" Python dict with only built-in data types, other than calling serialize() and then json.loads()-ing the result. We could add something, but I don't know if the complexity of doing that is worth the minor performance benefits. In any event, that's not an issue in this repo, so I'm going to close this, but feel free to open another issue in cti-python-stix2 to discuss 😀

Thanks again!

[Cyb3rWard0g]

Thank you so much for all the detailed information and for teaching me the right way to do things. I learned a lot in this thread issue. Thank you again. and any future issue will be open in the STIX repo. 👍 I hope you all have a great weekend!!!

[Cyb3rWard0g]

Thank you @emmanvg @chisholm @gtback 👍

[Morikko]

There's not currently a way to convert a stix2 object to a "clean" Python dict with only built-in data types, other than calling serialize() and then json.loads()-ing the result

Just for confirmation as I found nothing in the doc. Is it still the recommended solution today if we need data as native dict ?

[isaisabel]

As far as I know yes that's the best way to do it. You may get more info if you ask on the cti-python-stix2 issue tracker though since the team over there would know more about the capabilities of their library.

FWIW if you're trying to use ATT&CK with Pandas in 2021, we have an official way of doing that: https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/attackToExcel#accessing-the-pandas-dataframes