adding an attribtue to an event changes timestamps of all attributes, adding shadow attributes crahses

[govcert-ch]

Whenever an attribute is added to an event, PyMisp obviously exports the full event data and uploads the full data again, which causes large queries for only small changed. More importantly, it seems that all timestamps of all attributes in the event are changed to now, which basically destroys the whole timeline history. I'm not sure if this is a bug of PyMisp itself, or a problem of the REST API, where I don't see a possibility to add an attribute (only shadow attributes) to an event.

also, adding shadow attributes via PyMisp (proposal=True) always crashes, it seems teh attributes object is not serializable.

[Rafiot]

Yeah, proposals in the API weren't used much and are very broken, I started to work on it, but it isn't there...

I'm also working on a commit to allow pushing only one attribute instead of the whole event, but again, it isn't ready right now. And regarding the fact all the timestamps are updated to now even if the attribute isn't changed is weird. @iglocska is it expected?

[govcert-ch]

Thanks... it indeed is the case that the history (timestamps) are overwritten - teh large datatransfers are ugly, but I could live with it; the overwritten timestamps are a no-go for us. My current solution is to use the REST API directly to add a shadow attribute, and then to accept the proposal in a second step. This works, but of course it bypasses PyMisp. Also, I did not yet figure out how to add a (non-sample) attachment this way, just adding "data=" to the json object did not work (maybe it requires a uuid=... as well, it complains about wrong event). I will continue to try it out.

[iglocska]

Just saw this, as for the attributes being updated - yeah, an edit where the timestamp is omitted counts as a valid edit that fast forwards the timestamp. This is something we abuse for example when tagging attributes.

[govcert-ch]

Not sure if I understood this right - are you suggesting I should set explicitely a timestamp in my add_named_attribute call (can I do that at all?), or is it the PyMisp code itself that forgot to set the old timestamps in the newly constructed attributes? So is this something that will be fixed on your side?

[govcert-ch]

PS: About the other thing I mentioned... adding proposed attribtues fails... I suspect the line

response = session.post(url, data=json.dumps(query))

in __query_proposal should actually read

response = session.post(url, data=json.dumps(query, cls=EncodeUpdate))

if I understood the logic right. Otherwise, it crashes with a "TypeError: <pymisp.mispevent.MISPAttribute object at 0x10d9ff190> is not JSON serializable". But this is just speculation from my side (and not important for me right now, just wanted to mention it).

[iglocska]

I mean from the MISP core side it's expected behaviour, so I guess we need to make a small change to PyMISP.

Basically what happens: MISP will update any attributes explicitly included in an event edit that meet the timestamp criteria (timestamp > old timestamp or timestamp not set). AFAIK Raphael has reworked the way you add attributes to an event via PyMISP and it now doesn't fetch the event, make changes and push it back but instead allows you to simply add singular attributes (which should help with performance and remove any weird edit artifacts like the one you're encounterig).

[Rafiot]

Adding proposals is being reworked and isn't working properly now.

Since yesterday evening, you can add attributes to an existing event by passing the event ID (and not the full event): https://github.com/MISP/PyMISP/blob/master/pymisp/api.py#L384

This will work with all the methods relying on add_named_attribute and won't update the timestamp of all the attributes any more.

[govcert-ch]

Thanks.. if I get it right, I just need to pass the event ID as number instead of object? Somehow it still does not work, but maybe I'm doing something wrong?

>>> misp = pymisp.PyMISP('https://....', '....', True, 'json')
>>> evt = misp.get_event(529)
>>> evt
c7m6fx6f1tlzITywcLZ ...
>>> a=misp.add_named_attribute(529, "text", "test", category="External analysis", comment="comment")
>>> a
<Response [404]>
>>> a.text
u'{"name":"Could not save the attribute. ","message":"Could not save the attribute. ","url":"\\/attributes\\/add\\/529"}'
>>> b=misp.add_attachment(529, "/.../x.dat", category='External analysis', comment="comment")
>>> b
<Response [404]>
>>> b.text
u'{"name":"Could not save the attribute. ","message":"Could not save the attribute. ","url":"\\/attributes\\/add\\/529"}'
>>> a=misp.add_named_attribute(evt, "text", "test", category="External analysis", comment="comment")
>>> a
a
{u'Event': {u'...

The last one does add the attribute, but of course again with changed timestamps.

I'm a bit surprised about the "/attributes/add/[event_id]" URL - in the REST API documentation, I only see "/shadow_attributes/add/[event_id]" (https://circl.lu/doc/misp/automation/index.html).

I already previoiusly did wiretap a GUI session and captured this URI, but with POST parameters like

    '_method': "POST",
    'data[_Token][key]': '6e....',
    'data[_Token][fields]': '7c....',
    'data[_Token][unlocked]': '',
    'data[Attribute][event_id]' : 529,
    'data[Attribute][category]' : "External analysis" ,
    'data[Attribute][type]' :  "text",
    'data[Attribute][distribution]' : 5 ,
    'data[Attribute][value]' : "test" ,
    'data[Attribute][comment]' : "comment" ,
    'data[Attribute][to_ids]' : 0 ,
    'data[Attribute][batch_import]' : 0 

This does not look like REST API (it's not JSON, just standard POST parameters), but some internal, undocumented API only the GUI is using (that's my guess - is this correct, or is there any way to translate these requests to a correct REST API request?). I already previously tried using such a request directly, with the Authorization header and without the "_Token" parts, but nothing I tried worked. It seems your code tried to send a

{"category": "External analysis", 
 "comment": "comment", 
 "to_ids": false, 
 "value": "test", 
 "distribution": "5", 
 "type": "text", 
 "disable_correlation": false
}

This returns above error message. Of course it might be the MISP itself needs an update (I don't have direct access to it)? Currently I just upgraded my local PyMisp. The MISP itself is on version 2.4.65.

EDIT: We updated the MISP to 2.4.69, and now it works.. thanks ;-) Still interested how and if the GUI POST parameters can be translated to REST API.

[Rafiot]

I just did a new commit in PyMISP where you can also do add a new attribute as proposal and it works the same way as adding an attribute (it isn't tagged as new version for now). It will probably make your life easier do to it.

If I got it right I can close this event then, please reopen/create a new one if you have any other remark.