Questions re' 2.4.120 first_seen last_seen PyMISP behavior

[github-germ]

A few questions regarding the excellent addition of first_seen and last_seen into the MISP attribute model with 2.4.120 with respect to PyMISP.

1. All pre-existing attributes have those two values set to null. Might it make sense that the 2.4.120 database update to the new schema also set the last_seen column with the existing attributes.timestamp? Just checking before we write our own script to do this to meet our needs.

2. Can PyMISP, when ingesting an attribute that already exists in an event, with last_seen set to a date greater than the prior last_seen, update last_seen? My tests show it not being updated and a trigger inserting a validation error into the logs table stating A similar attribute already exists for this event.

Questions 3 and 4 are considered in order to support existing PyMISP apps that do not set these two new *_seen arguments (i.e. backwards support). I understand these are arguable as ingesting now doesn't necessarily mean now is first_seen or last_seen; hence, prior code may need to be enhanced. Wanted to ask the questions before going back to modify prior code.

3. Any possibility to offer a way when ingesting an attribute that already exists in an event that the last_seen would be updated, even if not set in the API call (and also not inserting a validation error into the logs table)? (related to question 2 above)

4. Same when adding a new attribute: set first_seen even if that value is not passed to the API function?

[Rafiot]

It is not implemented nor tested in PyMISP yet, I'll work on it asap, probably within the next ~10 days.

[github-germ]

It is not implemented nor tested in PyMISP yet, I'll work on it asap, probably within the next ~10 days.

That's real good news. Thank you!

[Rafiot]

So, first of all, first_seen and last_seen are now supported by PyMISP (in the master branch, it will be in the v2.4.121 package).

A few questions regarding the excellent addition of first_seen and last_seen into the MISP attribute model with 2.4.120 with respect to PyMISP.

1. All pre-existing attributes have those two values set to `null`. Might it make sense that the 2.4.120 database update to the new schema also set the `last_seen` column with the existing `attributes.timestamp`? Just checking before we write our own script to do this to meet our needs.

No it won't be done automatically, because it doesn't makes sense for most of the use cases, so it won't be the default. Feel free to write your own script.
If you want to share it publicly, feel free to do so, and let us know.

2. Can PyMISP, when ingesting an attribute that already exists in an event, with `last_seen` set to a date greater than the prior `last_seen`, update `last_seen`? My tests show it not being updated and a trigger inserting a validation error into the `logs` table stating `A similar attribute already exists for this event`.

That sounds like a bug, I'll give it a try and figure out what's going on.

The following test case works. Can you share your tests, please? a8ff8f8

Questions 3 and 4 are considered in order to support existing PyMISP apps that do not set these two new *_seen arguments (i.e. backwards support). I understand these are arguable as ingesting now doesn't necessarily mean now is first_seen or last_seen; hence, prior code may need to be enhanced. Wanted to ask the questions before going back to modify prior code.

1. Any possibility to offer a way when ingesting an attribute that already exists in an event that the `last_seen` would be updated, _even if not set in the API call_ (and also not inserting a validation error into the `logs` table)? (related to question 2 above)

Not sure that I understand what you're asking there: What does even if not set in the API call mean?
The only reasonable way to update an event/object/attribute is to call the relevant update methods.

2. Same when adding a new attribute: set `first_seen` _even if that value is not passed to the API function_?

No, first seen will only be set if the value is given bu the user, we will never auto-fill the value without the user encoding it, it would be way too confusing otherwise.

[packet-rat]

+1. Note that an *earlier* First_seen assertion from a given source should also replace same on ingestion of a duplicate for that source. Patrick Maroney

On Jan 22, 2020, at 2:50 PM, github-germ ***@***.***> wrote:

 A few questions regarding the excellent addition of first_seen and last_seen into the MISP attribute model with 2.4.120 with respect to PyMISP. All pre-existing attributes have those two values set to null. Might it make sense that the 2.4.120 database update to the new schema also set the last_seen column with the existing attributes.timestamp? Just checking before we write our own script to do this to meet our needs. Can PyMISP, when ingesting an attribute that already exists in an event, with last_seen set to a date greater than the prior last_seen, update last_seen? My tests show it not being updated and a trigger inserting a validation error into the logs table stating A similar attribute already exists for this event. Questions 3 and 4 are considered in order to support existing PyMISP apps that do not set these two new *_seen arguments (i.e. backwards support). I understand these are arguable as ingesting now doesn't necessarily mean now is first_seen or last_seen; hence, prior code may need to be enhanced. Wanted to ask the questions before going back to modify prior code. Any possibility to offer a way when ingesting an attribute that already exists in an event that the last_seen would be updated, even if not set in the API call (and also not inserting a validation error into the logs table)? (related to question 2 above) Same when adding a new attribute: set first_seen even if that value is not passed to the API function? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.