-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Acceptance tests #166
Comments
WRT the zen cheeseburger, these are the CDM statements that are missing for each TA1 (from the engagement data)
Obviously no one has implemented anything about Also, it appears there are a bunch more sources for which we have no samples. What should the default be for those? Expect statements of every type? |
Couple of points here…
Yes, not many folks had any memory objects in E1. Values were similarly unused, as their only application is for representing arguments to functions. I don’t think anyone was really equipped to do that. It doesn’t surprise me that FAROS is missing RegstryKeyObjects, esp. since they’re not monitoring Windows systems…
For CDM13, I think only 6 of the source values were used…1 for each TA1 provider. The others in your list appear to be sub-classes of those providers. I wonder whether that list is aspirational (if it came from the CDM13 spec), or whether it’s the CDM14 enum set (which is probably also aspirational).
Thanks!
/d
… On Jan 13, 2017, at 3:57 PM, Alec Theriault ***@***.***> wrote:
WRT the zen cheeseburger, these are the CDM statements that are missing for each TA1 (from the engagement data)
SOURCE_FREEBSD_DTRACE_CADETS: MemoryObject, ProvenanceTagNode, RegistryKeyObject, SrcSinkObject, TagEntity, Value
SOURCE_ANDROID_JAVA_CLEARSCOPE: MemoryObject, RegistryKeyObject, TagEntity, Value
SOURCE_WINDOWS_DIFT_FAROS: MemoryObject, RegistryKeyObject, TagEntity, Value
SOURCE_WINDOWS_FIVEDIRECTIONS: MemoryObject, TagEntity, Value
SOURCE_LINUX_THEIA: Value
SOURCE_LINUX_AUDIT_TRACE: Value
Obviously no one has implemented anything about MemoryObject or Value, so it seems a bit foolish to have tests for that. That said, I'm a bit surprised SOURCE_WINDOWS_DIFT_FAROS doesn't have RegistryKeyObject. Should we include a test for this anyways?
Also, it appears there are a bunch more sources <https://github.com/GaloisInc/adapt/blob/b78c202857bc873ccbbd1763060c2f6a4d9f7959/AdaptJVM/src/main/scala/com/galois/adapt/cdm13/CDM13.scala#L31-L43> for which we have no samples. What should the default be for those? Expect statements of every type?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub <#166 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJN5v4KwWx-6d1LFbhILBhJwHEBJcZNNks5rSA9rgaJpZM4LiZ8H>.
|
For now, I think we should immediately fail when detecting a source for which we haven't planned tests. I think Dave is right that CDM13 started off being a bit more aspirational. But I don't think most teams are going to add that in CDM 14. So let's just make an early test that fails (and the remainder abort?) unless the source data claims to be one of the types we plan to see. |
Agreed!
David W. Archer, PhD
Principal Investigator
Galois, Inc.
421 SW 6th Avenue, Suite 300
Portland, OR 97204
email: dwa@galois.com
mobile: 503-701-0235
… On Jan 14, 2017, at 2:42 PM, Ryan Wright ***@***.***> wrote:
For now, I think we should immediately fail when detecting a source for which we haven't planned tests. I think Dave is right that CDM13 started off being a bit more aspirational. But I don't think most teams are going to add that in CDM 14. So let's just make an early test that fails (and the remainder abort?) unless the source data claims to be one of the types we plan to see.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub <#166 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJN5v9bwTCDoYW5Pcn3pPZS0DKlc1LcTks5rSU8_gaJpZM4LiZ8H>.
|
Consolidating from email, here are more (syntactic) tests to do eventually
|
@harpocrates We need to add a test to ensure UUID uniqueness (ironically)—to ensure that only one node ever claims to have any given UUID. There is talk by TA1s about updating data by issuing a new event that reuses an old UUID (and so it would replace earlier data). We need to shut that down every way possible. So let's encode it in a test ASAP. |
PS - @harpocrates after that's done, you should close this issue. :-) |
Fixed (not super efficiently) with 1a25b8c. |
@harpocrates did you deploy these tests yet? I don't see them being run in the version I just ran a moment ago. |
We will communicate with TA1 providers our expectations about their data via acceptance tests (basically unit tests to enforce syntactic/semantic validity, with respect to our understanding, of their data). These are to be delivered using the adapt-tester, and should be constantly refined until April.
This issue will track progress on this front and provide a place for keeping note of things suggested.
So far, we have come up with the following ideas of things to test as a bare minimum:
Later on, once CDM14 is out, we will be able to leverage "epoch" also.
The text was updated successfully, but these errors were encountered: