[cybereason] Initial Release for Cybereason #9595
Conversation
muskan-agarwal26
changed the title
jamiehynds
added
the
Team:Security-Service Integrations
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
| post_request( | ||
| state.url + "/login.html", | ||
| "application/x-www-form-urlencoded", | ||
| "username="+ state.username.replace("@", "%40", -1) + "&password=" + state.password.replace("@", "%40", -1) |
There was a problem hiding this comment.
| "username="+ state.username.replace("@", "%40", -1) + "&password=" + state.password.replace("@", "%40", -1) | |
| "username":[state.username],"password":[state.password]}.format_query() |
| ( | ||
| [{}] | ||
| ) |
There was a problem hiding this comment.
| ( | |
| [{}] | |
| ) | |
| [{}] |
(similar throughout)
| resp.Request.Response.Request.Response.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID")).size() > 0 ? | ||
| (resp.Request.Response.Request.Response.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID"))[0].split(";").filter(e, e.contains("JSESSIONID")).size() > 0 ? | ||
| resp.Request.Response.Request.Response.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID"))[0].split(";").filter(e, e.contains("JSESSIONID"))[0] |
There was a problem hiding this comment.
What is the intention here? This looks to me like it is trying to
resp.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID")).as(sess, sess.size() > 0 ?
sess[0].split(";").as(sub, sub.filter(e, e.contains("JSESSIONID")).size() > 0 ?
sub[0]
:
""
)
but it is extremely unclear, so I am not sure.
Note that resp.Request.Response.Request.Response = resp since the response hold a copy of the request and vice versa.
There was a problem hiding this comment.
@kcreddy Has explained what this field walk is for. This makes sense, though it looks like there are two steps too many. The reason should be explained in a comment in the code.
There was a problem hiding this comment.
Please add the comments explaining this (and in the other cases where this is done).
| )).as(body, { | ||
| "cookie": body, | ||
| }) | ||
| ).as(cookie_obj, ( |
There was a problem hiding this comment.
I don't understand why you are doing this; you have state decorated to include the request object, but it's called cookie_obj and you continue to use the original state object. What is the intention? I don't think that you need to do the initial state.with since AFAICS you are just POSTing to get the sess cookie, so ISTM that this whole preamble should be something like:
post_request(
state.url + "/login.html",
"application/x-www-form-urlencoded",
"username":[state.username],"password":[state.password]}.format_query()
).do_request().as(resp, resp.Request.URL.contains("error") ? "" :
resp.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID")).as(sess, sess.size() > 0 ?
sess[0].split(";").as(sub, sub.filter(e, e.contains("JSESSIONID")).size() > 0 ?
sub[0]
:
""
)
:
""
)
| fields: | ||
| - username | ||
| - password | ||
| program: | |
There was a problem hiding this comment.
I think probably this should look something like this, though there are still outstanding questions (see in code comments)
program: |
post_request(
state.url + "/login.html",
"application/x-www-form-urlencoded",
"username":[state.username],"password":[state.password]}.format_query()
).do_request().as(resp, resp.Request.URL.contains("error") ? "" :
resp.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID")).as(sess, sess.size() > 0 ?
sess[0].split(";").as(sub, sub.filter(e, e.contains("JSESSIONID")).size() > 0 ?
sub[0]
:
""
)
:
""
)
).as(cookie, cookie == "" ? {"events": []} :
post_request(
state.url + "/rest/visualsearch/query/simple",
"application/json",
{
"queryPath": [
{
"requestedType": "LogonSession",
"filters": [
{
"facetName": "creationTime",
"filterType": "Between",
"values": [
(
state.?cursor.last_timestamp.orValue(null) != null ?
state.cursor.last_timestamp
:
int(now - duration(state.initial_interval)) * 1000
),
int(now) * 1000,
]
}
],
"isResult": "true"
}
],
"totalResultLimit": state.batch,
"perGroupLimit": 100,
"perFeatureLimit": 100,
"templateContext": "SPECIFIC",
"queryTimeout": 120000,
"customFields": [
"processes",
"ownerMachine",
"user",
"remoteMachine",
"logonType",
"creationTime",
"endTime",
"elementDisplayName",
]
}.encode_json(),
).with({
"Header": {
"Cookie": [cookie],
"Content-Type": ["application/json"],
},
}).do_request().as(resp,
resp.StatusCode == 200 ?
bytes(resp.Body).decode_json().as(body, {
"events": !has(body.?data.resultIdToElementDataMap) ? [{}] : // Why are we returning a single empty event in this case?
(
body.data.resultIdToElementDataMap.size() == 0 ? [{}] : // Why are we returning a single empty event in this case?
(
body.data.resultIdToElementDataMap.map(e, {
"message": [
body.data.resultIdToElementDataMap[e].encode_json(),
body.data.suspicionsMap.encode_json(),
body.data.evidenceMap.encode_json(),
],
})
)
)
"cursor": {
"last_timestamp": int(now) * 1000,
},
"username": state.username,
"password": state.password,
"initial_interval": state.initial_interval,
"batch": state.batch,
})
:
{
"events": [{
"error": {
"code": string(resp.StatusCode),
"id": string(resp.Status),
"message": string(resp.Body),
}
}],
"username": state.username,
"password": state.password,
"initial_interval": state.initial_interval,
"batch": state.batch,
}
)
)
General comments:
- avoid re-evals by binding results so that they can be reused
- please ensure that indentation is correctly used; opening an closing parens, braces and brackets should be on the same column as the first non-whitespace character of the opening, and the column depth increased by the indent width for each line that increases the depth of the syntax
- please use commas on all array and object lines (this simplifies code mutation later and we don't have JSON rules)
- avoid unnecessary parens; only use these when the compiler cannot disambiguate from the existing syntax
The programs below look similar and will need equivalent treatment.
There was a problem hiding this comment.
Reply to the questions in the code comments:
As every input requires a corresponding output, in the case of null data in the response, we return empty events.
There was a problem hiding this comment.
As every input requires a corresponding output, in the case of null data in the response, we return empty events.
What do you mean by this? If you mean that an expression evaluates to a value and the value has to go somewhere and if there is a place to go there must be a value, this is not quite true since we have optional types now. If you eval an optional type, either by a .? operator, or by an explicit pair of optional.of(x)/opional.none(), then you can define the field that it's going into with ? prefix, {?"field": state.?maybe_exists} will eval to either {} or {"field":<whatever was in state.maybe_exists>} depending on whether state has a field maybe_exists.
1. Added comments in cel code in malware datastream regarding the cookie. 2. Resolved the general comments
|
|
||
| ## Compatibility | ||
|
|
||
| This module has been tested against the latest Cybereason On-Prem version **23.2**. |
There was a problem hiding this comment.
| This module has been tested against the latest Cybereason On-Prem version **23.2**. | |
| This module has been tested against the latest Cybereason On-Premises version **23.2**. |
packages/cybereason/data_stream/logon_session/agent/stream/cel.yml.hbs
Outdated
Show resolved
Hide resolved
packages/cybereason/data_stream/logon_session/agent/stream/cel.yml.hbs
Outdated
Show resolved
Hide resolved
| { | ||
| "startTime": | ||
| ( | ||
| state.?cursor.last_timestamp.orValue(null) != null ? |
There was a problem hiding this comment.
Is cursor.last_timestamp ever set to null? I can't see this happen anywhere.
packages/cybereason/data_stream/poll_malop/agent/stream/cel.yml.hbs
Outdated
Show resolved
Hide resolved
packages/cybereason/data_stream/suspicions_process/agent/stream/cel.yml.hbs
Outdated
Show resolved
Hide resolved
packages/cybereason/data_stream/suspicions_process/agent/stream/cel.yml.hbs
Outdated
Show resolved
Hide resolved
packages/cybereason/data_stream/suspicions_process/agent/stream/cel.yml.hbs
Outdated
Show resolved
Hide resolved
1. Suggested changes in readme. 2. Made suggested changes in the cel code.
|
/test |
| resp.Request.Response.Request.Response.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID")).as(sess, sess.size() > 0 ? | ||
| sess[0].split(";").as(sub, sub.filter(e, e.contains("JSESSIONID")).size() > 0 ? | ||
| sub[0] | ||
| : | ||
| "" | ||
| ) | ||
| : | ||
| "" | ||
| ) |
There was a problem hiding this comment.
This in incorrectly indented; inconsistent with the other code, which is at 2.
| "" | ||
| ) | ||
| ).as(cookie, cookie == "" ? {"events": []} : | ||
| post_request( |
| "application/x-www-form-urlencoded", | ||
| {"username":[state.username],"password":[state.password]}.format_query() | ||
| ).do_request().as(resp, resp.Request.URL.contains("error") ? "" : | ||
| resp.Request.Response.Request.Response.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID")).as(sess, sess.size() > 0 ? |
| "application/x-www-form-urlencoded", | ||
| {"username":[state.username],"password":[state.password]}.format_query() | ||
| ).do_request().as(resp, resp.Request.URL.contains("error") ? "" : | ||
| resp.Request.Response.Request.Response.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID")).as(sess, sess.size() > 0 ? |
| - username | ||
| - password | ||
| program: | | ||
| ( |
There was a problem hiding this comment.
Please don't wrap the complete program in parens; it's not necessary.
| (!state.?want_more.orValue(false) | ||
| ? |
There was a problem hiding this comment.
| (!state.?want_more.orValue(false) | |
| ? | |
| !state.?want_more.orValue(false) ? |
along with associated remove of its partner.
| : | ||
| "" | ||
| ).as(cookie, cookie == "" && !has(state.cookie) ? {"events": []} : | ||
| post_request( |
| resp.Request.Response.Request.Response.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID")).size() > 0 ? | ||
| (resp.Request.Response.Request.Response.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID"))[0].split(";").filter(e, e.contains("JSESSIONID")).size() > 0 ? | ||
| resp.Request.Response.Request.Response.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID"))[0].split(";").filter(e, e.contains("JSESSIONID"))[0] |
There was a problem hiding this comment.
Please add the comments explaining this (and in the other cases where this is done).
| fields: | ||
| - username | ||
| - password | ||
| program: | |
There was a problem hiding this comment.
As every input requires a corresponding output, in the case of null data in the response, we return empty events.
What do you mean by this? If you mean that an expression evaluates to a value and the value has to go somewhere and if there is a place to go there must be a value, this is not quite true since we have optional types now. If you eval an optional type, either by a .? operator, or by an explicit pair of optional.of(x)/opional.none(), then you can define the field that it's going into with ? prefix, {?"field": state.?maybe_exists} will eval to either {} or {"field":<whatever was in state.maybe_exists>} depending on whether state has a field maybe_exists.
|
/test |
1. Fixed indentation in cel code files. 2. Removed unnecessary line of code in construction of events.
|
/test |
| "events": !has(body.?data.resultIdToElementDataMap) ? [{}] : | ||
| body.data.resultIdToElementDataMap.map(e, { | ||
| "message": [ | ||
| body.data.resultIdToElementDataMap[e].encode_json(), | ||
| body.data.suspicionsMap.encode_json(), | ||
| body.data.evidenceMap.encode_json(), | ||
| ], | ||
| }), | ||
| "cursor": { | ||
| "last_timestamp": ( | ||
| has(body.?data.resultIdToElementDataMap) && body.data.resultIdToElementDataMap.size() > 0 ? | ||
| int(body.data.resultIdToElementDataMap.map(e, int(body.data.resultIdToElementDataMap[e].simpleValues.malopLastUpdateTime.values[0])).max()) | ||
| : | ||
| state.?cursor.last_timestamp.orValue(int(now - duration(state.initial_interval)) * 1000) | ||
| ) | ||
| }, | ||
| "username": state.username, | ||
| "password": state.password, | ||
| "initial_interval": state.initial_interval, | ||
| "batch": state.batch, |
| { | ||
| "queryPath": [ | ||
| ).as(cookie, cookie == "" ? {"events": []} : | ||
| post_request( |
There was a problem hiding this comment.
This whole block is indented 4, but the rest of the file is indented 2. This is the case for the block under do_request as well.
| { | ||
| "queryPath": [ | ||
| ).as(cookie, cookie == "" ? {"events": []} : | ||
| post_request( |
| "Content-Type": ["application/json"], | ||
| }, | ||
| }).do_request().as(resp, | ||
| resp.StatusCode == 200 ? |
| "events": !has(body.?data.resultIdToElementDataMap) ? [{}] : | ||
| body.data.resultIdToElementDataMap.map(e, { | ||
| "message": [ | ||
| body.data.resultIdToElementDataMap[e].encode_json(), | ||
| body.data.suspicionsMap.encode_json(), | ||
| body.data.evidenceMap.encode_json(), | ||
| ], | ||
| }), | ||
| "cursor": { | ||
| "last_timestamp": ( | ||
| has(body.?data.resultIdToElementDataMap) && body.data.resultIdToElementDataMap.size() > 0 ? | ||
| int(now) * 1000 | ||
| : | ||
| state.?cursor.last_timestamp.orValue(int(now) * 1000) | ||
| ), | ||
| }, | ||
| "username": state.username, | ||
| "password": state.password, | ||
| "initial_interval": state.initial_interval, | ||
| "batch": state.batch, |
| "events": !has(body.?data.resultIdToElementDataMap) ? [{}] : | ||
| body.data.resultIdToElementDataMap.map(e, { | ||
| "message": [ | ||
| body.data.resultIdToElementDataMap[e].encode_json(), | ||
| body.data.suspicionsMap.encode_json(), | ||
| body.data.evidenceMap.encode_json(), | ||
| ], | ||
| }), | ||
| "cursor": { | ||
| "last_timestamp": ( | ||
| has(body.?data.resultIdToElementDataMap) && body.data.resultIdToElementDataMap.size() > 0 ? | ||
| int(now) * 1000 | ||
| : | ||
| state.?cursor.last_timestamp.orValue(int(now) * 1000) | ||
| ), | ||
| }, | ||
| "username": state.username, | ||
| "password": state.password, | ||
| "initial_interval": state.initial_interval, | ||
| "batch": state.batch, |
| "application/x-www-form-urlencoded", | ||
| {"username":[state.username],"password":[state.password]}.format_query() | ||
| ).do_request().as(resp, resp.Request.URL.contains("error") ? "" : // When the base request redirects to a new request, | ||
| resp.Request.Response.Request.Response.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID")).as(sess, sess.size() > 0 ? // we obtain the request object for the initial request's response. |
| "application/x-www-form-urlencoded", | ||
| {"username":[state.username],"password":[state.password]}.format_query() | ||
| ).do_request().as(resp, resp.Request.URL.contains("error") ? "" : // When the base request redirects to a new request, | ||
| resp.Request.Response.Request.Response.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID")).as(sess, sess.size() > 0 ? // we obtain the request object for the initial request's response. |
| (!state.?want_more.orValue(false) ? | ||
| post_request( | ||
| state.url.trim_right("/") + "/login.html", | ||
| "application/x-www-form-urlencoded", | ||
| {"username":[state.username],"password":[state.password]}.format_query() | ||
| ).do_request().as(resp, resp.Request.URL.contains("error") ? "" : // When the base request redirects to a new request, | ||
| resp.Request.Response.Request.Response.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID")).as(sess, sess.size() > 0 ? // we obtain the request object for the initial request's response. | ||
| sess[0].split(";").as(sub, sub.filter(e, e.contains("JSESSIONID")).size() > 0 ? // To retrieve the appropriate header based on the redirect, | ||
| sub[0] // we've implemented the following method. | ||
| : | ||
| "" | ||
| ) | ||
| : | ||
| "" | ||
| ) | ||
| ) | ||
| : | ||
| "" |
There was a problem hiding this comment.
| (!state.?want_more.orValue(false) ? | |
| post_request( | |
| state.url.trim_right("/") + "/login.html", | |
| "application/x-www-form-urlencoded", | |
| {"username":[state.username],"password":[state.password]}.format_query() | |
| ).do_request().as(resp, resp.Request.URL.contains("error") ? "" : // When the base request redirects to a new request, | |
| resp.Request.Response.Request.Response.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID")).as(sess, sess.size() > 0 ? // we obtain the request object for the initial request's response. | |
| sess[0].split(";").as(sub, sub.filter(e, e.contains("JSESSIONID")).size() > 0 ? // To retrieve the appropriate header based on the redirect, | |
| sub[0] // we've implemented the following method. | |
| : | |
| "" | |
| ) | |
| : | |
| "" | |
| ) | |
| ) | |
| : | |
| "" | |
| ( | |
| state.?want_more.orValue(false) ? "" : post_request( | |
| state.url.trim_right("/") + "/login.html", | |
| "application/x-www-form-urlencoded", | |
| {"username":[state.username],"password":[state.password]}.format_query() | |
| ).do_request().as(resp, resp.Request.URL.contains("error") ? "" : // When the base request redirects to a new request, | |
| resp.Request.Response.Request.Response.Header["Set-Cookie"].filter(e, e.contains("JSESSIONID")).as(sess, sess.size() > 0 ? // we obtain the request object for the initial request's response. | |
| sess[0].split(";").as(sub, sub.filter(e, e.contains("JSESSIONID")).size() > 0 ? // To retrieve the appropriate header based on the redirect, | |
| sub[0] // we've implemented the following method. | |
| : | |
| "" | |
| ) | |
| : | |
| "" | |
| ) | |
| ) |
| "events": has(body.?data.malwares) ? body.data.malwares.map(e, { "message": e.encode_json() }) : [{}], | ||
| "want_more": body.?data.hasMoreResults.orValue(false), | ||
| "batch": state.batch, | ||
| "initial_interval": state.initial_interval, | ||
| "page_no": body.?data.hasMoreResults.orValue(false) ? int(state.page_no) + 1 : 0, | ||
| "username": state.username, | ||
| "password": state.password, | ||
| "cookie": state.?cookie.orValue(cookie), | ||
| "cursor": { | ||
| "last_timestamp": ( | ||
| has(body.?data.malwares) && body.data.malwares.size() > 0 ? | ||
| ( | ||
| has(state.?cursor.last_timestamp) && body.data.malwares.map(e, e.timestamp).max() < state.cursor.last_timestamp ? | ||
| state.cursor.last_timestamp | ||
| : | ||
| int(body.data.malwares.map(e, e.timestamp).max()) | ||
| ) | ||
| : | ||
| state.?cursor.last_timestamp.orValue(int(now - duration(state.initial_interval)) * 1000) | ||
| ), | ||
| "first_timestamp": ( | ||
| has(state.?cursor.first_timestamp) && has(body.?data.malwares) ? | ||
| ( | ||
| body.?data.hasMoreResults.orValue(false) ? | ||
| state.cursor.first_timestamp | ||
| : | ||
| state.cursor.last_timestamp | ||
| ) | ||
| : | ||
| int(now - duration(state.initial_interval)) * 1000 | ||
| ), | ||
| "less_than_timestamp": ( | ||
| has(state.?cursor.less_than_timestamp) && has(body.?data.malwares) ? | ||
| ( | ||
| body.?data.hasMoreResults.orValue(false) ? | ||
| state.cursor.less_than_timestamp | ||
| : | ||
| int(now) * 1000 | ||
| ) | ||
| : | ||
| int(now) * 1000 | ||
| ), | ||
| } |
🚀 Benchmarks reportTo see the full report comment with |
1. Indentation corrected in all cel files.
|
/test |
💚 Build Succeeded
History
|
|
|
Package cybereason - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=cybereason |
|
IMO Also, it would desirable to use ECS where possible. You may want to take a look at STIX-shifter for a more compact and semantic translation of the data model: https://github.com/opencybersecurityalliance/stix-shifter/tree/develop/stix_shifter_modules/cybereason |
What does this PR do?
Integration release checklist
This checklist is intended for integrations maintainers to ensure consistency when creating or updating a Package, Module or Dataset for an Integration.
All changes
New Package
Log dataset changes
How to test this PR locally
elastic-package test -v
Automated Test
test-log-cybereason.log
Screenshots