Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

securityonion-elastic: update dns domain info for elasticsearch ingest #1667

Closed
dougburks opened this issue Nov 26, 2019 · 8 comments
Closed

securityonion-elastic: update dns domain info for elasticsearch ingest #1667

dougburks opened this issue Nov 26, 2019 · 8 comments
Assignees
Projects

Comments

@dougburks
Copy link
Contributor

@dougburks dougburks commented Nov 26, 2019

No description provided.

@dougburks dougburks added this to To do in 16.04.6.4 via automation Nov 26, 2019
@dougburks dougburks changed the title securityonion-elastic: update dns domain info securityonion-elastic: update dns domain info for elasticsearch ingest Nov 26, 2019
@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Nov 26, 2019

Full logstash config includes fields like:

  • highest_registered_domain
  • subdomain
  • subdomain_length
  • top_level_domain
  • parent_domain
  • parent_domain_length

These are related to logstash tld filter plugin:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-tld.html

There may not be a readily available equivalent processor for elasticsearch ingest.

@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Dec 5, 2019

Jim shared a painless script to get the domain info missing in the bro_dns pipeline into ES:

{ "script" :
    {   
        "lang": "painless",
        "source": """
            def dparts = /\\./.split(ctx.query);
            if (dparts.length > 1) {
                def dps = new ArrayList();
                for (int i = 0; i < dparts.length; i++) {
                    dps.add(dparts[i])
                }
                def domain; def trd; def tld; def sld;
                int le = dparts.length - 1 ;
                for (int i = le; i >= 0; i--) {
                    if (i == le) {
                        tld = dps.remove(i)
                    } else if (i == le - 1) {
                        sld = dps.remove(i)
                    }
                }
                ctx.top_level_domain = tld;
                ctx.parent_domain = sld;
                ctx.highest_registered_domain = sld + '.' + tld;
                if (dps.size() > 0) {
                    ctx.subdomain = String.join('.', dps)
                }
            }
        """,
        "if"  : "ctx.query_type_name != 'NB' && ctx.query_type_name != 'TKEY' && ctx.query_type_name != 'NBSTAT' && ctx.query_type_name != 'PTR'",
        "on_failure" : [
            {   
                "set" : { "field" : "error", "value" : "{{ _ingest.on_failure_message }}" }
            }
        ]
    }
}

and Jim said:

This could probably be done better, but it seems to work, finally
Some notes:
The triple quotes don't work with curl
oops
If you assign a value to an already defined 'dotted' value (like tld which has tld.subdomain), you'll get no error in either the ES logs, or in the event itself as the event is just dropped

@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Dec 10, 2019

Note from Jim:

BTW, that painless script for bro_dns also works well for bro_http/virtual_host
@jimhranicky

This comment has been minimized.

Copy link

@jimhranicky jimhranicky commented Dec 11, 2019

When used with the bro_http/virtual_host, the "if" field should be changed
to something like this:

"if": "!(ctx.virtual_host =~ /^\\d+\\.\\d+\\.\\d+\\.\\d+$/)",
@jimhranicky

This comment has been minimized.

Copy link

@jimhranicky jimhranicky commented Dec 11, 2019

Well, that causes problems:

"if": "ctx.virtual_host != null && !(ctx.virtual_host =~ /^\\d+\\.\\d+\\.\\d+\\.\\d+$/)",
dougburks added a commit to Security-Onion-Solutions/securityonion-elastic that referenced this issue Dec 27, 2019
@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Dec 27, 2019

Hi @jimhranicky ,

I discovered that a combination of lastIndexOf and substring made this a little easier to work into our existing ingest parser format. Please take a look at Security-Onion-Solutions/securityonion-elastic@13f7527 and let me know what you think.

Thanks!

@dougburks dougburks moved this from To do to In progress in 16.04.6.4 Dec 27, 2019
@dougburks dougburks self-assigned this Jan 3, 2020
@dougburks dougburks moved this from In progress to In Testing in 16.04.6.4 Jan 4, 2020
@defensivedepth

This comment has been minimized.

Copy link
Collaborator

@defensivedepth defensivedepth commented Jan 14, 2020

Looks like these are parsing out just fine from what I can see.

@defensivedepth defensivedepth moved this from In Testing to Tested in 16.04.6.4 Jan 14, 2020
@dougburks

This comment has been minimized.

@dougburks dougburks closed this Feb 5, 2020
16.04.6.4 automation moved this from Tested to Done Feb 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
16.04.6.4
  
Done
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.