Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Osquery daemon integration in Hubble (#507)
* Fixes for returner code * Refactor `data` as `block_data` and pass `chained` to fdg module * Add module loading for fdg * import CommandExecutionError Also fixed it in the nova grep module because I noticed it was missing * Add fdg grep module * Fix some copy pasta, missing imports, bad variable names * Make grep_args optional * Blocks don't return statuses * Only need stdout from grep * Fixes for ease of use with grep * Apparently *args is a tuple, not a list * More error checking for blocks * adding systemd conf for coreos * updating dev Dockerfile for coreos to include systemd conf * fixing afterinstall script for debian 8 and 9 * adding control script to manage hubble and schedule it using cron * Update afterinstall.sh * rebasing * Adding cron job in separate file * adding cleanup step for cron.d * Add trailing newline to hubble-autostart * pylint setup from salt, adjust config, provide travis configs * DRY #1: move the hec to a container * DRY #2: Remove all duplicate classes … There's minor differences between these hec classes that will all need to be addressed, but this is a good start. * DRY #4: unfolded tesseract merge: pull my unifnished HEC re-write back in * my global _get_options() replacement — aka get_splunk_options() Theoretically we can recurse down various trees with it: l1 = get_splunk_options('hubblestack:returner:splunk') l2 = list() for l1_item in l1: l2.extend( get_splunk_options('something:else:here', **l1_item) l2 has all the opts of something:else:here plus whatever was in hubblestack:returner:splunk that wasn't replaced in something:else:here * my new deck class —- should survive various pickle and sqlite issues, will require testing * instruct the daemon to populate the grains and opts in hubblestack.hec.opt * remove this older hec in favor of the one that "does everything" * the new improved object almost certainly doesn't work yet, but this is sorta the final layout * teach the new hubblestack.hec to provide get_splunk_options and http_event_collector * teach the splunklogger to use the new hec; time for some initial testing * fix an enormous number of problems (still working through the failed merge) * DRY _get_options() — aka get_splunk_options() * finalize disk queueing and some fake logging (to prevent log→ hec→log→hec loops) * pull in diskcache * make some minor changes to dq and test it for correctness * add encoding line * use __name__ * eliminate fakelogging * this should work. testing. * IT WORKS!! * provide a method for testing the __opts__ generators * at a minimum, we must pull the right sourcetype from the configs * testing required, but this should fix the various … sourcetype config problems and give a clear path for adding other local customizations * changed the execution to use setup.py instead of pip install -e. this fixes a bug in SaltStacks code. I also moved the updating of pip until after salt is installed because pip 18 is vastly different than pip 9 * removed the requirement of building on Server 2012r2. This has been fixed with the build env fix. * rephrase to avoid confusion about the retry. * remove default from get_splunk_options() arguments * while I intended for _nick to add to nicknames... this causes a real problem: if we have sourcetype_nova → sourcetype and also sourcetype_log → sourcetype in the mapping... then it's a crapshoot which one sets sourcetype, so when nova sets sourcetype_nova → sourcetype, it may not be pulling sourcetype from the right config field. Jeepers. Colton figured it out. * sourcetype= defaults won't work as expected, change to sourcetype_nova= Colton found this out too * Add the skeleton and docs * Add the code * Add a space between command and args * Handle empty args * Whoops * emit some grains to syslog this helps to identify and correlate host syslog data with hubble data * Allow starting "chained" argument in FDG * fixed trying to load firewall information before it is needed. This will stop errors and improve performance * Add initial topfile support for fdg * instantiated variable __firewalldata__ * Fix a bad comment and some trailing whitespace * Saltify topfile paths * Change fdg.top return to a dict This will allow the user to identify which result belongs to which fdg file * Fix copy pasta * Add docs for fdg.top return format * log the actual error. why is this ever happening (fairly rarely) * Add fdg osquery module * Add some logging between fdg executions * Fix syntax error * Use debug logs (since this can be big output) * log the whole traceback when the mystery exception raises * Add fdg curl module * add logic to configure the grains to be emitted to syslog unless the configuration option is not explicitly configured the default set of grains are emitted * Remove the port (requests includes it in the url) * Actually call json() function * Clarify return format * Revert "log the whole traceback when the mystery exception raises" This reverts commit f62b570. * this is where the filter goes... I need to prove it's useful though. * ∀ k∈ fields | k∈ event → event[k + _meta] = event.pop(k) * status container * import, create instance, setup dumpster * add durations as optional stats; add set_status_dumpster() location * provide a super uncomplicated-decorator to measure durations of function calls * fix goofy typo * move the set_status_dumpster() invocation to a less stupid place * setup a couple watches, leave a (false) note that set_status_dumpster() starts signal handling * provide the signal handler * fix some @property/@classmethod issues and add a HEALTH footer to the status.json * Add json function to readfile FDG module * track top/queries in nebula and top/process in pulsar * update the docstrings so people can figure this beast out someday * we should probably count nova results too * one last note * try to add a good deal more documentation * One setting isn't going to do the trick. Make status.py use __opts__ like everything else * Add yaml to readfile in FDG * more documentation * setup various hubble.status options and defaults * make hubblestack.status.__opts__ = {'blah': …} actually work * cachedir + dumpster / SIGUSR1 checker on windows * use __opts__[cachedir] + hubble:status:dumpster to construct dumpster location * if signal lacks 'SIGUSR1' attribute, do not attempt to set the kill -10 handler * FDG data processing functions * FDG data processing functions * Fix win_secedit error Fixes #493 * Add vulners to hook * Add diskcache to requirements files Also remove vulners pin * Undo accidental piece of last commit * Add documentation/design for readfile.config * Add readfile.config code * Fix a few issues found in testing * Another fix * Strip newlines * Update nebula_osquery.py Adding osqueryd monitor and log parsing functionality * Adding splunk returner for osquery daemon logs * Adding sample conf for osquery monitor and log parser * Update hubble * add brief note about returners * update config path and add code block * FDG data processing module * FDG data processing functions * Use print() function in both Python 2 and Python 3 __print()__ is a function in Python 3. * Python 3 syntax error: 0755 --> 0o755 $ __python3 -c "0755"__ # --> SyntaxError: invalid token * Travis CI: Run tests in parallel on Python 3 in allow_failures mode * Python 3 syntax error: '\Users:' --> r'\Users:' Python 3 will treat __\U__ is a string as a Unicode escape and will raise a syntax error so let use an r'string' instead of a normal 'string'. * $ __python3 -c "print('\U')__ # SyntaxError: (unicode error) * $ __python3 -c "print(r'\U')__ # __\U__ * Python 3 syntax error: '\Users:' --> r'\Users:' Python 3 will treat __\U__ in a string as a Unicode escape and will raise a syntax error so let's use an r'string' instead of a normal 'string'. * $ __python3 -c "print('\U')__ # SyntaxError: (unicode error) * $ __python3 -c "print(r'\U')__ # __\U__ * parameter name fix * trivial change * a few more docs * move the json formatting to a class method * an execution module for the scheduler * fix various bad namings of hubblestack.status (formerly hubble.status) * required for os.path.join * minor nitfix * incorporating splunk returner's refactoring * FDG grep on strings and path formatting * Initial commit of fdg splunk returner * Make sure add_query_to_sourcetype is present in splunk config * remove format_pattern argument and add claifying note * Add logging between fdg blocks for ease of debugging * Import json * On second thought, just dump to a string * Normalize fdg_file and fix one reference * Small lint fixes * Don't strip empty dict entries in HEC code The stripping of empty fields already happens in the returners themselves. Additionally, doing this recursively was going overboard -- if I have have nested data with None values, I want to know. If we do need to bring this back eventually, we need to use copy.deepcopy or similar to avoid side effects. This was hard to track down. * Don't strip fdg_results if it's empty * adding try blocks for file open, osqueryd_monitor will not check for restart condition if it has just started the osqueryd, servicename is now fixed to hubble_osqueryd * import, create instance, setup dumpster * add durations as optional stats; add set_status_dumpster() location * fix goofy typo * one last note * Add json function to readfile FDG module * Update nebula_osquery.py Adding osqueryd monitor and log parsing functionality * revamping masking feature * Add globbing support to object masking in nebula * Some day I will remember the import * Add dockerfile for cent7 dev * testing functions for readfile.json * tests for invalid json file * Comment fixes * adding osquerd binary to hubble package * add function to replace empty str with None * fix indent * add function to replce empty str with None * creating a separate folder for osqueryd * Fix osqueryd sourcetype override and adding Globbing support * Resolving osqueryd's unsafe permission error * Removing osqueryd safe permissions issue * Create osquerd_safe_permission check * Register osqueryd as a service * Updating osqueryd path * Debugging * Debugging * Debugging * Debugging * Debugging * uncommenting windows log rotation code * Debugging * Fixing the osqueryd unsafe permission issue * Fixing the osqueryd unsafe permissions issue * fixing rebase break * Fixing log rotation and doing refactoring * resolving conflict * resolving diff in daemon * Fixing masking bugs * Removing globbing due to performance issue * Debugging * Adding info logs to osqueryd implementation * Uninstall osqueryd * Adding dependancy of osqueryd on hubble * conditionally enable globbing in nebula masking * Updating osqueryd service name in linux * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Fixing log rotation in windows * Fixing jsonify columns * Taking hubble path as parameter * Taking hubble path as parameter * Passing hubble install directory to osqueryd * Removing osqueryd service registration from this file * Using hubble path passed from the args * Debugging * Debbugging * creating osqueryd log and backup folders * updating as per the review comments * tweaking performance * Removing redundant code * Update file * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update safe permissions * Debugging * Update hubble-setup.nsi * minor change for consistency * Fixing osquery default path in server * Adding debug log * Updating new returner as per latest code changes * Adding examples, updating docstrings * Fixing default params value * Shortening sourcetype names * Fixing review comments * closing file pointers explicitly * review comments done
- Loading branch information
