Fix up pulsar for non-beacon use #1
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
basepi
pushed a commit
that referenced
this pull request
Jul 24, 2017
Adding new function to misc module: check_all_ports_firewall_rules
jettero
referenced
this pull request
in jettero/hubble
Jan 22, 2018
Fix up pulsar for non-beacon use
jettero
referenced
this pull request
in jettero/hubble
Jan 22, 2018
Adding new function to misc module: check_all_ports_firewall_rules
jettero
referenced
this pull request
in jettero/hubble
Feb 1, 2018
jettero
referenced
this pull request
in jettero/hubble
Apr 2, 2018
jettero
referenced
this pull request
in jettero/hubble
Sep 12, 2018
jettero
referenced
this pull request
in jettero/hubble
Sep 12, 2018
jettero
referenced
this pull request
in jettero/hubble
Sep 17, 2018
jettero
referenced
this pull request
in jettero/hubble
Sep 20, 2018
jettero
referenced
this pull request
in jettero/hubble
Sep 27, 2018
jettero
referenced
this pull request
in jettero/hubble
Oct 23, 2018
jettero
referenced
this pull request
in jettero/hubble
Oct 24, 2018
basepi
pushed a commit
that referenced
this pull request
Jan 25, 2019
* Fixes for returner code * Refactor `data` as `block_data` and pass `chained` to fdg module * Add module loading for fdg * import CommandExecutionError Also fixed it in the nova grep module because I noticed it was missing * Add fdg grep module * Fix some copy pasta, missing imports, bad variable names * Make grep_args optional * Blocks don't return statuses * Only need stdout from grep * Fixes for ease of use with grep * Apparently *args is a tuple, not a list * More error checking for blocks * adding systemd conf for coreos * updating dev Dockerfile for coreos to include systemd conf * fixing afterinstall script for debian 8 and 9 * adding control script to manage hubble and schedule it using cron * Update afterinstall.sh * rebasing * Adding cron job in separate file * adding cleanup step for cron.d * Add trailing newline to hubble-autostart * pylint setup from salt, adjust config, provide travis configs * DRY #1: move the hec to a container * DRY #2: Remove all duplicate classes … There's minor differences between these hec classes that will all need to be addressed, but this is a good start. * DRY #4: unfolded tesseract merge: pull my unifnished HEC re-write back in * my global _get_options() replacement — aka get_splunk_options() Theoretically we can recurse down various trees with it: l1 = get_splunk_options('hubblestack:returner:splunk') l2 = list() for l1_item in l1: l2.extend( get_splunk_options('something:else:here', **l1_item) l2 has all the opts of something:else:here plus whatever was in hubblestack:returner:splunk that wasn't replaced in something:else:here * my new deck class —- should survive various pickle and sqlite issues, will require testing * instruct the daemon to populate the grains and opts in hubblestack.hec.opt * remove this older hec in favor of the one that "does everything" * the new improved object almost certainly doesn't work yet, but this is sorta the final layout * teach the new hubblestack.hec to provide get_splunk_options and http_event_collector * teach the splunklogger to use the new hec; time for some initial testing * fix an enormous number of problems (still working through the failed merge) * DRY _get_options() — aka get_splunk_options() * finalize disk queueing and some fake logging (to prevent log→ hec→log→hec loops) * pull in diskcache * make some minor changes to dq and test it for correctness * add encoding line * use __name__ * eliminate fakelogging * this should work. testing. * IT WORKS!! * provide a method for testing the __opts__ generators * at a minimum, we must pull the right sourcetype from the configs * testing required, but this should fix the various … sourcetype config problems and give a clear path for adding other local customizations * changed the execution to use setup.py instead of pip install -e. this fixes a bug in SaltStacks code. I also moved the updating of pip until after salt is installed because pip 18 is vastly different than pip 9 * removed the requirement of building on Server 2012r2. This has been fixed with the build env fix. * rephrase to avoid confusion about the retry. * remove default from get_splunk_options() arguments * while I intended for _nick to add to nicknames... this causes a real problem: if we have sourcetype_nova → sourcetype and also sourcetype_log → sourcetype in the mapping... then it's a crapshoot which one sets sourcetype, so when nova sets sourcetype_nova → sourcetype, it may not be pulling sourcetype from the right config field. Jeepers. Colton figured it out. * sourcetype= defaults won't work as expected, change to sourcetype_nova= Colton found this out too * Add the skeleton and docs * Add the code * Add a space between command and args * Handle empty args * Whoops * emit some grains to syslog this helps to identify and correlate host syslog data with hubble data * Allow starting "chained" argument in FDG * fixed trying to load firewall information before it is needed. This will stop errors and improve performance * Add initial topfile support for fdg * instantiated variable __firewalldata__ * Fix a bad comment and some trailing whitespace * Saltify topfile paths * Change fdg.top return to a dict This will allow the user to identify which result belongs to which fdg file * Fix copy pasta * Add docs for fdg.top return format * log the actual error. why is this ever happening (fairly rarely) * Add fdg osquery module * Add some logging between fdg executions * Fix syntax error * Use debug logs (since this can be big output) * log the whole traceback when the mystery exception raises * Add fdg curl module * add logic to configure the grains to be emitted to syslog unless the configuration option is not explicitly configured the default set of grains are emitted * Remove the port (requests includes it in the url) * Actually call json() function * Clarify return format * Revert "log the whole traceback when the mystery exception raises" This reverts commit f62b570. * this is where the filter goes... I need to prove it's useful though. * ∀ k∈ fields | k∈ event → event[k + _meta] = event.pop(k) * status container * import, create instance, setup dumpster * add durations as optional stats; add set_status_dumpster() location * provide a super uncomplicated-decorator to measure durations of function calls * fix goofy typo * move the set_status_dumpster() invocation to a less stupid place * setup a couple watches, leave a (false) note that set_status_dumpster() starts signal handling * provide the signal handler * fix some @property/@classmethod issues and add a HEALTH footer to the status.json * Add json function to readfile FDG module * track top/queries in nebula and top/process in pulsar * update the docstrings so people can figure this beast out someday * we should probably count nova results too * one last note * try to add a good deal more documentation * One setting isn't going to do the trick. Make status.py use __opts__ like everything else * Add yaml to readfile in FDG * more documentation * setup various hubble.status options and defaults * make hubblestack.status.__opts__ = {'blah': …} actually work * cachedir + dumpster / SIGUSR1 checker on windows * use __opts__[cachedir] + hubble:status:dumpster to construct dumpster location * if signal lacks 'SIGUSR1' attribute, do not attempt to set the kill -10 handler * FDG data processing functions * FDG data processing functions * Fix win_secedit error Fixes #493 * Add vulners to hook * Add diskcache to requirements files Also remove vulners pin * Undo accidental piece of last commit * Add documentation/design for readfile.config * Add readfile.config code * Fix a few issues found in testing * Another fix * Strip newlines * Update nebula_osquery.py Adding osqueryd monitor and log parsing functionality * Adding splunk returner for osquery daemon logs * Adding sample conf for osquery monitor and log parser * Update hubble * add brief note about returners * update config path and add code block * FDG data processing module * FDG data processing functions * Use print() function in both Python 2 and Python 3 __print()__ is a function in Python 3. * Python 3 syntax error: 0755 --> 0o755 $ __python3 -c "0755"__ # --> SyntaxError: invalid token * Travis CI: Run tests in parallel on Python 3 in allow_failures mode * Python 3 syntax error: '\Users:' --> r'\Users:' Python 3 will treat __\U__ is a string as a Unicode escape and will raise a syntax error so let use an r'string' instead of a normal 'string'. * $ __python3 -c "print('\U')__ # SyntaxError: (unicode error) * $ __python3 -c "print(r'\U')__ # __\U__ * Python 3 syntax error: '\Users:' --> r'\Users:' Python 3 will treat __\U__ in a string as a Unicode escape and will raise a syntax error so let's use an r'string' instead of a normal 'string'. * $ __python3 -c "print('\U')__ # SyntaxError: (unicode error) * $ __python3 -c "print(r'\U')__ # __\U__ * parameter name fix * trivial change * a few more docs * move the json formatting to a class method * an execution module for the scheduler * fix various bad namings of hubblestack.status (formerly hubble.status) * required for os.path.join * minor nitfix * incorporating splunk returner's refactoring * FDG grep on strings and path formatting * Initial commit of fdg splunk returner * Make sure add_query_to_sourcetype is present in splunk config * remove format_pattern argument and add claifying note * Add logging between fdg blocks for ease of debugging * Import json * On second thought, just dump to a string * Normalize fdg_file and fix one reference * Small lint fixes * Don't strip empty dict entries in HEC code The stripping of empty fields already happens in the returners themselves. Additionally, doing this recursively was going overboard -- if I have have nested data with None values, I want to know. If we do need to bring this back eventually, we need to use copy.deepcopy or similar to avoid side effects. This was hard to track down. * Don't strip fdg_results if it's empty * adding try blocks for file open, osqueryd_monitor will not check for restart condition if it has just started the osqueryd, servicename is now fixed to hubble_osqueryd * import, create instance, setup dumpster * add durations as optional stats; add set_status_dumpster() location * fix goofy typo * one last note * Add json function to readfile FDG module * Update nebula_osquery.py Adding osqueryd monitor and log parsing functionality * revamping masking feature * Add globbing support to object masking in nebula * Some day I will remember the import * Add dockerfile for cent7 dev * testing functions for readfile.json * tests for invalid json file * Comment fixes * adding osquerd binary to hubble package * add function to replace empty str with None * fix indent * add function to replce empty str with None * creating a separate folder for osqueryd * Fix osqueryd sourcetype override and adding Globbing support * Resolving osqueryd's unsafe permission error * Removing osqueryd safe permissions issue * Create osquerd_safe_permission check * Register osqueryd as a service * Updating osqueryd path * Debugging * Debugging * Debugging * Debugging * Debugging * uncommenting windows log rotation code * Debugging * Fixing the osqueryd unsafe permission issue * Fixing the osqueryd unsafe permissions issue * fixing rebase break * Fixing log rotation and doing refactoring * resolving conflict * resolving diff in daemon * Fixing masking bugs * Removing globbing due to performance issue * Debugging * Adding info logs to osqueryd implementation * Uninstall osqueryd * Adding dependancy of osqueryd on hubble * conditionally enable globbing in nebula masking * Updating osqueryd service name in linux * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Fixing log rotation in windows * Fixing jsonify columns * Taking hubble path as parameter * Taking hubble path as parameter * Passing hubble install directory to osqueryd * Removing osqueryd service registration from this file * Using hubble path passed from the args * Debugging * Debbugging * creating osqueryd log and backup folders * updating as per the review comments * tweaking performance * Removing redundant code * Update file * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update safe permissions * Debugging * Update hubble-setup.nsi * minor change for consistency * Fixing osquery default path in server * Adding debug log * Updating new returner as per latest code changes * Adding examples, updating docstrings * Fixing default params value * Shortening sourcetype names * Fixing review comments * closing file pointers explicitly * review comments done
basepi
pushed a commit
that referenced
this pull request
Feb 14, 2019
* Add fdg grep module * Fix some copy pasta, missing imports, bad variable names * Make grep_args optional * Blocks don't return statuses * Only need stdout from grep * Fixes for ease of use with grep * Apparently *args is a tuple, not a list * More error checking for blocks * adding systemd conf for coreos * updating dev Dockerfile for coreos to include systemd conf * fixing afterinstall script for debian 8 and 9 * adding control script to manage hubble and schedule it using cron * Update afterinstall.sh * rebasing * Adding cron job in separate file * adding cleanup step for cron.d * Add trailing newline to hubble-autostart * pylint setup from salt, adjust config, provide travis configs * DRY #1: move the hec to a container * DRY #2: Remove all duplicate classes … There's minor differences between these hec classes that will all need to be addressed, but this is a good start. * DRY #4: unfolded tesseract merge: pull my unifnished HEC re-write back in * my global _get_options() replacement — aka get_splunk_options() Theoretically we can recurse down various trees with it: l1 = get_splunk_options('hubblestack:returner:splunk') l2 = list() for l1_item in l1: l2.extend( get_splunk_options('something:else:here', **l1_item) l2 has all the opts of something:else:here plus whatever was in hubblestack:returner:splunk that wasn't replaced in something:else:here * my new deck class —- should survive various pickle and sqlite issues, will require testing * instruct the daemon to populate the grains and opts in hubblestack.hec.opt * remove this older hec in favor of the one that "does everything" * the new improved object almost certainly doesn't work yet, but this is sorta the final layout * teach the new hubblestack.hec to provide get_splunk_options and http_event_collector * teach the splunklogger to use the new hec; time for some initial testing * fix an enormous number of problems (still working through the failed merge) * DRY _get_options() — aka get_splunk_options() * finalize disk queueing and some fake logging (to prevent log→ hec→log→hec loops) * pull in diskcache * make some minor changes to dq and test it for correctness * add encoding line * use __name__ * eliminate fakelogging * this should work. testing. * IT WORKS!! * provide a method for testing the __opts__ generators * at a minimum, we must pull the right sourcetype from the configs * testing required, but this should fix the various … sourcetype config problems and give a clear path for adding other local customizations * changed the execution to use setup.py instead of pip install -e. this fixes a bug in SaltStacks code. I also moved the updating of pip until after salt is installed because pip 18 is vastly different than pip 9 * removed the requirement of building on Server 2012r2. This has been fixed with the build env fix. * rephrase to avoid confusion about the retry. * remove default from get_splunk_options() arguments * while I intended for _nick to add to nicknames... this causes a real problem: if we have sourcetype_nova → sourcetype and also sourcetype_log → sourcetype in the mapping... then it's a crapshoot which one sets sourcetype, so when nova sets sourcetype_nova → sourcetype, it may not be pulling sourcetype from the right config field. Jeepers. Colton figured it out. * sourcetype= defaults won't work as expected, change to sourcetype_nova= Colton found this out too * Add the skeleton and docs * Add the code * Add a space between command and args * Handle empty args * Whoops * emit some grains to syslog this helps to identify and correlate host syslog data with hubble data * Allow starting "chained" argument in FDG * fixed trying to load firewall information before it is needed. This will stop errors and improve performance * Add initial topfile support for fdg * instantiated variable __firewalldata__ * Fix a bad comment and some trailing whitespace * Saltify topfile paths * Change fdg.top return to a dict This will allow the user to identify which result belongs to which fdg file * Fix copy pasta * Add docs for fdg.top return format * log the actual error. why is this ever happening (fairly rarely) * Add fdg osquery module * Add some logging between fdg executions * Fix syntax error * Use debug logs (since this can be big output) * log the whole traceback when the mystery exception raises * Add fdg curl module * add logic to configure the grains to be emitted to syslog unless the configuration option is not explicitly configured the default set of grains are emitted * Remove the port (requests includes it in the url) * Actually call json() function * Clarify return format * Revert "log the whole traceback when the mystery exception raises" This reverts commit f62b570. * this is where the filter goes... I need to prove it's useful though. * ∀ k∈ fields | k∈ event → event[k + _meta] = event.pop(k) * status container * import, create instance, setup dumpster * add durations as optional stats; add set_status_dumpster() location * provide a super uncomplicated-decorator to measure durations of function calls * fix goofy typo * move the set_status_dumpster() invocation to a less stupid place * setup a couple watches, leave a (false) note that set_status_dumpster() starts signal handling * provide the signal handler * fix some @property/@classmethod issues and add a HEALTH footer to the status.json * Add json function to readfile FDG module * track top/queries in nebula and top/process in pulsar * update the docstrings so people can figure this beast out someday * we should probably count nova results too * one last note * try to add a good deal more documentation * One setting isn't going to do the trick. Make status.py use __opts__ like everything else * Add yaml to readfile in FDG * more documentation * setup various hubble.status options and defaults * make hubblestack.status.__opts__ = {'blah': …} actually work * cachedir + dumpster / SIGUSR1 checker on windows * use __opts__[cachedir] + hubble:status:dumpster to construct dumpster location * if signal lacks 'SIGUSR1' attribute, do not attempt to set the kill -10 handler * FDG data processing functions * FDG data processing functions * Fix win_secedit error Fixes #493 * Add vulners to hook * Add diskcache to requirements files Also remove vulners pin * Undo accidental piece of last commit * Add documentation/design for readfile.config * Add readfile.config code * Fix a few issues found in testing * Another fix * Strip newlines * Update nebula_osquery.py Adding osqueryd monitor and log parsing functionality * Adding splunk returner for osquery daemon logs * Adding sample conf for osquery monitor and log parser * Update hubble * add brief note about returners * update config path and add code block * FDG data processing module * FDG data processing functions * Use print() function in both Python 2 and Python 3 __print()__ is a function in Python 3. * Python 3 syntax error: 0755 --> 0o755 $ __python3 -c "0755"__ # --> SyntaxError: invalid token * Travis CI: Run tests in parallel on Python 3 in allow_failures mode * Python 3 syntax error: '\Users:' --> r'\Users:' Python 3 will treat __\U__ is a string as a Unicode escape and will raise a syntax error so let use an r'string' instead of a normal 'string'. * $ __python3 -c "print('\U')__ # SyntaxError: (unicode error) * $ __python3 -c "print(r'\U')__ # __\U__ * Python 3 syntax error: '\Users:' --> r'\Users:' Python 3 will treat __\U__ in a string as a Unicode escape and will raise a syntax error so let's use an r'string' instead of a normal 'string'. * $ __python3 -c "print('\U')__ # SyntaxError: (unicode error) * $ __python3 -c "print(r'\U')__ # __\U__ * parameter name fix * trivial change * a few more docs * move the json formatting to a class method * an execution module for the scheduler * fix various bad namings of hubblestack.status (formerly hubble.status) * required for os.path.join * minor nitfix * incorporating splunk returner's refactoring * FDG grep on strings and path formatting * Initial commit of fdg splunk returner * Make sure add_query_to_sourcetype is present in splunk config * remove format_pattern argument and add claifying note * Add logging between fdg blocks for ease of debugging * Import json * On second thought, just dump to a string * Normalize fdg_file and fix one reference * Small lint fixes * Don't strip empty dict entries in HEC code The stripping of empty fields already happens in the returners themselves. Additionally, doing this recursively was going overboard -- if I have have nested data with None values, I want to know. If we do need to bring this back eventually, we need to use copy.deepcopy or similar to avoid side effects. This was hard to track down. * Don't strip fdg_results if it's empty * adding try blocks for file open, osqueryd_monitor will not check for restart condition if it has just started the osqueryd, servicename is now fixed to hubble_osqueryd * import, create instance, setup dumpster * add durations as optional stats; add set_status_dumpster() location * fix goofy typo * one last note * Add json function to readfile FDG module * Update nebula_osquery.py Adding osqueryd monitor and log parsing functionality * revamping masking feature * Add globbing support to object masking in nebula * Some day I will remember the import * Add dockerfile for cent7 dev * testing functions for readfile.json * tests for invalid json file * Comment fixes * adding osquerd binary to hubble package * add function to replace empty str with None * fix indent * add function to replce empty str with None * creating a separate folder for osqueryd * Fix osqueryd sourcetype override and adding Globbing support * Resolving osqueryd's unsafe permission error * Removing osqueryd safe permissions issue * Create osquerd_safe_permission check * Register osqueryd as a service * Updating osqueryd path * Debugging * Debugging * Debugging * Debugging * Debugging * uncommenting windows log rotation code * Debugging * Fixing the osqueryd unsafe permission issue * Fixing the osqueryd unsafe permissions issue * fixing rebase break * Fixing log rotation and doing refactoring * resolving conflict * resolving diff in daemon * Fixing masking bugs * Removing globbing due to performance issue * Debugging * Adding info logs to osqueryd implementation * Uninstall osqueryd * Adding dependancy of osqueryd on hubble * conditionally enable globbing in nebula masking * Updating osqueryd service name in linux * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Changing osqueryd binary name * Fixing log rotation in windows * Fixing jsonify columns * Taking hubble path as parameter * Taking hubble path as parameter * Passing hubble install directory to osqueryd * Removing osqueryd service registration from this file * Using hubble path passed from the args * Debugging * Debbugging * creating osqueryd log and backup folders * updating as per the review comments * tweaking performance * Removing redundant code * Update file * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update safe permissions * Debugging * Update hubble-setup.nsi * minor change for consistency * Fixing osquery default path in server * Adding debug log * Updating new returner as per latest code changes * Adding examples, updating docstrings * Fixing default params value * Shortening sourcetype names * Fixing review comments * closing file pointers explicitly * review comments done * Improving masking feature * Improving masking feature and adding hubble metadata function * Fixing masking of custom_mask_key var itself
sumanmehta
pushed a commit
that referenced
this pull request
Nov 27, 2019
daveed12
referenced
this pull request
in daveed12/hubble
Jan 29, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.