network map diffs over time #2
Comments
|
As the work for this would be something I implement in moloch (and then do a pull request upstream when it's done) I've opened this issue for discussion on the Moloch github: |
|
Making good progress: arkime/arkime#1162 (comment) |
mmguero
added a commit
that referenced
this issue
Mar 4, 2020
…issue #2 development rather than aol/moloch releases
31453
referenced
this issue
in arkime/arkime
Mar 19, 2020
* beginning work on #1162, baseline comparison for connections visualization consisting of (all in viewer.js): 1. factored out some code from buildSessionQuery into a determineQueryTimes function for calculating startTime, stopTime, and interval values given a req, and modified buildSessionQuery to call that function instead, as well as a new bit of code in buildConnections 2. In buildConnections, if network graph baseline is enabled (enabled: req.query.baseline=1, disabled:req.query.baseline=0 or undefined) then two queries will be run (ie., run buildSessionQuery->searchPrimary->process twice): first for the original specified time frame and second for the same time frame immediately preceding it. Nodes have an .inresult attribute where: Dec Bin Description 0 = 00 = not in either result set (although you'll never see these, obviously) 1 = 01 = seen during the "current" time frame but not in the "baseline" time frame (ie., "new") 2 = 10 = seen during the "baseline" time frame but not in the "current" time frame (ie., "old") 3 = 11 = seen during both the "current" time frame and the "baseline" time frame This is only performed where startTime/startTime are defined, and never for "all" time range (date=-1). * Work in progress on network diff for #1162, but based on tags/v2.2.2wq * added 'Compare against baseline' checkbox (work in progress)' * added 'Compare against baseline' checkbox (work in progress)' * use border to indicate actual/baseline/both resultsets for connections nodes * use border to indicate actual/baseline/both resultsets for connections nodes * java script is weird, getting baseline vue to work with checkbox "true" vs. true not meaning the same thing * mess with font size for baselining * comments * tried out some different icons and things as per the instructions of @31453 to make nodes more visible in baselining. see #1162 (comment) * use circleslash and sparkles for old/new * added "baseline node visibility" selector, to hide/show nodes and their labels based on which nodes you want to see. still in a half-working state, as i need to figure out showing/hiding the links * disable changing stroke width and color for baseline connections visualization: as we're now putting text style and an icon on there, I think it is too much visually going on with the node stroke too * comments * added 'Baseline Node Visibility' selector (see #1162 (comment)) * only enable 'baseline node visibility' if baseline is selected * visual rearrange of baseline controls and padding * update api test to include new item in connections node structure * put debug comments only in debug blocks * added baselineDate to query parameters for baselining, where baselineDate is the duration in hours for the baseline time frame, immediately preceding the 'actual' time frame * work in progress on buildConnections (using async/await promises to execute actual and baseline query in tandem) * better error handling * don't abort on result set processing early (by having an error handler callback) * fix async problems with result set processing in buildConnections as well * use correct query time frame for indices * added visual controls for baseline time frame (and remove checkbox) * clean up node/link drawing attributes I decided not to use * added API test for connection baselining * rewrote buildConnections code for baseline feature (see #1162 and idaholab/Malcolm#2) to use pure callbacks rather than a combination of callbacks, promises, and async * rewrote buildConnections code for baseline feature (see #1162 and idaholab/Malcolm#2) to use pure callbacks rather than a combination of callbacks, promises, and async * update test * fix lint errors * update api-connections.t to remove length= parameter. The test doesn't need it, as it passes with the default value, and none of the others specify it explicitly. I did discover issue #1395 while putting this together, but I have logged that issue to track that as it's not related to this fork * refactored resultset processing up into its own buildConnections.processResultSets function * support old time frames (like around the epoch) when detecting startTime/stopTime in determineQueryTimes * to unclutter the tool bar, move the unlock and export buttons to the svg frame (similar to zoom, text size, line length) in connections view * connections view: shorten labels, and hide control for baseline node visibility when baseline is disabled * use mjson variable for the connection multis test, even for an error case * fix comment
|
Closed with arkime/arkime#1392 and arkime/arkime#1162, will be in Malcolm v2.0.0 |
mmguero
added a commit
that referenced
this issue
Apr 8, 2020
Tons of stuff. * bump moloch to 2.2.0 * reduce log noise * bump version for development to 1.8.2 * set elastalert index settings for a single node * fix issue #97, when using tcpdump the capture files are named .pcap.pcap * check moloch viewer status page periodically for docker container health check * fix docker-compose log verbosity * made kibana_index_refresh.py more robust as suggested by @fabrie in issue #100 use a _find API to get the index ID for a given index name instead of just the saved_objects/index-pattern API Example in test environment: Before adding new fields (dry run): ``` /home/user/devel/github/malcolm/kibana/scripts/kibana_index_refresh.py Arguments: ['-v', '-n', '-k', 'http://192.168.0.11:5601/kibana'] Arguments: Namespace(debug=True, dryrun=True, index='sessions2-*', url='http://192.168.0.11:5601/kibana') Kibana version is 7.5.1 Index ID for sessions2-* is sessions2-* sessions2-* would have 465 fields success (dry run only, no write performed) ``` After adding new fields (dry run): ``` /home/user/devel/github/malcolm/kibana/scripts/kibana_index_refresh.py Arguments: ['-v', '-n', '-k', 'http://192.168.0.11:5601/kibana'] Arguments: Namespace(debug=True, dryrun=True, index='sessions2-*', url='http://192.168.0.11:5601/kibana') Kibana version is 7.5.1 Index ID for sessions2-* is sessions2-* sessions2-* would have 481 fields success (dry run only, no write performed) ``` After adding new fields (update index mapping): ``` /home/user/devel/github/malcolm/kibana/scripts/kibana_index_refresh.py Arguments: ['-v', '-k', 'http://192.168.0.11:5601/kibana'] Arguments: Namespace(debug=True, dryrun=False, index='sessions2-*', url='http://192.168.0.11:5601/kibana') Kibana version is 7.5.1 Index ID for sessions2-* is sessions2-* sessions2-* would have 481 fields success ``` * added plugin for detecting cve-2020-0601 * work on issue #102, log access to Malcolm web interface(s) to Elasticsearch for analysis in Kibana * nginx/php adjustments for issue #101, uploading very large pcap files may fail * fix a few of the control bash scripts to use GNU coreutils where applicable (issue #103) A few uses of "grep" and "find" use flags unique to the GNU versions of those tools. As GNU coreutils was already required for a few other utilities I've done the same thing to detect and use ggrep and gfind when needed. As suggested, it might be a good idea to rewrite these to use Python instead to be more portable (although I'll have to take care to make them work with both python 2/3 for various platforms). * bump malcolm version to 1.9.0, moloch version to 2.2.1 * work on implementing control scripts (start,stop,restart,wipe,logs) in python rather than bash for portability (see issue #103). have not as of yet removed the bash versions, as I am still testing these new implementations. for now the auth_setup.sh and malcolm_appliance_packager.sh are still in Bash as well * have ISO use new scripts * bump elastic to 7.5.2 * update iso build scripts to use new python scripts for install * compatibility fixes for scripts under linux * don't source missing files * more reworking of scripts from bash -> python (not complete yet, may be in a broken state) * more work on auth_setup * remove reference to files we're not longer using * Revert "bump elastic to 7.5.2" This reverts commit 440c859. * fix default for external password question * handle missing python package in windows * documentation updates * tweak some codenames * use specified path (rather than absolute path) for compose file * make python3 the default * Revert "make python3 the default" This reverts commit 52e53f4. * fixes for new control scripts * fix logs script hanging * pin filters by default in kibana * put a hack/fix in for vagrant not liking dhcp nat in 6.1 * create a zeek.service_version field to track protocol version in a single place; also, move password up to the zeek root level * added security overview dashboard wip * added freq.Dockerfile to detecting string entropy * if designated by the FREQ_LOOKUP (true) environment variable, look up DNS query hostnames using freq_server.py * update docker ignore ifle * ask about string freq lookup in install.py * added security overview dashboard wip * added security overview dashboard wip * use a ruby block rather than an http filter in order to better handle arrays * fix volume mapping for local.zeek in docker-compose.yml for testing * fix volume mapping for local.zeek in docker-compose.yml for testing * clean up symlinks as well * initial code for generating and parsing smb_cmd.log * initial code for generating and parsing smb_cmd.log * more work on smb command mapping * more work on smb * more work on smb * bump version to 2.0.0 * some field normalization for 2.0.0 - restored kibana swimlane visualization as it has been fixed for 7.5.x - remove some unused fields from records (agent.ephemeral_id, input.type, path portion of log.file.path) - remove "_jsonparsefailure" tag on cleanup - change some places where we were doing calculations to get count values when we already know the count is "1" - normalization of "action" or "command" values to "zeek.action" field - normalization of mime type, file names, fuids, and service version * fixes to SMB action mapping * remove useless prefix before smb action * exclude some domains from freq. analysis * utility script to repackage zeek logs for upload: * fix issue #111, moloch/etc mount in docker-compose.yml causes custom Zeek fields not to be loaded * fix Malcolm issue #110, submitting hunt job crashes viewer unless Zeek logs are filtered out (temporary patch of fix for Moloch issue 1374, arkime/arkime#1374) * Added smb_cmd fields to WISE * fix dashboard referring to zeek_smb.action -> zeek.action * remove tunnel:: prefix from tunnel type * added 'action' panel to overview * added security overview dashboard (wip) to directory * more work on issue #108, create security overview dashboard in kibana * working on issue #109, create ICS security overview dashboard * added ipv4/ipv6 * working on issue #109, create ICS security overview dashboard * working on issue #109, create ICS security overview dashboard * added network layer to connections * fix max font size * bring sensor local.zeek up to match malcolm's * fixed spacing of navigation menu * fix issue #112, region maps not working because of incorrect redirect * fix issue #112, region maps not working because of incorrect redirect * fix issue #112, region maps not working because of incorrect redirect * fix issue #112, region maps not working because of incorrect redirect * comments * ignore logs that have been renamed and are in transit being archived * updates to dashboards * fix kibana_index_refresh.py for python2 * fix non-ics/iot protocols dashboard * bump version to 7.6.0 for elastic * working with es 7.6, but elastalert had to be temporarily disabled. will work on fixing this next * don't include known_certs in outdated/insecure protocols * Tons of work refining dashboards * tweak connections view * improved maps * remove warnings * improvements to how notices can be used througout the other dashboards * improvements to how notices can be used througout the other dashboards * do frequency analysis on zeek_ssl.server_name * merge src/dst mac/oui fields into network.mac and network.oui arrays, respectively to provide aggregated fields that can be used for asset inventory (issue 113) * experimenting with creating a merged network.mac_oui field that looks like this: ... "network": { "type": "ipv4", "mac_oui": { "00:10:db:ff:10:01": "Juniper Networks", "8c:85:90:65:85:8f": "Apple, Inc." } }, ... however, I may revert this for now because although this works kibana does't really play nicely with the data in visualizations * Revert "experimenting with creating a merged network.mac_oui field that looks like this:" This reverts commit 8bdcefa. * Revert "merge src/dst mac/oui fields into network.mac and network.oui arrays, respectively to provide aggregated fields that can be used for asset inventory (issue 113)" This reverts commit ae60cf2. * make installer work better for vms * build virtualbox guest debs in a clean environment * fixed vagrantfile for malcolm build * only keep vmware/virtualbox guest packages in the right environments * increase build memory requirements * fix typo * updating sensor-iso to match malcolm-iso * fix relative path * removed docker-gen in nginx container, we're not using it any more * update software saved search * Fixed installation of elastalert kibana plugin, but still broken due to this issue: bitsensor/elastalert-kibana-plugin#141 * fix issue #104, Upload without trailing slash redirects to incorrect hostname and scheme this fix includes a few things: - modifying the index.html page to prepend "upload/" before relative HREF/SRC references - removing some useless code in the file-upload default nginx config - added the trailing slash to the proxy-pass directive for the upload section of the main nginx proxy - handle "/server/php" as a separate proxy redirect as that's the XHR where the uploads seem to go - remove the unused Moloch upload page * update copyright * update style of upload screen to match the rest of the app * working on network diff code (wip) * work in progress on network time diff, viewer.js not actually used yet * bump moloch to 2.2.2 * work in progress for network diff * network diff work in progress * comments/work in progress * some test files * update zeek to 3.0.2 * fix reference to zeek::af_packet * install zeek::af_packet with zkg * Revert "install zeek::af_packet with zkg" This reverts commit a20fa9b. * added docker files for running moloch regression tests * added vim to test harnest * fix af_packet zeek build * added promotional poster: * switch test harness branch * temporarily pull from https://github.com/mmguero-dev/moloch fork for issue #2 development rather than aol/moloch releases * update elastic to 7.6.1 for security and bug fixes * update moloch to 2.2.3 * update moloch to 2.2.3 * update zeek to 3.0.3 * update psutil to fix security alert GHSA-qfc5-mcwq-26q8 * zeek updated website, fix broken link * restore stuff for generating web documentation * fix URL for relocated MITRE ATTACK BZAR plugin * fix broken links for build of Zeek, MITRE ATT&CK BZAR plugin * fixed duplicate plugin URL in script * update moloch version in docs to 2.2.3 * remove files no longer needed for testing * remove files no longer needed for testing and update moloch version in documentation * should fix issue #114. I discovered that even though moloch-capture isn't writing the PCAP files, the pcapDir and maxFileSizeG values still matter for viewer to be able to delete managed pcap files. * should fix issue #114. I discovered that even though moloch-capture isn't writing the PCAP files, the pcapDir and maxFileSizeG values still matter for viewer to be able to delete managed pcap files. * proof of concept for a segment mapping form * work in progress on the segment mapping ui * more work on the segment mapping ui * more work on the segment mapping ui * more work on the segment mapping ui * more work on the segment mapping ui * more work on the segment mapping ui * more work on the segment mapping ui * apply tooltip for table columns * scroll back and forth to selected item * beautify with icons * basic validation client-side * more work on the segment mapping ui (integration with malcolm scripts on logstash startup) * more work on the segment mapping ui (creation of docker image, integration with malcolm's nginx reverse proxy) * Added new icon to malcolm iso for subnet mapping editor * documentation updates * start logstash under supervisord in order to add a process that will watch for changes to the name matching * more work on name-map-ui, allow uploading of the JSON file so it can be pushed to the docker image volume automatically * map location of host/subnet mapping to correct location under name-map-ui container * integrate upload with name-map-ui * add the ability to signal logstash from the net-map-ui container * clear out previous maps between restarts * add ability to save net-map.json from web ui * basic control for restarting logstash via ui controls * put save/restart confirmations in UI * added import button to name map ui * send save-state post value to restart-logstash.php * update documentation * update documentation * remove unused variable * documentation updates * use fonts-symbola instead of fonts-noto-color-emoji * re-enable swimlane visualization * update elasticsearch to 7.6.2; also, fix issue #119 * use default theme in elastalert kibana editor * update kibana plugin version * add user to vboxsf group for using shared folders * the 'run a separate instance of Zeek locally' use case isn't really a big enough use case to have a whole separate docker-compose file for it; especially with the ISO and live capture methods. * ensure all services have a health check * reduce verbosity of health checks in logs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The ability to specify a baseline time range from the Connections view and/or API in Moloch over some time range, then show changes/additions to those connections. This would highlight changes to the logical network diagram and could be used to alert on new devices appearing in the network.
The text was updated successfully, but these errors were encountered: