Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changelog and README.md cleanup #1648

Merged
merged 7 commits into from Mar 24, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
50 changes: 25 additions & 25 deletions CHANGELOG
Expand Up @@ -253,7 +253,7 @@ NOTICE: Restart wiseService before capture when upgrading
2.2.1 2020/01/21
- capture - fix --skip not working with ES 7.x
- capture - update TLS ciphers
- capture - increaase offlineDispatchAfter default to 2500
- capture - increase offlineDispatchAfter default to 2500
- capture - cert decode publicAlgorithm and curve
- db - optimize-admin doesn't wait for other optimizations to finish
- lua - save/pre_save callbacks, can now get most fields
Expand Down Expand Up @@ -282,7 +282,7 @@ NOTICE: Restart wiseService before capture when upgrading
- capture - disable the 100-Continue feature of curl to reduce bulk errors
- capture - smtp parse now maps a few encodings to standards glib understands
- s3 - support maxFileTimeM
- db - support creating ILM policys and assigning them for sessions2 and history
- db - support creating ILM policies and assigning them for sessions2 and history
- db - new optimize-admin that only optimizes admin indices
- parliament - Click on ES health goes to ES Nodes tab

Expand Down Expand Up @@ -330,7 +330,7 @@ NOTICE: Restart wiseService before capture when upgrading
new maxAggSize setting
- viewer - added right-click replacer for %DBFIELD% (thanks tlacuache)
- viewer - display JA3s and hassh
- viewer - support file expresssion in more places (issue #1172)
- viewer - support file expression in more places (issue #1172)
- viewer - fix files date display (issue #1164)
- viewer - csp header support
- viewer - Fix some XSS/rXSS by setting correct content type
Expand Down Expand Up @@ -372,7 +372,7 @@ NOTICE: Restart wiseService before capture when upgrading
- viewer - add bytes as a graphing choice
- viewer - support ip == ipv4 and ip == ipv6 expressions
- viewer - pivot dropdown option in spiview (issue #1135)
- viewer - optional milisecond display
- viewer - optional millisecond display
- viewer - Support view parameter for unique/multiunique
- viewer - Support ES client auth and insecure better (thanks Scott)
- viewer - Lots of stats summing, avg, sorting fixes
Expand All @@ -383,7 +383,7 @@ NOTICE: Restart wiseService before capture when upgrading
- capture - fixed ja3 mishandling of 11 extension types (thanks Norwegian Healthcare CERT)
- capture - Added startsWith,contains,endsWith rule expression modifier
- capture - honor the caTrustFile directive (thanks Matt)
- capture - fix data bytes calulations for icmp/udp (thanks Brian)
- capture - fix data bytes calculations for icmp/udp (thanks Brian)
- capture - initial vxlan support
- capture - Myricom/AFPacket improvements (thanks Scott)
- capture - updates to classifiers: telnet, mpls
Expand Down Expand Up @@ -427,7 +427,7 @@ NOTICE: Restart wiseService before capture when upgrading
- capture - libfuzzer support and initial fixes
- parliament - add no alert cluster type
- parliament - remove selected acknowledged issues
- parliment - add help page
- parliament - add help page


1.7.0 2019/01/17
Expand Down Expand Up @@ -471,7 +471,7 @@ NOTICE: Restart wiseService before capture when upgrading
- parliament - add option to provide link to dashboard in alert notifications
- viewer - configure connection node/link popup data
- release - build snf plugin with screwdriver
- capture - fix tls parser inifite loop
- capture - fix tls parser infinite loop
- viewer - can customize fields in the info column
- viewer - new es recovery tab
- viewer - stats page shows when data is being loaded from server
Expand Down Expand Up @@ -617,7 +617,7 @@ NOTICE: Restart wiseService before capture when upgrading

1.1.0 2018/04/30
- all - basic sctp capturing, no decoding (issue #828)
- all - intial unencrypted 802.11 Data Frame support (issue #834)
- all - initial unencrypted 802.11 Data Frame support (issue #834)
- db - new segments option to expire and optimize
- release - curl, node, cyberchef
- viewer - http.uri and host* allows pasting a URL and doing the right thing (pull #831)
Expand Down Expand Up @@ -666,7 +666,7 @@ NOTICE: Restart wiseService before capture when upgrading
- capture - fix disable fields
- capture - src/dst ip/port can be used to trigger rules now
- capture - ip fields in rules can now be CIDR
- capture - simple writer now flushes after 10 seconds of no writting
- capture - simple writer now flushes after 10 seconds of no writing
there still can be pagesize bytes unwritten (issue #777)

1.0.0-alpha2 2018/01/31
Expand Down Expand Up @@ -696,7 +696,7 @@ NOTICE: Restart wiseService before capture when upgrading
- all - new hourly6 rotateIndex, for every 6 hours
- parliament - first version of alerts
- parliament - rename server.js to parliament.js
- wise - trim spaces after spliting config values
- wise - trim spaces after splitting config values
- capture - better pop3 detection
- capture - correctly lowercase user
- release - added --install to easybutton (issue #812)
Expand Down Expand Up @@ -791,7 +791,7 @@ NOTICE: Restart wiseService before capture when upgrading
- viewer - add href to nav tab links (issue #651)
- viewer - save spiview fields (issue #715)
- easybutton - Upgrade yara, glib, curl versions
- viewer - session columns are resizeable (issue #676)
- viewer - session columns are resizable (issue #676)
- wise - ja3 support
- capture - fixed capture crash when wiseService is restarted
- release - build a better NOTICE file for binary releases
Expand Down Expand Up @@ -878,7 +878,7 @@ NOTICE: Restart wiseService before capture when upgrading
- capture - handle certs with pre epoch times
- capture - more magicMode basic detections
- db - Handle timeouts and nodeNames better with upgrading from db version 30 to 34
- capture - fix race condition when quiting and processing unsaved sessions
- capture - fix race condition when quitting and processing unsaved sessions

0.18.0 2017/03/21
- NOTICE: Upgrading to 0.18 requires an outage for db.pl upgrade (~5 min)
Expand Down Expand Up @@ -936,7 +936,7 @@ NOTICE: Restart wiseService before capture when upgrading
- all - Initial pcap encoding support, variables
* simpleEncoding: aes-256-ctr or xor-2048
* simpleKEKId - The kek id to use from the keks section
* [keks] - A section with multipl kekid=passwords
* [keks] - A section with multiple kekid=passwords
- capture - Fixed yara 1.7 multithread crash (issue #568)
- capture - Handle frag gre with frag ip inside
- viewer - New Help UI (based on new ui)
Expand Down Expand Up @@ -1050,7 +1050,7 @@ NOTICE: Restart wiseService before capture when upgrading
0.14.1 2016/06/02
- NOTICE: glib 2.40 required (should be using 2.47 or higher)
- capture - initial lua scripting support
- wise - initial bro quering support
- wise - initial bro querying support
- capture - debug all config requests with --debug
- viewer - fixed XSS (issue #471)
- capture - simple ldap parser
Expand Down Expand Up @@ -1102,9 +1102,9 @@ NOTICE: Restart wiseService before capture when upgrading
- tests - http.referer tests
- capture - smtp now handles no space for mail from, rcpt to (issue #442)
- capture - basic jabber/sip protocol detection
- capture - http:password set for case insensitve password= now
- capture - http:password set for case insensitive password= now
- capture - rdp "Cookie" if present is stored in user field
- viewer - support auto complete on all fields (experimental)
- viewer - support autocomplete on all fields (experimental)
- capture - fix for tagger.so crash on XFF fields introduced in 0.12.2
- easybutton - node 0.10.43, libpcap 1.7.3
- easybutton - disable bluetooth in libpcap (issue #445)
Expand Down Expand Up @@ -1184,15 +1184,15 @@ NOTICE: Restart wiseService before capture when upgrading
- viewer - Switched hacking viewer
- viewer - Decoding is now handled with node streams which will allow
chaining and plugin decoders eventually - npm update required
- capture - Added cookie value parsing, default off controled by
- capture - Added cookie value parsing, default off controlled by
parseCookieValue (issue #371)
- viewer - EXISTS! fixes, tests, and docs (issue #367)
- viewer - Reuse Uncompress and Files settings between sessions
- viewer - Anonymous users still have a singled saved settings
- viewer - Switch actions/views menu to jquery-ContextMenu which looks more like a menu
- capture - Handle ASCII formated SMB strings
- capture - payload8 wasn't always all 8 bytes
- viewer - Initial configration of displayed columns (issue #257)
- viewer - Initial configuration of displayed columns (issue #257)
- capture - reenabled ftp/lmtp classify
- capture - vnc classify (issue #382)
- capture - fixed tcp_flags for netflow (pr #386)
Expand Down Expand Up @@ -1268,7 +1268,7 @@ NOTICE: Restart wiseService before capture when upgrading
- if upgrading (cd plugins ; rm emergingthreats.detail.jade opendns.detail.jade threatq.detail.jade threatstream.detail.jade)
- New offlineFilenameRegex setting to control witch files are matched with -R (issue #313)
- monitor + recursive should monitor new directories (issue #305)
- Fixed addUser.js error with when mulitple es nodes are listed in config.ini (issue #322)
- Fixed addUser.js error with when multiple es nodes are listed in config.ini (issue #322)
- WISE - Tagger files can have views defined with #view:
- New cert.notbefore, cert.notafter, cert.validfor fields (issue #329)
- New starttime, stoptime, view fields (issue #307)
Expand Down Expand Up @@ -1341,7 +1341,7 @@ NOTICE: Restart wiseService before capture when upgrading
- Mouse over view names shows expression (issue #220)
- Display SPI Data even if node is unavailabe (issue #219)
- Netflow plugin timestamp fixes (issue #241)
- Comma separate list of elasticsearch hosts (issue #176)
- Comma separated list of elasticsearch hosts (issue #176)
- New includes directive (issue #144)
- Initial bigendian support in viewer (issue #259)
- List queries can now have wildcard and regex items.
Expand Down Expand Up @@ -1397,7 +1397,7 @@ NOTICE: Restart wiseService before capture when upgrading
round robin is supported
- UI: Fix Search/Actions showing up on second line on page load
- capture now does memlock and max schedule priority on startup (issue #199)
- when yara is disabled dont retain extra data
- when yara is disabled don't retain extra data
- parse email user names
- antiSynDrop config option
- remove schedule priority change for now
Expand Down Expand Up @@ -1550,7 +1550,7 @@ NOTICE: Restart wiseService before capture when upgrading
- Work around for tcp seq number wrapping causing viewer exit
- dns parsing core fix
- switch to nonblocking pcap saves
- more debuging info on proxy failure
- more debugging info on proxy failure
- Fixed bug when setting viewUrl
- Limit number of libnids errors (issue #115)
- Display possible reasons for libnids IP Header error
Expand Down Expand Up @@ -1586,7 +1586,7 @@ NOTICE: Restart wiseService before capture when upgrading
- Deal with non data ES nodes
- Viewer prints error if it can't find pcapDir setting
- New setting dbFlushTimeout that controls how often we flush to ES
- New setting compressES that turns on compresesion to ES, requires
- New setting compressES that turns on compression to ES, requires
http.compression: true in elasticsearch yml file
- libnids was overreporting traffic, switch to libpcap stats,
bytes/sec and total bytes/sec in stats will be lower
Expand Down Expand Up @@ -1678,7 +1678,7 @@ NOTICE: Restart wiseService before capture when upgrading
- config spiDataMaxIndices controls how many indices to run against since
spiview feature can cause elastic search to blowup memory.
- display date as year/mon/day
- Lots of UI cleanup, slighly less ugly as before hopefully
- Lots of UI cleanup, slightly less ugly as before hopefully
- 32 bit builds should work
- Fixed bug where status codes/http methods weren't always recorded
- New SMTP plugin callbacks, more to come
Expand All @@ -1694,7 +1694,7 @@ NOTICE: Restart wiseService before capture when upgrading
- fixed user name XSS issue [thanks z0mbiehunt3r]
- fixed many viewer exits
- timestamp display option in sessionDetail
- graph now uses seconds if less then 30 minutes and hours if more
- graph now uses seconds if less than 30 minutes and hours if more
then 5 days. This makes display faster
- Refactored how capture stores spi data in memory
- Refactored hash table code
Expand Down
6 changes: 3 additions & 3 deletions CODE_OF_CONDUCT.md
Expand Up @@ -17,10 +17,10 @@ This Code of Conduct makes our expectations of participants in this community ex
We expect participants in this community to conduct themselves professionally. Since our primary mode of communication is text on an online forum (e.g. issues, pull requests, comments, emails, or chats) devoid of vocal tone, gestures, or other context that is often vital to understanding, it is important that participants are attentive to their interaction style.

* **Assume positive intent.** We ask community members to assume positive intent on the part of other people’s communications. We may disagree on details, but we expect all suggestions to be supportive of the community goals.
* **Respect participants.** We expect participants will occasionally disagree. Even if we reject an idea, we welcome everyone’s participation. Open Source projects are learning experiences. Ask, explore, challenge, and then respectfully assert if you agree or disagree. If your idea is rejected, be more persuasive not bitter.
* **Respect participants.** We expect participants will occasionally disagree. Even if we reject an idea, we welcome everyone’s participation. Open Source projects are learning experiences. Ask, explore, challenge, and then respectfully assert if you agree or disagree. If your idea is rejected, be more persuasive, not bitter.
* **Welcoming to new members.** New members bring new perspectives. Some may raise questions that have been addressed before. Kindly point them to existing discussions. Everyone is new to every project once.
* **Be kind to beginners.** Beginners use open source projects to get experience. They might not be talented coders yet, and projects should not accept poor quality code. But we were all beginners once, and we need to engage kindly.
* **Consider your impact on others.** Your work will be used by others, and you depend on the work of others. We expect community members to be considerate and establish a balance their self-interest with communal interest.
* **Consider your impact on others.** Your work will be used by others, and you depend on the work of others. We expect community members to be considerate and establish a balance between their self-interest and communal interest.
* **Use words carefully.** We may not understand intent when you say something ironic. Poe’s Law suggests that without an emoticon people will misinterpret sarcasm. We ask community members to communicate plainly.
* **Leave with class.** When you wish to resign from participating in this project for any reason, you are free to fork the code and create a competitive project. Open Source explicitly allows this. Your exit should not be dramatic or bitter.

Expand All @@ -34,7 +34,7 @@ Participants remain in good standing when they do not engage in misconduct or ha
* **Don't disrupt.** Sustained disruptions in a discussion.
* **Let us help.** Refusal to assist the Response Team to resolve an issue in the community.

We do not list all forms of harassment, nor imply some forms of harassment are not worthy of action. Any participant who *feels* harassed or *observes* harassment, should report the incident. Victim of harassment should not address grievances in the public forum, as this often intensifies the problem. Report it, and let us address it off-line.
We do not list all forms of harassment, nor imply some forms of harassment are not worthy of action. Any participant who *feels* harassed or *observes* harassment, should report the incident. Victims of harassment should not address grievances in the public forum, as this often intensifies the problem. Report it, and let us address it off-line.

### Reporting Issues
If you experience or witness misconduct, or have any other concerns about the conduct of members of this project, please report it by contacting our Response Team at opensource-conduct@verizonmedia.com who will handle your report with discretion. Your report should include:
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Expand Up @@ -135,7 +135,7 @@ The documentation lives on our website at [arkime.com/api](https://arkime.com/ap

### Code of Conduct

We encourage inclusive and professional interactions on our project. We welcome everyone to open an issue, improve the documentation, report bug or submit a pull request. By participating in this project, you agree to abide by our [Code of Conduct](CODE_OF_CONDUCT.md). If you feel there is a conduct issue related to this project, please raise it per the Code of Conduct process and we will address it.
We encourage inclusive and professional interactions on our project. We welcome everyone to open an issue, improve the documentation, report a bug or submit a pull request. By participating in this project, you agree to abide by our [Code of Conduct](CODE_OF_CONDUCT.md). If you feel there is a conduct issue related to this project, please raise it per the Code of Conduct process and we will address it.

---

Expand Down
4 changes: 2 additions & 2 deletions capture/DESIGN.md
Expand Up @@ -4,7 +4,7 @@ Capture is a multithreaded glib2 application

In general capture tries to not use locks for anything but queues when communicating between threads.
When possible we use read only complex data structures shared across threads.
When those data structures need to be updated we create a new one and replace the old one, which is schedule to be freed at a later time (moloch_free_later) so any curernt readers don't crash.
When those data structures need to be updated we create a new one and replace the old one, which is scheduled to be freed at a later time (moloch_free_later) so any current readers don't crash.

## capture
The main thread, all http requests are on the main thread.
Expand Down Expand Up @@ -36,7 +36,7 @@ A single thread that is responsible for writing out to disk the completed pcap b

# Parsers vs Plugins

In reality there isn't much difference between parsers and plugins, other then when they are loaded and when they are initialized.
In reality there isn't much difference between parsers and plugins, other than when they are loaded and when they are initialized.

## Parsers
Anything in the parsers directories (parsersDir) are auto loaded and the moloch_parser_init function is called when loaded.
Expand Down
2 changes: 1 addition & 1 deletion viewer/README.md
Expand Up @@ -61,7 +61,7 @@ npm run start:testuser

For this to work, your `tests/config.test.ini` must be valid.

These first command adds an "admin" user. The second command starts the node server and bundles all Vue app files into `viewer/vueapp/dist`.
The first command adds an "admin" user. The second command starts the node server and bundles all Vue app files into `viewer/vueapp/dist`.

Webpack watches for changes to relevant Vue files, and re-bundles the Vue app after each save.

Expand Down
2 changes: 1 addition & 1 deletion wiseService/README.md
Expand Up @@ -56,7 +56,7 @@ To start the app for development and testing:
* Move to the top level Arkime directory
* run `npm run wise:dev`

This command starts the app with the necessary config options set (`-c ../tests/config.test.ini`) and bundles the unminified application files into into the `wiseService/vueapp/dist` folder.
This command starts the app with the necessary config options set (`-c ../tests/config.test.ini`) and bundles the unminified application files into the `wiseService/vueapp/dist` folder.

`npm run wise:dev` uses webpack to package the files then watches for changes to relevant files, and re-bundles the app after each save.

Expand Down
2 changes: 1 addition & 1 deletion wiseService/molochwise.bro
Expand Up @@ -177,7 +177,7 @@ event file_hash(f: fa_file, kind: string, hash: string)
flush_all();
}

# Make sure nothing is waiting more then 5 seconds
# Make sure nothing is waiting more than 5 seconds
event wisetimer()
{
if (|wise_next_lookups["md5"]| > 0 || |wise_next_lookups["ip"]| > 0) {
Expand Down
2 changes: 1 addition & 1 deletion wiseService/simpleSource.js
Expand Up @@ -31,7 +31,7 @@ const iptrie = require('iptrie');
* Sources need to
* * implement WISESource#initSource
* * implement SimpleSource#simpleSourceLoad
* * they can optionaly call this.load() if they want to force a reload of data
* * they can optionally call this.load() if they want to force a reload of data
* @extends WISESource
*/
class SimpleSource extends WISESource {
Expand Down