New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter large uconns #291

Merged
merged 22 commits into from Dec 10, 2018

Conversation

Projects
None yet
3 participants
@meljbruno
Copy link
Collaborator

meljbruno commented Dec 7, 2018

Counts the number of connections between two unique hosts. If the number exceeds 250,000, any additional connections are thrown out and the source IP, destination IP, and connection count are added to a separate table. This will cause a slight increase in import time, but greatly improve analysis time for networks with hosts exchanging hundreds of thousands or millions of connections between two hosts, often as the result of placement of the packet capture. Additional connections between two hosts beyond 250,000 do not need to be analyzed, because at the point where a host is reaching out twice every second vs five times every second does not give us greater insight into whether the connection is a beacon or not.

Closes #270

lisaSW and others added some commits Nov 27, 2018

@meljbruno meljbruno self-assigned this Dec 7, 2018

@meljbruno meljbruno requested review from ethack and lisaSW Dec 7, 2018

@ethack ethack force-pushed the 518-filter-large-uconns branch from 55bffeb to 53c16aa Dec 10, 2018

@ethack

ethack approved these changes Dec 10, 2018

@@ -152,11 +161,22 @@ func indexFiles(files []string, indexingThreads int,
//threads to use to parse the files, whether or not to sort data by date,
// a MogoDB datastore object to store the bro data in, and a logger to report
//errors and parses the bro files line by line into the database.
func parseFiles(indexedFiles []*fpt.IndexedFile, parsingThreads int, datastore Datastore, logger *log.Logger) {
func (fs *FSImporter) parseFiles(indexedFiles []*fpt.IndexedFile, parsingThreads int, datastore Datastore, logger *log.Logger) {

This comment has been minimized.

@ethack

ethack Dec 10, 2018

Collaborator

Adding parseFiles to FSImporter to gain access to the fs.res value. This is used below to access fs.res.Config.T.Structure.ConnTable.

We should either pass the ConnTable value in as a parameter and remove the (fs *FSImporter), or remove the parsingThreads parameter since that can now be accessed via fs.parseThreads instead. This trickles down to fs.bulkRemoveHugeUconns which would also need some work to convert.

Not critical for the deadline but just some code smell.

fs.bulkRemoveHugeUconns(datastore, indexedFiles[0].TargetDatabase, filterHugeUconnsMap, connMap)
}

// robomongo verification stuf:

This comment has been minimized.

@ethack

ethack Dec 10, 2018

Collaborator

This might make a good integration test case.

@ethack ethack merged commit 5478740 into master Dec 10, 2018

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@meljbruno meljbruno removed the needs review label Dec 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment