Skip to content

Commit

Permalink
Cont3xt6 (#1859)
Browse files Browse the repository at this point in the history
* remove cont3xt.ini

* allow caches to be configured per user or shared

* rename to PT/VT so W can complain again later

* make sure cont3xt.ini exists for now

* misc integration summary improvements

focus input
add asn org name
add text itype display
use human readable short number for counts
vertically center buttons and wrap integration header content

* fix shodan count

* return order

* make PT icons all different

* pixel perfect!

* added _severity high for theatstream

* order integration buttons

gray for no results and red for bad ones

* spur improvements

* fix for cont3xt6 hopefully

* hopefully fix cont3xt tests

* add reload button for integration cards

* stats page

* remove toBool

* disable bad test for now

* format/search stats page

add shared vue filters to common

* cont3xt pacakge.json, nodemon cleanup, save packagelock

* move some of the cont3xt specific node_modules into cont3xt package.json
* use package.json nodemonConfig instead of command line
* Save package lock file in rpm/dev so folks know what it was built with

* right justify everything, always include numbers after decimal

* improve v6 queries

* create/delete link groups on settings page

* remove role check for now

* add link groups to cont3xt page

use the store to store link groups

* update link group

* switch back to index

* use subdirs under integrations for everything

* add back icon

* lint

* hopefully fix integrations copy

* initial abuseipdb

* add CAA decode

* lint

* debug, rename threatstream user/key, minor UI changes for Elyse to redo :)

* cacheTimeout can have suffix s,m,h,d,w

* initial twilio

* lint

* initial alien vault otx

* add builtwith, ms/seconds display

* bpgview and count cleanup

* add initial malware bazaar

* lint

* urlhaus

* fix vt, order/card can be overriden in config

* threatfox, date type itype stats

* more date, lint

* first attempt at no result

* support not using ES for cont3xt

* fix shortform not supporting . notation

* misc ui stuff

fix some icons
display no results in integration card
add itype stats
display start time in stats
allow table display for objects

* put tooltip on button, not icon

* fix button tooltips to the bottom

* fix tooltips for integration buttons

* add raw link group edit

* fix link group errors to bottom of page

* sort tables

* more date types

* added /api/user endpoint

* fix allRoles for lmdb

* fix lmdb link group retrieving

* check roles for link groups

* set link colors

* add default sort for stats tables (wise/cont3xt)

set active tab in url for stats tables (wise/cont3xt)
fix hasRoles breaking if no roles

* checkboxes for links

display dns record type value length
display link group creator if user has edit priv

* add settings tabs

fix open all links logic
use _editable instead of hasRole

* switch to 1 api for getting list of link groups

* update linkgroup endpoint

* BGPView, AlienVault and AbuseIPDB cards (#1788)

* added bgpview card

* fixed card js spacing

* added AlienVault OTX card js

* updated AbuseIPDB card js

Co-authored-by: wsalusky <william.salusky@verizonmedia.com>

* add key form

* key settings fixes

* add raw edit of integration settings

* redo settings, user can disable, show if global set on config page

* use ini in raw edit mode

* add a home page for integrations

* fix anonymous using db cont3xt data

* pretty forms

* more text

* don't color non-required key setting field

* drag/drop links

* Ability to get/put cont3xt savings

* user.save needs a cb

* make sure user.cont3xt exists

* reorder link groups

* include unordered link groups

* redo link groups

* fix selected link group on reorder

* minor link group style changes

* added cymru

* search tables in integration card

sort dates in integration card (default desc)
fix settings nav item not active when selecting sub page
don't show link group if it doesn't have any links
fix save integration alert being too big
fix long link group name overflowing container

* fix integration table search

truncate long link names in view

* updates to card js for threatstream, shodan, alienvault and virustotal (#1790)

Co-authored-by: wsalusky <william.salusky@verizonmedia.com>

* redo order

* lint

* add defaultSortField and defaultSortDirection to table fields

* refang indicator for links

* support disabled true or 'true'

* only display search for integration tables if there is more than one row

* can disable threatstream

* add startTS and stopTS and document on help page

* initial stab at censys integration for Andy to fix up (#1791)

* initial stab at censys integration for Andy to fix up

* use axios auth

* remove censys package

* lint

Co-authored-by: wsalusky <william.salusky@verizonmedia.com>
Co-authored-by: Andy Wick <andywick@gmail.com>

* fully 0 pad ipv6 addresses

* new ipv6 regex that matches compressed ipv6

* protect cont3xt! only users with cont3xtUser/Admin roles can view
cont3xt

* more error checking

* lint!

* display user defined roles

* add missing User config

* fix large searches getting cut off in results

* toggle array/table values in integration results

* add copy raw and csv string

* can click entire row to toggle table/array visiblity

* share link copy button

* add toggle to skip cache

add button for report (back end not implemented yet)

* c8 cont3xt

* download report in json format

remove general settings tab

* misc ui improvements

name report download
add csv download button for tables
add json download button for integration cards
base64 encode the query parameter
don't flash no link groups configured while loading
add text for creators of link groups

* doable includes if disabled

* reverse skipIntegrations to doIntegrations

* select integrations dropdown

fix issues with selected links not opening all

* continue not return when skipping an integration

* fix query params

use b for base64 encoded, still accept q, but redirect to b
pivot uses b
fix (un)select all integrations removing query

* sort integrations in select dropdown

* url classify requires http?s now

* lint

* fix urlhaus count

* add roles as "role:roleName"

display roles that are assigned to a link group but don't exist

* update for async roles calls

* make hasRole await

* make sure editRoles/viewRoles are arrays

* remove shelljs

* redo since I messed them up

* first pass at using roles and removing createEnabled (#1815)

* first pass at using roles and removing createEnabled

* lint

* first pass at permission -> roles in vue

* forgot new file

* fix tests

don't display settings button on users page if user isn't arkimeAdmin

Co-authored-by: 31453 <elyse.rinne@verizonmedia.com>

* use _allRoles for roles returned to viewer

* fix spelling

* make sure roles are arrays

remove more createEnabled stuffs

* Common users vue (#1818)

* start common users page

* finish users common table

* create user form

* reorganize common vueapp files

add search users api to cont3xt

* add users page to cont3xt

put common user apis in common/user

* fix styles and has-role checks for common users page

* don't let non usersAdmin see users page

* fix ui tests

remove some user tests (need to figure out b-table)

* update jsdoc to include common files

* fix sass warnings

* fix unnecessary comment

* fix expandRoles for digest/header auth

* fix user not being defined in common users page

put timezoneDateString in common filters

* maybe fix undefined timezoneDateString filter

fix cont3xt navbar has-permission -> has-role

* try registering timezoneDateString filter in Users component

check for user before testing for user roles

* expose function for timezoneDateString

* 1 getuser to rule them all, fix createAdmin compatibility (#1820)

* 1 getuser to rule them all, fix createAdmin compatibility

* handle if roles isn't defined

* negate some user permission flags

arkime data removal, hunting, and email search

* link group edit improvements

add separator
add create button between every link
add push to top/bottom button for every link

* addUser can work with cont3xt section

* display roles as title of roles dropdown

* make roles button text smaller

only show + for link groups if on link group settings

* remove role: and sort roles for dropdown title

* put user defined roles first

use slice instead of replace

* add health endpoint to cont3xt

* display single integration values on single line

* scroll long list of roles dropdown

* select fields to search in tables

* search integrations

* add icons to integration settings

* maybe fix users table changing when opening user detail

* fallback for domain registrar creation date

* (un)select all fields for table searching

* add spur infrastructure to ip results

* if vt hash response_code is 0, return 0 count

* allow selecting itypes for link separators

* use _cont3xt object to store app variables (#1822)

* use _cont3xt object to store app variables

* set _cont3xt if undefined in integrations.js

so we don't assume every integration has to set _cont3xt

* remove empty _cont3xt from rdap

* make sure missing _cont3xt object is created

* Cont3xt views (#1824)

* front end for cont3xt views

mocks for backend apis

* es crud for integration views

* add lmdb for views

* put views in settings page

move create view to modal

* misc view improvements

* put time inputs by search bar

misc css fixes/improvements

* make time inputs xs

* cont3xt works with node 16!

* display no views if there are views shared with you but you can't edit
them

* display who shared link group with you

* beautiful ??=

* cont3xtHost support

* link group ui improvements (#1830)

* link group ui improvements

* minor fixes and better link filtering

* make link groups take up all vertical space on cont3xt page

what a pain

* more text to show link group view only

* use just bundle for now

* make no assumptions about link groups

* fix duplicate computed prop on help.vue

* add helmet security to cont3xt (#1837)

* add helmet security to cont3xt

frameguard, hidepoweredby, xssfilter, nosniff, hsts
xsrf cookies protect all modification endpoints
use fetch for common vueapp services

* fix tests and add serverError to common

* put serverError in arkimeUtil

* fix success/error props getting saved with view

* debug cont3xtGet in tests

* maybe see what/where

* hopefully fix tests failing, stupid mac case unsens

Co-authored-by: Andy Wick <andywick@gmail.com>

* check for User stuff, 4.x

* remove iptrie

* add simple arkimeUser role test

* more 8 support

* remove tmpRolesSupport flag

* tests to make sure can't auth as role

* fix paging for users table

condense tables in cont3xt even more
use smaller font for users table in viewer

* search user roles

* add -o to cont3xt, use in tests, fix help in wise

* use arkime_ prefix for usersPrefix by default

* first pass at adding users to parliament (#1845)

* first pass at adding users to parliament

* fix tests when no settings set

* use commonAuth in parliament

hide login/logout
use setup code for common user and password auth

* fix not showing settings when using common auth only

* noauth

* use isAdmin/isUser

hide settings unless isAdmin
hide parliament edit unless isUser
hide issue edit unless isUser

* fix tests

* parliament tests!

Co-authored-by: 31453 <elyse.rinne@verizonmedia.com>

* use generic wording for user page

because some "users" are roles

* fix parliament login not showing settings page until refresh

* Hide passwordSecret, usersElasticsearchAPIKey, usersElasticsearchBasicAuth (#1847)

* change default to 100 per page, add 200, remove 5

* forgot comma

* don't disable roles for lastUsed

* first pass at Users help

* upgrade async version and handle hunt errors better to stop async com… (#1853)

* upgrade async version and handle hunt errors better to stop async complaining

* remove direct portfinder usage

* remove more direct portfinder usage

* express 4.17.3 (#1856)

* add password change to cont3xt settings (#1851)

* add password change to cont3xt settings

* use Auth instead of middleware for regressionTests and noPasswordSecret

* case insensitive mac is silly

* change user password from users page

* fix tests

set regressionTests in auth

* maybe do Auth.initilize correctly

* initialize passport in header mode too

* require usersAdmin to update password for users

* allow user to change own password! duh

Co-authored-by: Andy Wick <andywick@gmail.com>

* remove initial shortcuts sync

fixes #1710

* upgrade sqlite and splunk (#1857)

* moved timebounding help to 1 place, probably needs some clean up

* update viewer endpoints to newest apis (#1858)

* update viewer endpoints to newest apis

* fix test

* cleanup time bounding help

* use _docs for all doc output files

* rename viewer docs output file

Co-authored-by: 31453 <elyse.rinne@verizonmedia.com>
Co-authored-by: William Salusky <125899+WilliamSalusky@users.noreply.github.com>
Co-authored-by: wsalusky <william.salusky@verizonmedia.com>
  • Loading branch information
4 people committed Apr 1, 2022
1 parent eb99ee3 commit f016a6b
Show file tree
Hide file tree
Showing 203 changed files with 17,620 additions and 3,547 deletions.
4 changes: 4 additions & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,7 @@
/viewer/node_modules
/viewer/public
/viewer/version.js
/cont3xt/vueapp/build/
/cont3xt/vueapp/config/
/cont3xt/vueapp/dist/
/cont3xt/node_modules
9 changes: 9 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,12 @@ capture/capture
# tests
tests/GeoLite2*
tests/oui.txt
tests/lmdbCache

# Require force update
tests/cont3xt.ini
tests/test.config.ini
tests/test.config.json

# viewer
viewer/public/CyberChef*.zip
Expand All @@ -47,6 +53,9 @@ parliament/vueapp/dist
# wiseservice
wiseService/vueapp/dist

# cont3xt
cont3xt/vueapp/dist

# Makefiles
Makefile
Makefile.in
Expand Down
2 changes: 1 addition & 1 deletion Makefile.am
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
SUBDIRS = . capture db viewer parliament wiseService release tests
SUBDIRS = . capture db viewer parliament wiseService cont3xt release tests
install-exec-local:
npm ci
cp -pr common @prefix@
Expand Down
58 changes: 55 additions & 3 deletions common.css
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,17 @@ body {

/* MISC -------------------------------------- */
.margin-for-nav {
margin-top: 70px;
margin-top: 70px !important;
}

/* widths */
.w-40 { width: 40% !important; }

/* cursors */
.cursor-help, .help-cursor { cursor: help; }
.cursor-text, .text-cursor { cursor: text; }
.cursor-move, .move-cursor { cursor: move; }
.cursor-grab, .grab-cursor { cursor: grab; }
.cursor-pointer, .pointer-cursor { cursor: pointer; }
.cursor-crosshair, .crosshair-cursor { cursor: crosshair; }

Expand All @@ -52,9 +56,14 @@ body {
.no-overflow-y { overflow-y: hidden; }
.break-word { word-break: break-word; }
.word-break { word-break: break-word; }
.break-all { word-break: break-all; }
.all-break { word-break: break-all; }

.ellipsis { text-overflow: ellipsis; }

/* display */
.display-inline { display: inline; }
.display-inline-block { display: inline-block; }

/* remove underline for anchor tag hover */
.no-decoration { text-decoration: none !important; }
Expand All @@ -78,6 +87,30 @@ body {
border-radius: 3px;
}

/* user roles dropdown scrolls for long lists */
.roles-dropdown ul {
overflow: auto;
max-width: 220px;
max-height: 200px;
}

/* xs input groups */
.input-group-xs > .form-control,
.input-group-xs > .custom-select,
.input-group-xs > .input-group-prepend > .input-group-text,
.input-group-xs > .input-group-append > .input-group-text,
.input-group-xs > .input-group-prepend > .btn,
.input-group-xs > .input-group-append > .btn {
line-height: 1;
padding: 0.25rem;
font-size: 0.825rem;
border-radius: 2px;
}
.input-group-xs > .custom-select,
.input-group-xs > .form-control:not(textarea) {
height: calc(0.825em + 0.5rem + 4px);
}

/* INFO AREA --------------------------------- */
/* displays large text for important information
* note: must contain an inner div with the text
Expand Down Expand Up @@ -212,7 +245,7 @@ body.dark .info-area > div {
}

/* darken pre */
body.dark pre {
body.dark pre:not([class*="text-"]) {
color: white !important;
}

Expand All @@ -238,7 +271,8 @@ body.dark .dropdown-menu {
background-color: #222;
border: 1px solid #333;
}
body.dark .dropdown-menu .dropdown-item {
body.dark .dropdown-menu .dropdown-item,
body.dark .dropdown-menu .b-dropdown-form {
color: #EEE;
}
body.dark .dropdown-menu .dropdown-item:hover,
Expand Down Expand Up @@ -293,6 +327,9 @@ body.dark .table-hover .table-secondary:hover > th,
body.dark .table-hover .table-secondary:hover > td {
background-color: #4F4F4F;
}
body.dark .table-striped tbody tr:nth-of-type(odd) {
background-color: rgba(255, 255, 255, 0.05);
}

/* lighten table text in dark mode */
body.dark table.table {
Expand Down Expand Up @@ -389,6 +426,21 @@ body.dark .pagination .page-item.active > button:focus {
background-color: #111;
}

/* dark modal */
body.dark .modal-content {
color: #EEE;
background-color: #222;
}
body.dark .modal-content .modal-header,
body.dark .modal-content .modal-footer {
background-color: #333;
}

/* dark panel */
body.dark .b-sidebar-body {
color: #EEE;
background-color: #333;
}

/* ANIMATIONS -------------------------------- */
/* for transition groups (any block element)
Expand Down
130 changes: 104 additions & 26 deletions common/arkimeCache.js
Original file line number Diff line number Diff line change
Expand Up @@ -28,25 +28,28 @@ const ArkimeUtil = require('../common/arkimeUtil');
class ArkimeCache {
constructor (options) {
this.cacheSize = parseInt(options.cacheSize ?? 100000);
this.cacheTimeout = parseInt(options.cacheTimeout ?? 24 * 60 * 60);
this.cache = {};
this.cacheTimeout = ArkimeUtil.parseTimeStr(options.cacheTimeout ?? 24 * 60 * 60);
this.cache = LRU({ max: this.cacheSize });
}

// ----------------------------------------------------------------------------
get (query, cb) {
const cache = this.cache[query.typeName];
cb(null, cache ? cache.get(query.value) : undefined);
get (key, cb) {
// promise version
if (!cb) {
return new Promise((resolve, reject) => {
resolve(this.cache.get(key));
});
}

cb(null, this.cache.get(key));
}

// ----------------------------------------------------------------------------
set (query, result) {
let cache = this.cache[query.typeName];
if (!cache) {
cache = this.cache[query.typeName] = LRU({ max: this.cacheSize });
}
cache.set(query.value, result);
set (key, result) {
this.cache.set(key, result);
}

// ----------------------------------------------------------------------------
static createCache (options) {
switch (options.type) {
case 'memory':
Expand All @@ -55,8 +58,10 @@ class ArkimeCache {
return new ArkimeRedisCache(options);
case 'memcached':
return new ArkimeMemcachedCache(options);
case 'lmdb':
return new ArkimeLMDBCache(options);
default:
console.log('Unknown cache type', options.type);
console.log('ERROR - Unknown cache type', options.type);
process.exit(1);
}
};
Expand All @@ -79,15 +84,28 @@ class ArkimeRedisCache extends ArkimeCache {
}

// ----------------------------------------------------------------------------
get (query, cb) {
get (key, cb) {
// Convert promise to cb by calling ourselves
if (!cb) {
return new Promise((resolve, reject) => {
this.get(key, (err, data) => {
if (err) {
reject(err);
} else {
resolve(data);
}
});
});
}

// Check memory cache first
super.get(query, (err, result) => {
super.get(key, (err, result) => {
if (err || result) {
return cb(err, result);
}

// Check redis
this.client.getBuffer(query.typeName + '-' + query.value, (err, reply) => {
this.client.getBuffer(key, (err, reply) => {
if (err || reply === null) {
return cb(null, undefined);
}
Expand All @@ -102,15 +120,15 @@ class ArkimeRedisCache extends ArkimeCache {
bsonResult[source].result = newResult;
}
}
super.set(query.value, bsonResult); // Set memory cache
super.set(key, bsonResult); // Set memory cache
cb(null, bsonResult);
});
});
};

// ----------------------------------------------------------------------------
set (query, result) {
super.set(query, result);
set (key, result) {
super.set(key, result);

let newResult;
if (this.redisFormat === 3) {
Expand All @@ -124,7 +142,7 @@ class ArkimeRedisCache extends ArkimeCache {
}

const data = BSON.serialize(newResult, false, true, false);
this.client.setex(query.typeName + '-' + query.value, this.cacheTimeout, data);
this.client.setex(key, this.cacheTimeout, data);
};
};

Expand All @@ -139,30 +157,90 @@ class ArkimeMemcachedCache extends ArkimeCache {
}

// ----------------------------------------------------------------------------
get (query, cb) {
get (key, cb) {
// Convert promise to cb by calling ourselves
if (!cb) {
return new Promise((resolve, reject) => {
this.get(key, (err, data) => {
if (err) {
reject(err);
} else {
resolve(data);
}
});
});
}

// Check memory cache first
super.get(query, (err, result) => {
super.get(key, (err, result) => {
if (err || result) {
return cb(err, result);
}

// Check memcache
this.client.get(query.typeName + '-' + query.value, (err, reply) => {
this.client.get(key, (err, reply) => {
if (err || reply === null) {
return cb(err, undefined);
}
const bsonResult = BSON.deserialize(reply, { promoteBuffers: true });
super.set(query.value, bsonResult); // Set memory cache
super.set(key, bsonResult); // Set memory cache
cb(null, bsonResult);
});
});
};

// ----------------------------------------------------------------------------
set (query, result) {
super.set(query, result);
set (key, result) {
super.set(key, result);

const data = BSON.serialize(result, false, true, false);
this.client.set(query.typeName + '-' + query.value, data, { expires: this.cacheTimeout }, () => {});
this.client.set(key, data, { expires: this.cacheTimeout }, () => {});
};
};

/******************************************************************************/
// LMDB Cache
/******************************************************************************/
class ArkimeLMDBCache extends ArkimeCache {
constructor (options) {
super(options);

// eslint-disable-next-line no-shadow
const { open } = require('lmdb-store');

const path = options.getConfig('lmdbDir');

if (typeof (path) !== 'string') {
console.log('ERROR - lmdbDir must be set');
process.exit(1);
}

try {
this.store = open({
path: path,
compression: true
});
} catch (err) {
console.log('ERROR -', err);
process.exit(1);
}
}

// ----------------------------------------------------------------------------
get (key, cb) {
if (!cb) {
return this.store.get(key);
}

return new Promise((resolve, reject) => {
this.store.get(key)
.then(data => cb(null, data))
.catch(err => cb(err, null));
});
}

// ----------------------------------------------------------------------------
set (key, result) {
this.store.put(key, result);
};
};
Loading

0 comments on commit f016a6b

Please sign in to comment.