Require common auth for Parliament #2349
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
add parliamentHost remove auth setup code update changelog
awick
approved these changes
Aug 2, 2023
awick
added a commit
that referenced
this pull request
Oct 24, 2023
* start 5 branch * try again * Remove old users api and switch tests to new (#2290) * Remove old users api and switch tests to new * user assignment tests * remove old views, cron, notifiers apis (#2291) * remove old apis (#2292) * remove old apis * remove title and filesize apis totally since not used * cleaned up stats/esadmin/connections/history * left some .json, looks like parliament might stil use and I like ;) * lint * removed old aip for hunts, shortcuts, and most of sessions (#2294) * removed old aip for hunts, shortcuts, and most of sessions * lint * Convert most moloch's to arkime in capture (#2295) * remove x-moloch-auth (#2296) * better capture defaults: zstd, s3 gap, dedup (#2297) * better capture defaults: zstd, s3 gap, dedup s3Compression/simpleCompression now default to zstd s3WriteGzip removed, use s3Compression=gzip s3GapPacketPos defaults to TRUE enablePacketDedup defaults to TRUE * disable packet dedup for tests * added authMode and basic auth (#2299) * backend of /api/users.csv with tests #2281 (#2300) * backend of /api/users.csv with tests #2281 * Made a User.#apiGetUsersCommon * download users csv button --------- Co-authored-by: 31453 <ecrinne@gmail.com> * remove user.views and user.notifiers (#2301) * ViewerUtils.noCache -> ArkimeUtil.noCache (#2302) * use userNameHeader if authMode isn't set and warn (#2308) * Can now set up ops with override-ip (#2310) * Can now set up ops with override-ip * load override-ip after plugins * added overrideIpFiles setting (#2312) * Packet ip files (#2314) * added packetIpFiles setting * change setting to packetDropIpFiles * fix changelog * change to overrideIpsFiles and packetDropIpsFiles * first pass of config config impl (#2316) This change makes a new ArkimeConfig common class on the javascript side that knows how to fetch configuration from files, http, os/es, redis in both ini and json format. wise/cont3xt/viewer have been moved to use it, although viewer (plus tools) still use the config.js wrapper. On the capture side config is supported from the same excluding redis. Also added a some tests to make sure we can load the various formats. * UI reshuffle cont3xt (#2354) * ui reshuffle * settings panes scroll individually * highlight hovered indicator nodes and fix grid height * improve link groups and integration panel hover * link groups take up full width of panel (and no longer use dynamic arrangement :'O) * link group collapse/expand all button behavior fixed (and disabled when no link groups to apply to) * integration panel hover timeout is cancelled when leaving stub early * integration panel fixed width * the integration panel's width was being fought by the results content, so it now uses min-width * minor changes to linkgroup appearance * fix shadow cut-off and adjust welcome screen * also modified go-to-top button to work in new card panel * linkGroupsPanelOpen moved to localStorage * change keybind to toggle link groups * link groups reflect active indicator * reduce unnecessary node padding * time-range input info hint will no longer wrap * rdap name is now a badge to conserve space * add panel for link groups * fix clipping and overflow issues * fixed overview selector being mis-aligned and clipped (using right alignment) * multi-badges are clipped to their parent container, not the panel—as before * constrain badges to node parent * Require common auth for Parliament (#2349) * work off dev5 instead of main * trying to fix tests * fix tests and user arkimeutil servererror * upgrade parliament version to 4 add parliamentHost remove auth setup code update changelog * fix missing token error * add color counts for indicator nodes (#2358) * shorten link groups start stop (#2359) * shorten link groups start stop * allow link group endDate interpolation * allow either ${end...} or ${stop...} in link interpolation to support UI now saying "End" * updated help page to show ${end...} instead of ${stop...} * updated link group link tooltip * regex OR full terms * better search-ability :) * bulk search (#2361) * bulk search * remove single-search root * purpose init sends all indicators queried * send all indicators queried in purpose:'init' & the total between all their integrations * don't send purpose:'link' for root indicators * update tests * folding and bulk navigation (#2362) * folding and bulk navigation * shift minus and keybind for view panel * focus navigated indicator node (#2363) * using hjkl moves focus point to node so a tab will bring you to the first integration button * and set to overview when keyboard navigating (this wasn't being done, so we'd stay on whatever the last integration was when moving with the keyboard) * update to 5.0.0 * fix overview selector jitter (#2365) * some fun css so the hotkey O does not shift the rest of the UI * split on /[ |,\t]/ (#2376) * fix stale table data on overview cards (#2379) pixel perfect tidbits with severity indicators put overview button with integration buttons split overview button so clicking it once sets overview and clicking caret opens dropdown * add vars for arrays of indicators in links (#2380) * add vars for arrays of indicators in links * use json to define array placeholder in links * match or ${array, * use common config for parliament (#2377) * use common config for parliament add parliament ini (file/port/cert/key needed here instead of command line args) use upgrade to write ini based on commonAuth settings remove commonAuth from UI remove commonAuth update API update parliament help * fix parliament tests * don't use anonymous default * add another link to parliament settings page * add parliament release stuff * right parliament json location * support --elasticsearch with parliament * New elasticsearch integration (#2382) * New elasticsearch integration [elasticsearch:foo] name=Foo index=foo* itypes=domain queryField=domain url=http://localhost:9200 * support opensearch icon, update es icon * allow for multiple array placeholders in links (#2383) * allow for multiple array placeholders in links add more help fix help page not scrolling * do things the millenial way * config file can now have viewRoles per integration (#2385) * Cleaned up how config is done - switch to ArkimeConfig if you just want to get things from config file for integrations - if configName is set use that for config file and user config - if section is set use that for config file and name for user config * When viewRoles in config file is set for integration check that for both list and run operations * merge fix * merge parliament/viewer notifiers (#2387) * merge parliament/viewer notifiers put parliament notifiers in users es add alerts and on props to viewer notifiers add common notifier component to share use fetch instead of axios * fix parliament tests add more ui notifier tests fix some perl ugly use some ??= * whoops * stupid lint * Added ArkimeUtil.checkArkimeSchemaVersion --------- Co-authored-by: Andy Wick <andywick@amazon.com> * start saving a arkimeDbVersion also so we can switch for 6 (#2391) * support old check for default section * add clusters dropdown menu to stats page (#2392) * add clusters dropdown menu to stats page * first pass at supporting cluster for apiStats calls * fix statsGraph for multi simpleGather requires req.query.cluster be set * add cluster param for es tabs select one for es tabs that's "smart" deselect all but first cluster when switching to es tab but don't update the route params so we can go back to capture tabs display notification if removing selected clusters on es tabs display cluster selection note when no results * misc stats cluster dropdown fixes fix count for cluster dropdown when "none" put the dropdown on the right always move search to navbar add notification that you have no clusters selected and don't issue query * more fixes, nodemon multies, LRU library * esAdmin shows up and kind of works * fix a test, start files/history support * send cluster param on esadmin reqs abstract out validating cluster selection * fix multiple stats calls * esadmin tab should work better * add cluster param to files req * fix link * add cluster param to history api req * fix history error display * don't set cluster param in non-multiviewer mode * fix ui tests * fixes #2393, put quotes around both gte/lte values for epoch_second to work * remove req.query.cluster if not multiES * support cluster with include/exclude * add cluster param for include/exclude make sure post includes body before params everywhere * add cluster param to delete index * add cluster param to shrink get esstats * more cluster fun * fix shrink stuffs * shrink fixes * task fixes * fix tests * start switch to remoteCluster for send stuff * use remoteCluster for send api * redo _skipReplace to only set for some apis * fix tests * more tests * It's almost like linting should just happen --------- Co-authored-by: Andy Wick <andywick@amazon.com> * TLS Cert validity with negative values (#2390) * Negative validity period for TLS certs * Update tests * Correct remaining calculations * Correct tests * Fix overflow in subtraction --------- Co-authored-by: Owen McGill <sqooba_owen@knight.zoo> * Move http server creation and listen to common function (#2396) * Move http server creation and listen to common function * improve comments * fix some random compiler warning with clang 16 (#2397) * Remove Parliament JSON (#2395) * create parliament index add parliament to index on upgrade * remove parliament json! update apis to use db add parliament to index by name use uuid to create unique ids for groups and clusters map new cluster ids to issues json if parliament json doesn't exist create stub * fix tests and put parliament class in parliament server * misc parliment fixes use qq in tests lowercase parliament as default name add dynamic string template backup, repair, restore parliament index specific endpoint for updating parliament order use copy of parliament before deleting settings object pass in less stuff to upgrade * db.pl init delete/create parliament, document clean * fix parliament tests and don't overwrite/delete "parliament", use "parliamenttest" instead * move most of api defs to class * add dynamic mapping to parliament not views * more fixes add --name option replace parliament cache when doing setParliament type checking --------- Co-authored-by: Andy Wick <andywick@amazon.com> * move parliament to ArkimeUtil.createHttpServer (#2398) * move parliament to ArkimeUtil.createHttpServer * add common to parliament lint and fix * Auth init refactor (#2399) * Auth.init has section and can config itself * fix passwordSecret defaut location * commonize prefix formatting and for esproxy tests (#2400) * commonize prefix formatting and for esproxy tests * lint * sane debug and adminRole (#2401) * sane debug and adminRole - debug only lives in ArkimeConfig, Config, WISESourceAPI now - moved adminRole and check from Util to Auth, maybe YT is right * change middleware location * cppcheck fixes * Http request response fields (#2402) * redid http.request/response expression * fix wise test * count new fields * delete old expression and add new with reverse alias * wise/tagger must use official names and not aliases * add editRoles to shortcuts, views, and queries (#2404) * add editRoles to shortcuts, views, and queries use middleware to check for resource access added tooltip to dropdown for when roles are selected but there are 2 dropdowns fix bad logic in hasRole function on client (how did that ever work!?) remove roles columns from settings tables update docs fixed typos * update min db version * fix tests add test for bad logic in hasRole add tests for editRoles * add editRoles tests for shortcuts * require 4.x for upgrade and don't rerun everything if already on 79 (#2406) * Transfer Resource Ownership (#2407) * transfer ownership of views * make error/success messages more obvious on viewer settings page * transfer shortcuts ownership * transfer periodic queries improve transfer modal improve checks for transfer userId * transfer link groups * Generalize functions function to set owner function to check resource access for update/delete change ownership added to views and overviews update tests update changelog test transfer resource component * use param for cron key * don't need req.body.key for crons fix cont3xt views showing no views message using see all option * shorten open matches cron button * make common form auth ui (#2408) * make common form auth ui * WIP - added form auth, fix parliament missing json file To test locally you'll need authMode=digest authCookieSecure=false * fix wiseService debug and config default log * fix to use basePath * use simplest form possible but also make it pretty * remove unnecessary changes * remove unnecessary userservice function * change cookiename to ARKIME-SID * add logo and autofocus input * add logout button to parliament * add logout button to all apps if authmode is form remove logout text in button size the button dynamically from parent * supply base path to logout * use Auth.mode --------- Co-authored-by: Andy Wick <andywick@amazon.com> * First pass of ja4 (#2409) * First pass of ja4 * show JA4 * fix ja4 length * if JA4_b or JA4_c are empty output 0s instead of sha of empty * fix ja3/ja4 for smtp * fix some grease value issues * add ja4 test * ES maxResults/includeIndex/includeId settings * always build zstd 1.5.5 (except arch) (#2412) * glib 2.78.0, nghttp2 1.56.0, curl 8.3.0, maxmind 1.7.1, yara 4.3.2 (#2413) * zstd 1.5.5, glib 2.78.0, nghttp2 1.56.0, curl 8.3.0, maxmind 1.7.1, yara 4.3.2 * fix nghttp2 include * yara moved lib, build runs capture --version * try older version of glib2 * lower glib, remove -d zstd for non arch * node v18.18.0 except Centos 7 build which still uses v16 (#2414) * node v18.18.0 except Centos 7 build which still uses v16 * removed npm config set user root * upgrade to nodemon3 (#2415) * run viewer tests on arch (#2416) * Oldnodeubuntu18 (#2417) * Have to use old node on ubuntu 18 * minor update to call rebulid * upgrade express (#2418) upgrade express and finally fix flaky hunt test * yaml config file format, json/yaml array support (#2419) * yaml config file format, json/yaml array support * increase buf size * added js-yaml * always build ZSTD for 5.0 * start of reformatting and use unit8_t (#2422) * new HASH_FORALL v2 macros where code isn't passed in (#2427) * Fix NPM Warnings (#2426) * upgrade notifme * upgrade packages * upgrade webpack to fix min build error * show generic eshealth error if json is returned (#2430) * upgrade to aws v3, attempt 2 (#2431) * Add SPDX-License to all of capture (#2423) * highlight configexit msg, don't pass in ending \n (#2432) * finally fix db fields having to be created in order (#2433) * redo config AGAIN, central logger w/ config (#2434) * Use AStyle to reformat (#2435) * fix elastic/maxmind from config change, added tests (#2436) * Wise cont3xt links (#2438) * add wise and cont3xt urls to parliament settings display them in the navbar if configured * allow urls to be empty * remove max attr on url inputs * first pass at a csv/json integration that loads from file/redis (#2437) * first pass at a csv/json integration that loads from file/redis Elyse will need to redo the icons * reload files on change, url/redis periodic reload * remove leading 0s from ipv4 * fix reload check, cleanup fs.watch * fix icon location and default card * misc beauty changes better icons longer text for overview selector don't stretch buttons if there's only a few --------- Co-authored-by: 31453 <ecrinne@gmail.com> * redis cont3xt source (#2441) * support building on alpine (#2445) * support building on alpine * don't map gb2312 to CP936, try building alpine * fixed alpine image * added --no-cache and need to install curl/bash * need sudo too * need git * new cert callback, remove bool from viewer/config.js (#2446) * upgrade node and nghttp2 (#2447) * use unofficial node 18 on Centos 7 and Ubuntu 18 (#2448) * use unofficial node 18 on Centos 7 and Ubuntu 18 * pr on 18 * move ui testing to 8/9 (#2449) * install libzstd-dev on ubuntu22 * switch builds to use 5.0.0-2 (#2450) * switch builds to use 5.0.0-2 can now specify what branch to use to build containers * no pfring for centos 7 * initial arkime integration (#2452) * Increase simpleCompressionBlockSize default to 64000 (#2453) * move all the js code to SPDX-License (#2454) * moloch2arkime in viewer (#2455) * tests moloch->arkime (#2456) * fix warnings and replace ~~ usage with grep (#2457) * rename hideMolochNodes to hideArkimeNodes (#2458) * add license to vue files (#2459) * upgrade babel stuffs to fix @babel/traverse vuln (#2461) * remove code references to moloch from viewer vueapp (#2462) except session/local storage * basic+form, and basic+oidc added (#2463) * fix info to show all indices * shortcut an error if len is more than 4 bytes (#2464) * allow db 77 to upgrade * fix parliament upgrade failing for bad mapping (#2468) * change to main before merging into main --------- Co-authored-by: 31453 <ecrinne@gmail.com> Co-authored-by: Toby Salusky <50775507+TobySalusky@users.noreply.github.com> Co-authored-by: Owen McGill <mcgillowen@users.noreply.github.com> Co-authored-by: Owen McGill <sqooba_owen@knight.zoo>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.