diff --git a/README.md b/README.md index 87c8d03b29..bca5cccfc9 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,17 @@ ## Grapl -Grapl is a Graph Platform for Detection and Response. +Grapl is a Graph Platform for Detection and Response with a focus on helping Detection Engineers stop fighting their data and start working with it. At its core, Grapl leverages graph data structures to ensure that you can and connect your data efficiently, model attacker behaviors, and easily expand suspicious behaviors to encompass a full attack scope. For a more in depth overview of Grapl, [read this](https://insanitybit.github.io/2019/03/09/grapl). -In short, Grapl will take raw logs, convert them into graphs, and merge those graphs into a Master Graph. It will then orchestrate the execution of your attack signatures and provide tools for performing your investigations. +Essentially, Grapl will take raw logs, convert them into graphs, and merge those graphs into a Master Graph. It will then orchestrate the execution of your attack signatures, and provide tools for performing your investigations. Grapl supports nodes for: -- Processes (Beta) -- Files (Beta) -- Networking (Alpha) +- Processes +- Files +- Networking +- Plugin nodes, which can be used to arbitrarily extend the graph and currently parses Sysmon logs or a generic JSON log format to generate these graphs. @@ -24,31 +25,54 @@ and currently parses Sysmon logs or a generic JSON log format to generate these If you’re familiar with log sources like Sysmon, one of the best features is that processes are given identities. Grapl applies the same concept but for any supported log type, taking psuedo identifiers such as process ids and discerning canonical identities. +Grapl then combines this identity concept with its graph approach, making it easy to reason about entities and their behaviors. Further, this identity property means that Grapl stores only unique information from your logs, meaning that your data storage grows sublinear to the log volume. + This cuts down on storage costs and gives you central locations to view your data, as opposed to having it spread across thousands of logs. As an example, given a process’s canonical identifier you can view all of the information for it by selecting the node. ![](https://d2mxuefqeaa7sj.cloudfront.net/s_7CBC3A8B36A73886DC59F4792258C821D6717C3DB02DA354DE68418C9DCF5C29_1553026555668_image.png) -**Analyzers (Beta)** +**Analyzers** Analyzers are your attacker signatures. They’re Python modules, deployed to Grapl’s S3 bucket, that are orchestrated to execute upon changes to grapl’s Master Graph. -Analyzers execute in realtime as the master graph is updated. +Rather than analyzers attempting to determine a binary "Good" or "Bad" value for attack behaviors Grapl leverges a concept of Risk, and then automatically correlates risks to surface the riskiest parts of your environment. + +Analyzers execute in realtime as the master graph is updated, using constant time operations. Grapl's Analyzer harness will automatically batch, parallelize, and optimize your queries. By leveraging constant time and sublinear operations Grapl ensures that as your organization grows, and as your data volume grows with it, you can still rely on your queries executing efficiently. -Grapl provides an analyzer library (alpha) so that you can write attacker signatures using pure Python. See this [repo for examples](https://github.com/insanitybit/grapl-analyzers). +Grapl provides an analyzer library so that you can write attacker signatures using pure Python. See this [repo for examples](https://github.com/insanitybit/grapl-analyzers). Here is a brief example of how to detect a suspicious execution of `svchost.exe`, ```python - valid_parents = get_svchost_valid_parents() - p = ( - ProcessQuery() - .with_process_name(eq=valid_parents) - .with_children( - ProcessQuery().with_process_name(eq="svchost.exe") +class SuspiciousSvchost(Analyzer): + + def get_queries(self) -> OneOrMany[ProcessQuery]: + invalid_parents = [ + Not("services.exe"), + Not("smss.exe"), + Not("ngentask.exe"), + Not("userinit.exe"), + Not("GoogleUpdate.exe"), + Not("conhost.exe"), + Not("MpCmdRun.exe"), + ] + + return ( + ProcessQuery() + .with_process_name(eq=invalid_parents) + .with_children( + ProcessQuery().with_process_name(eq="svchost.exe") + ) ) - .query_first(client, contains_node_key=process.node_key) - ) + def on_response(self, response: ProcessView, output: Any): + output.send( + ExecutionHit( + analyzer_name="Suspicious svchost", + node_view=response, + risk_score=75, + ) + ) ``` Keeping your analyzers in code means you can: @@ -56,20 +80,16 @@ Keeping your analyzers in code means you can: - Write tests, integrate into CI - Build abstractions, reuse logic, and generally follow best practices for maintaining software -**Engagements (alpha)** - -Grapl provides a tool for investigations called an Engagement. Engagements are an isolated graph representing a subgraph that your analyzers have deemed suspicious. - -Using AWS Sagemaker hosted Jupyter Notebooks, Grapl will (soon) provide a Python library for interacting with the Engagement Graph, allowing you to pivot quickly and maintain a record of your investigation in code. - - -![](https://d2mxuefqeaa7sj.cloudfront.net/s_7CBC3A8B36A73886DC59F4792258C821D6717C3DB02DA354DE68418C9DCF5C29_1553037156946_file.png) +Check out Grapl's [analyzer deployer plugin](https://github.com/insanitybit/grapl-analyzer-deployer) to see how you can keep your analyzers in a git repo that automatically deploys them upon a push to master. +**Engagements** -Grapl provides a live updating view of the engagement graph as you interact with it in the notebook, currently in alpha. +Grapl provides a tool for investigations called an Engagement. Engagements are an isolated graph representing a subgraph that your analyzers have deemed suspicious. +Using AWS Sagemaker hosted Jupyter Notebooks and Grapl's provided Python library you can expand out any suspicious subgraph to encompass the full scope of an attack. +As you expand the attack scope with your Jupyter notebook the Engagement Graph will update, visually representing the attack scope. -![](https://raw.githubusercontent.com/insanitybit/grapl/master/images/engagement.gif) +![](https://s3.amazonaws.com/media-p.slid.es/uploads/650602/images/6646682/Screenshot_from_2019-10-11_20-24-34.png) **Event Driven and Extendable** @@ -77,12 +97,7 @@ Grapl was built to be extended - no service can satisfy every organization’s n This makes Grapl trivial to extend or integrate into your existing services. -![](https://d2mxuefqeaa7sj.cloudfront.net/s_7CBC3A8B36A73886DC59F4792258C821D6717C3DB02DA354DE68418C9DCF5C29_1553040182040_file.png) - - - -![](https://d2mxuefqeaa7sj.cloudfront.net/s_7CBC3A8B36A73886DC59F4792258C821D6717C3DB02DA354DE68418C9DCF5C29_1553040197703_file.png) - +Grapl also provides a Plugin system, currently in beta, that allows you to expand the platforms capabilities - adding custom nodes and querying capabilities. ## Setup @@ -111,6 +126,12 @@ It will require confirming some changes to security groups, and will take a few This will give you a Grapl setup that’s adequate for testing out the service. +At this point you just need to provision the Graph databases. You can use the Graph Provision notebook in this repo, and +the newly created 'engagement' notebook in your AWS account. + +![](https://s3.amazonaws.com/media-p.slid.es/uploads/650602/images/6396963/Screenshot_from_2019-07-27_22-27-35.png) + + You can send some test data up to the service by going to the root of the grapl repo and calling: `python ./gen-raw-logs.py `. diff --git a/engagement_ux/engagement_view/index.html b/engagement_ux/engagement_view/index.html index d89dcd0a98..435f4c66d8 100644 --- a/engagement_ux/engagement_view/index.html +++ b/engagement_ux/engagement_view/index.html @@ -18,6 +18,7 @@

Lenses

+ diff --git a/engagement_ux/engagement_view/index.js b/engagement_ux/engagement_view/index.js index 2f580ddf79..bb09b1efef 100644 --- a/engagement_ux/engagement_view/index.js +++ b/engagement_ux/engagement_view/index.js @@ -2,7 +2,7 @@ console.log('Loaded index.js'); -const engagement_edge = ""; +const engagement_edge = "https://jzfee2ecp8.execute-api.us-east-1.amazonaws.com/prod/"; console.log(`Connecting to ${engagement_edge}`); @@ -25,27 +25,24 @@ const nodeToTable = (lens) => { header += `score`; header += `link`; - output += `${lens.lens}>`; - output += `${lens.score}>`; + output += `${lens.lens}`; + output += `${lens.score}`; // output += `link>`; - output += `link>`; - + output += `link`; return `${header}` + `${output}`; }; -const getLensesLoop = () => { - -}; - -document.addEventListener('DOMContentLoaded', async (event) => { - console.log('DOMContentLoaded'); - +const getLensesLoop = async () => { const lenses = (await getLenses()).lenses; console.log(lenses); if (lenses.length === 0) { console.log("No active lenses"); + + setTimeout(async () => { + await getLensesLoop(); + }, 1000); return } @@ -59,10 +56,18 @@ document.addEventListener('DOMContentLoaded', async (event) => { } // Sort the lenses by their score lensRows.sort((row_a, row_b) => { - return row_a.score - row_b.score + return row_a.score - row_b.score }); - const lensRowsStr = lensRows.join("") + const lensRowsStr = lensRows.join(""); lenseTable.innerHTML = `${lensRowsStr}
`; + setTimeout(async () => { + await getLensesLoop(); + }, 1000) +}; + +document.addEventListener('DOMContentLoaded', async (event) => { + console.log('DOMContentLoaded'); + getLensesLoop(); }); \ No newline at end of file diff --git a/engagement_ux/engagement_view/lens.html b/engagement_ux/engagement_view/lens.html index cba65fcf9e..70d3c45380 100644 --- a/engagement_ux/engagement_view/lens.html +++ b/engagement_ux/engagement_view/lens.html @@ -8,17 +8,19 @@ - - - - -

Lens

- +

Lens

+
+
+ + - - + + + + + diff --git a/engagement_ux/engagement_view/lens.js b/engagement_ux/engagement_view/lens.js index 3b1d96cc2a..03de5440f3 100644 --- a/engagement_ux/engagement_view/lens.js +++ b/engagement_ux/engagement_view/lens.js @@ -1,7 +1,7 @@ // Stylesheets console.log('entry.js init'); -const engagement_edge = ""; +const engagement_edge = "https://jzfee2ecp8.execute-api.us-east-1.amazonaws.com/prod/"; if (engagement_edge.length === 0) { console.assert("Engagement Edge URL can not be empty. Run build.sh"); @@ -9,79 +9,579 @@ if (engagement_edge.length === 0) { console.log(`Connecting to ${engagement_edge}`); -class GraphManager { - constructor(graph) { - this.canvas = d3.select('#network'); - this.width = this.canvas.attr('width'); - this.height = this.canvas.attr('height'); - this.ctx = this.canvas.node().getContext('2d'); - this.r = 20; - this.color = d3.scaleOrdinal(d3.schemeCategory10); - this.simulation = d3.forceSimulation() - .force("x", d3.forceX(this.width/2)) - .force("y", d3.forceY(this.height/2)) - .force('collide', d3.forceCollide(this.r * 4)) - // .force('charge', d3.forceManyBody() - // .strength(-40)) - .force('link', d3.forceLink() - .id(d => d.uid)); - - - this.simulation.nodes(graph.nodes); - - this.simulation.force('link') - .links(graph.links); - - this.simulation.on('tick', () => { - this.update(); +const BKDRHash = (str) => { + const seed = 131; + const seed2 = 137; + let hash = 0; + // make hash more sensitive for short string like 'a', 'b', 'c' + str += 'x'; + // Note: Number.MAX_SAFE_INTEGER equals 9007199254740991 + const MAX_SAFE_INTEGER = parseInt(9007199254740991 / seed2); + for(let i = 0; i < str.length; i++) { + if(hash > MAX_SAFE_INTEGER) { + hash = parseInt(hash / seed2); + } + hash = hash * seed + str.charCodeAt(i); + } + return hash; +}; + + +/** + * Convert HSL to RGB + * + * @see {@link http://zh.wikipedia.org/wiki/HSL和HSV色彩空间} for further information. + * @param {Number} H Hue ∈ [0, 360) + * @param {Number} S Saturation ∈ [0, 1] + * @param {Number} L Lightness ∈ [0, 1] + * @returns {Array} R, G, B ∈ [0, 255] + */ +const HSL2RGB = (H, S, L) => { + H /= 360; + + const q = L < 0.5 ? L * (1 + S) : L + S - L * S; + const p = 2 * L - q; + + return [H + 1/3, H, H - 1/3].map((color) => { + if(color < 0) { + color++; + } + if(color > 1) { + color--; + } + if(color < 1/6) { + color = p + (q - p) * 6 * color; + } else if(color < 0.5) { + color = q; + } else if(color < 2/3) { + color = p + (q - p) * 6 * (2/3 - color); + } else { + color = p; + } + return Math.round(color * 255); + }); +}; + +const isArray = (o) => { + return Object.prototype.toString.call(o) === '[object Array]'; +}; + +/** + * Color Hash Class + * + * @class + */ +const ColorHash = function(options) { + options = options || {}; + + const LS = [options.lightness, options.saturation].map((param) => { + param = param || [0.35, 0.5, 0.65]; // note that 3 is a prime + return isArray(param) ? param.concat() : [param]; + }); + + this.L = LS[0]; + this.S = LS[1]; + + if (typeof options.hue === 'number') { + options.hue = {min: options.hue, max: options.hue}; + } + if (typeof options.hue === 'object' && !isArray(options.hue)) { + options.hue = [options.hue]; + } + if (typeof options.hue === 'undefined') { + options.hue = []; + } + this.hueRanges = options.hue.map(function (range) { + return { + min: typeof range.min === 'undefined' ? 0 : range.min, + max: typeof range.max === 'undefined' ? 360: range.max + }; + }); + + this.hash = options.hash || BKDRHash; +}; + +/** + * Returns the hash in [h, s, l]. + * Note that H ∈ [0, 360); S ∈ [0, 1]; L ∈ [0, 1]; + * + * @param {String} str string to hash + * @returns {Array} [h, s, l] + */ +ColorHash.prototype.hsl = function(str) { + let H, S, L; + let hash = this.hash(str); + + if (this.hueRanges.length) { + const range = this.hueRanges[hash % this.hueRanges.length]; + const hueResolution = 727; // note that 727 is a prime + H = ((hash / this.hueRanges.length) % hueResolution) * (range.max - range.min) / hueResolution + range.min; + } else { + H = hash % 359; // note that 359 is a prime + } + hash = parseInt(hash / 360); + S = this.S[hash % this.S.length]; + hash = parseInt(hash / this.S.length); + L = this.L[hash % this.L.length]; + + return [H, S, L]; +}; + +/** + * Returns the hash in [r, g, b]. + * Note that R, G, B ∈ [0, 255] + * + * @param {String} str string to hash + * @returns {Array} [r, g, b] + */ +ColorHash.prototype.rgb = function(str) { + const hsl = this.hsl(str); + return HSL2RGB.apply(this, hsl); +}; + + + +const graph3d = (elem) => { + const viz = ForceGraph3D()(elem) + .enableNodeDrag(true) + .onNodeHover(node => elem.style.cursor = node ? 'pointer' : null) + .graphData({nodes: [], links: []}) + .nodeLabel(node => node.nodeLabel) + .linkCurvature('curvature') + .nodeAutoColorBy('nodeType') + .linkThreeObjectExtend(true) + .nodeThreeObjectExtend(true) + .linkOpacity(0.5) + .linkDirectionalArrowLength(6) + .linkDirectionalArrowRelPos(1.05) + .linkThreeObject(link => { + const sprite = new SpriteText(mapLabel(link.label)); + sprite.color = 'cyan'; + sprite.textHeight = 3; + + return sprite; + }) + .linkPositionUpdate((sprite, { start, end }) => { + const middlePos = Object.assign(...['x', 'y', 'z'].map(c => ({ + [c]: start[c] + (end[c] - start[c]) / 2 // calc middle point + }))); + // Position sprite + Object.assign(sprite.position, middlePos); + }) + .nodeThreeObject(node => { + // use a sphere as a drag handle + const obj = new THREE.Mesh( + new THREE.SphereGeometry(4), + new THREE.MeshBasicMaterial({ depthWrite: false, transparent: false, opacity: 1 }) + ); + // add text sprite as child + + const sprite = new SpriteText(node.nodeLabel); + + sprite.color = 'red'; + sprite.textHeight = 4; + obj.add(sprite); + return obj; }); - this.canvas - .call(d3.drag() - .container(this.canvas.node()) - .subject(this.dragsubject) - .on('start', this.dragstarted) - .on('drag', this.dragged) - .on('end', this.dragended)); + // viz.d3Force('charge').strength(-220); + return viz +}; + +const mapLabel = (label) => { + if (label === 'children') { + return 'executed' + } + return label +}; + +const percentToColor = (percentile) => { + const hue = (100 - percentile) * 40 / 100; + + return `hsl(${hue}, 100%, 50%)`; +}; + +const calcNodeRiskPercentile = (_nodeRisk, _allRisks) => { + let nodeRisk = _nodeRisk; + if (typeof _nodeRisk === 'object') { + nodeRisk = _nodeRisk.risk; + } + const allRisks = _allRisks + .map(n => n || 0) + .sort((a, b) => a - b); - this.tooltip = d3.select('body') - .append('div') - .style('position', 'absolute') - .style('z-index', '100') - .html('Click a node to see its attributes'); + if (nodeRisk === undefined || nodeRisk === 0 || allRisks.length === 0) { + return 0 + } - this.graph = graph; + let riskIndex = 0; + for (const risk of allRisks) { + if (nodeRisk >= risk) { + riskIndex += 1; + } + } + + return Math.floor((riskIndex / allRisks.length) * 100) +}; + +const nodeSize = (node, Graph) => { + const nodes = [...Graph.graphData().nodes].map(node => node.risk); + const riskPercentile = calcNodeRiskPercentile(node.risk, nodes); + + if (riskPercentile >= 75) { + return 10 + } else if (riskPercentile >= 50) { + return 8 + } else if (riskPercentile >= 25) { + return 6 + } else { + return 4 + } +}; + +const riskColor = (node, Graph, colorHash) => { + const nodes = [...Graph.graphData().nodes].map(node => node.risk); + + const riskPercentile = calcNodeRiskPercentile(node.risk, nodes); + + if (riskPercentile === 0) { + const nodeColors = calcNodeRgb(node, colorHash); + return `rgba(${nodeColors[0]}, ${nodeColors[1]}, ${nodeColors[2]}, 1)`; + } + + return percentToColor(riskPercentile); +}; + +const calcLinkRisk = (link, Graph) => { + let srcNode = findNode(link.source, Graph.graphData().nodes) + || findNode(link.source.name, Graph.graphData().nodes); + let dstNode = findNode(link.target, Graph.graphData().nodes) + || findNode(link.target.name, Graph.graphData().nodes); + + if (srcNode === null) { + srcNode = {risk: 0} + } + + if (dstNode === null) { + dstNode = {risk: 0} + } + + const srcRisk = srcNode.risk || 0; + const dstRisk = dstNode.risk || 0; + + return Math.round((srcRisk + dstRisk) / 2) +}; + +const calcLinkRiskPercentile = (link, Graph) => { + const linkRisk = calcLinkRisk(link, Graph); + const nodes = [...Graph.graphData().nodes].map(node => node.risk); + + return calcNodeRiskPercentile(linkRisk, nodes); +}; + +const calcLinkParticleWidth = (link, Graph) => { + const linkRiskPercentile = calcLinkRiskPercentile(link, Graph); + if (linkRiskPercentile >= 75) { + return 8 + } else if (linkRiskPercentile >= 50) { + return 7 + } else if (linkRiskPercentile >= 25) { + return 6 + } else if (linkRiskPercentile >= 0) { + return 5 + } else { + return 4 + } +}; + +const calcLinkColor = (link, Graph) => { + const risk = calcLinkRiskPercentile(link, Graph); + if (risk === 0) {return undefined} + return percentToColor(risk); +}; + +const calcNodeRgb = (node, colorHash) => { + if (node.nodeType === 'Process') { + return [50, 153, 169] + } + + if (node.nodeType === 'File') { + return [89, 180, 39] + } + + return colorHash.rgb(node.nodeType) +} + +const sanitizeHTML = (str) => { + const temp = document.createElement('div'); + temp.textContent = str; + return temp.innerHTML; +}; + +const findNode = (id, nodes) => { + for (const node of (nodes || [])) { + if (node.id === id) { + return node + } + } + return null +}; + +/* +* Determines where the DirectionalArrow lies on the edge, based on +* the inferred size of the target node (where size is itself determined +* by risk) +* */ +const calcLinkDirectionalArrowRelPos = (link, Graph) => { + const node = findNode(link.target, Graph.graphData().nodes) + || findNode(link.target.name, Graph.graphData().nodes); + + if (node === null || node.risk === 0) { + return 1.0 + } + const nodes = [...Graph.graphData().nodes].map(node => node.risk); + const riskPercentile = calcNodeRiskPercentile(node.risk, nodes); + + if (riskPercentile === 0) {return 1.0} + + if (riskPercentile >= 75) { + return 0.8 + } else if (riskPercentile >= 50) { + return 0.9 + } else if (riskPercentile >= 25) { + return 0.95 + } else { + return 1.0 + } +}; + +const graph2d = (elem) => { + const colorHash = new ColorHash(); + + const Graph = ForceGraph()(elem) + .graphData({nodes: [], links: []}) + .onNodeHover(node => { + // highlightNodes = node ? [node] : []; + elem.style.cursor = node ? '-webkit-grab' : null; + }) + .linkDirectionalParticles(1) + .linkDirectionalParticleWidth((link) => { + return calcLinkParticleWidth(link, Graph); + }) + .linkDirectionalParticleColor((link) => { + return calcLinkColor(link, Graph) + }) + .linkDirectionalParticleSpeed(0.005) + .linkWidth(4) + .linkAutoColorBy((link) => { + return calcLinkColor(link, Graph) + }) + .linkDirectionalArrowLength(8) + // .linkDirectionalArrowColor(link => { + // return'rgba(323,421,543,224)' + // }) + .linkDirectionalArrowRelPos(link => { + return calcLinkDirectionalArrowRelPos(link, Graph); + }) + .linkCanvasObjectMode(() => 'after') + .linkCanvasObject((link, ctx) => { + const MAX_FONT_SIZE = 8; + const LABEL_NODE_MARGIN = Graph.nodeRelSize() * 1.5; + const start = link.source; + const end = link.target; + // ignore unbound links + link.color = calcLinkColor(link, Graph); + + if (typeof start !== 'object' || typeof end !== 'object') return; + + // calculate label positioning + const textPos = Object.assign(...['x', 'y'].map(c => ({ + [c]: start[c] + (end[c] - start[c]) / 2 // calc middle point + }))); + const relLink = { x: end.x - start.x, y: end.y - start.y }; + + const maxTextLength = Math.sqrt(Math.pow(relLink.x, 2) + Math.pow(relLink.y, 2)) - LABEL_NODE_MARGIN * 8; + + let textAngle = Math.atan2(relLink.y, relLink.x); + // maintain label vertical orientation for legibility + if (textAngle > Math.PI / 2) textAngle = -(Math.PI - textAngle); + if (textAngle < -Math.PI / 2) textAngle = -(-Math.PI - textAngle); + const label = mapLabel(link.label); + // estimate fontSize to fit in link length + ctx.font = '2px Sans-Serif'; + const fontSize = Math.min(MAX_FONT_SIZE, maxTextLength / ctx.measureText(label).width); + ctx.font = `${fontSize + 5}px Sans-Serif`; + + const textWidth = ctx.measureText(label).width; + const bckgDimensions = [textWidth, fontSize].map(n => n + fontSize * 0.2); // some padding + // draw text label (with background rect) + ctx.save(); + ctx.translate(textPos.x, textPos.y); + ctx.rotate(textAngle); + ctx.fillStyle = 'rgba(255, 255, 255, 0.8)'; + ctx.fillRect(- bckgDimensions[0] / 2, - bckgDimensions[1] / 2, ...bckgDimensions); + ctx.textAlign = 'center'; + ctx.textBaseline = 'middle'; + ctx.fillStyle = 'black'; + ctx.fillText(label, 0, 0); + ctx.restore(); + }) + .nodeCanvasObject((node, ctx, globalScale) => { + // add ring just for highlighted nodes + const NODE_R = nodeSize(node, Graph); + ctx.save(); + + // Risk outline color + ctx.beginPath(); + ctx.arc(node.x, node.y, NODE_R * 1.4, 0, 2 * Math.PI, false); + ctx.fillStyle = riskColor(node, Graph, colorHash); + ctx.fill(); + ctx.restore(); + + ctx.save(); + + // Node color + ctx.beginPath(); + ctx.arc(node.x, node.y, NODE_R * 1.2, 0, 2 * Math.PI, false); + + const nodeRbg = calcNodeRgb(node, colorHash); + + ctx.fillStyle = `rgba(${nodeRbg[0]}, ${nodeRbg[1]}, ${nodeRbg[2]}, 1)`; + ctx.fill(); + ctx.restore(); + + const label = node.nodeLabel; + + const fontSize = 15/globalScale; + + ctx.font = `${fontSize}px Sans-Serif`; + + const textWidth = ctx.measureText(label).width; + const bckgDimensions = [textWidth, fontSize].map(n => n + fontSize * 0.2); // some padding + + ctx.fillStyle = 'rgba(255, 255, 255, 0.8)'; + ctx.fillRect(node.x - bckgDimensions[0] / 2, node.y - bckgDimensions[1] / 2, ...bckgDimensions); + ctx.textAlign = 'center'; + ctx.textBaseline = 'middle'; + ctx.fillStyle = 'rgba(0, 0, 0, 0.8)'; + ctx.fillText(label, node.x, node.y); + + }) + .onNodeClick(node => { + const table = (document.getElementById('nodes')); + + const s = nodeToTable(node, Graph); + + table.innerHTML = ` +
+ + ${s} +
+
+ + `; + + }) + .onNodeRightClick(node => { + // Right click expands node + // Pulls edges/ properties down, but does not copy over to the engagement graph + // Increased opacity on 'phantom' nodes + }) + .d3VelocityDecay(0.75) + ; + + Graph.d3Force("link", d3.forceLink()); + Graph.d3Force('collide', d3.forceCollide(22)); + + Graph.d3Force("charge", d3.forceManyBody()); + + Graph.d3Force('box', () => { + const N = 100; + // console.log(Graph.width(), Graph.height()) + const SQUARE_HALF_SIDE = 20 * N * 0.5; + Graph.graphData().nodes.forEach(node => { + const x = node.x || 0, y = node.y || 0; + // bounce on box walls + if (Math.abs(x) > SQUARE_HALF_SIDE) { node.vx *= -1; } + if (Math.abs(y) > SQUARE_HALF_SIDE) { node.vy *= -1; } + }); + }); + return Graph + +}; + +// merges y into x, returns true if update occurred +const mergeNodes = (x, y) => { + let merged = false; + mapNodeProps(y, (prop) => { + if (!Object.prototype.hasOwnProperty.call(x, prop)) { + merged = true; + x[prop] = y[prop] + } + }); + + return merged; +}; + + +class GraphManager { + constructor(graph, dimension) { + const elem = document.getElementById("graph"); + + if (dimension === '3d') { + this.viz = graph3d(elem) + } else if (dimension === '2d') { + this.viz = graph2d(elem); + } + else { + this.viz = graph2d(elem); + } + + this.graph = {...graph}; } updateNode = (newNode) => { if (newNode.uid === undefined) {return} for (let node of this.graph.nodes) { if (node.name === newNode.name) { - node = newNode; - return; + return mergeNodes(node, newNode); } } + console.log('adding new node'); this.graph.nodes.push(newNode); + return true; }; updateLink(newLink) { for (const link of this.graph.links) { - if (link.source === newLink.source) { - if (link.target === newLink.target) { - return; + let src = link.source.name; + if (src === undefined) { + src = link.source; + } + + let tgt = link.target.name; + if (tgt === undefined) { + tgt = link.target; + } + + if (src === newLink.source) { + if (tgt === newLink.target) { + // if (link.label === newLink.label) { + + return false; + // } } } } + this.graph.links.push(newLink); + return true; } removeNode = (uid) => { for (let i = 0; i < this.graph.nodes.length; i++) { - console.log('checking', this.graph.nodes[i].uid); - console.log('checking against', uid); if (this.graph.nodes[i].uid === uid) { - console.log("Removing node"); this.graph.nodes.splice(i, 1); } } @@ -100,122 +600,47 @@ class GraphManager { }; removeNodesAndLinks = (toRemove) => { - console.log("Removing ", toRemove); for (const deadNode of toRemove) { this.removeNode(deadNode); } - this.simulation.nodes(this.graph.nodes); for (const deadLink of toRemove) { this.removeLink(deadLink); } - this.simulation.force('link') - .links(this.graph.links); - - console.log("Removed nodes and links ", this.graph.nodes, this.graph.links); + // console.log("Removed nodes and links ", this.graph.nodes, this.graph.links); }; updateGraph = (newGraph) => { - for (const newNode of newGraph.nodes) { - this.updateNode(newNode); + if (newGraph.nodes.length === 0 && newGraph.links.length === 0) { + return } - this.simulation.nodes(this.graph.nodes); - for (const newLink of newGraph.links) { - this.updateLink(newLink); + if (newGraph === this.graph) { + return } - this.simulation.force('link') - .links(this.graph.links); - - }; - - update = () => { - this.ctx.clearRect(0, 0, this.width, this.height); - - this.simulation.nodes(this.graph.nodes); - this.simulation.force('link') - .links(this.graph.links); - - this.ctx.beginPath(); - this.ctx.globalAlpha = 1.0; - this.ctx.strokeStyle = '#aaa'; - this.graph.links.forEach(this.drawLink); - this.ctx.stroke(); - - this.graph.nodes.forEach(d => this.drawNode(d)); - - }; - - dragsubject = () => this.simulation.find(d3.event.x, d3.event.y); - - - dragstarted = () => { - if (!d3.event.active) this.simulation.alphaTarget(0.3).restart(); - d3.event.subject.fx = d3.event.subject.x; - d3.event.subject.fy = d3.event.subject.y; - - this.start_x = d3.event.subject.x; - this.start_y = d3.event.subject.y; - }; - - dragged = () => { - d3.event.subject.fx = d3.event.x; - d3.event.subject.fy = d3.event.y; - }; - - dragended = () => { - if (!d3.event.active) this.simulation.alphaTarget(0); - d3.event.subject.fx = null; - d3.event.subject.fy = null; - - if (d3.event.subject.x === this.start_x) { - if (d3.event.subject.y === this.start_y) { - const node = d3.event.subject; - - const t = document.getElementById('tooltip'); - if (t !== null) { - t.remove(); - } - - - const s = nodeToTable(node); - - this.tooltip = d3.select('body') - .append('div') - .style('position', 'absolute') - .style('z-index', '100') - .style('visibility', 'hidden') - .html(` - - ${s} -
- `); + let updated = false; + for (const newNode of newGraph.nodes) { + if (this.updateNode(newNode)) { + updated = true; + } + } - this.tooltip.style('visibility', 'visible'); + for (const newLink of newGraph.links) { + if (this.updateLink(newLink)) { + updated = true; } } - this.start_x = null; - this.start_y = null; + if (updated) { + this.update(); + } }; - drawNode = (d) => { - this.ctx.beginPath(); - this.ctx.fillStyle = this.color(d.nodeType); - this.ctx.moveTo(d.x, d.y); - this.ctx.arc(d.x, d.y, this.r, 0, Math.PI * 2); - this.ctx.fill(); - - this.ctx.fillStyle = 'black'; - this.ctx.font = '16px Arial'; - this.ctx.fillText(d.nodeLabel, d.x - this.r, d.y + 35); - }; + update = () => { + this.viz.graphData({...this.graph}) - drawLink = (l) => { - this.ctx.moveTo(l.source.x, l.source.y); - this.ctx.lineTo(l.target.x, l.target.y); }; } @@ -251,22 +676,25 @@ const mapEdgeProps = (node, f) => { }; const _mapGraph = (node, visited, f) => { - if (visited.has(node.uid)) { - return - } - visited.add(node.uid); mapEdgeProps(node, (edgeName, neighbor) => { + if (visited.has(node.uid + edgeName + neighbor.uid)) { + return + } + + visited.add(node.uid + edgeName + neighbor.uid); + f(node, edgeName, neighbor); _mapGraph(neighbor, visited, f) }) }; const mapGraph = (node, f) => { - const visited = new Set(); - mapEdgeProps(node, (edgeName, neighbor) => { - f(node, edgeName, neighbor); - _mapGraph(neighbor, visited, f) - }) + const visited = new Set(); + mapEdgeProps(node, (edgeName, neighbor) => { + + f(node, edgeName, neighbor); + _mapGraph(neighbor, visited, f) + }) }; @@ -280,6 +708,7 @@ const edgeLinksFromNode = (node) => { links.push({ source: node.uid, target, + curvature: 2 }); }); return links; @@ -287,13 +716,13 @@ const edgeLinksFromNode = (node) => { const lensToAdjacencyMatrix = (lens) => { - console.log('lensNode', lens); const nodes = new Map(); const links = new Map(); mapGraph(lens, (fromNode, edgeName, toNode) => { nodes.set(fromNode.uid, fromNode); nodes.set(toNode.uid, toNode); + let edgeList = links.get(fromNode.uid); if (edgeList === undefined) { edgeList = new Map(); @@ -328,13 +757,14 @@ const dgraphNodesToD3Format = (dgraphNodes) => { if (!riskNode.risk_score) { continue } + if (node.risk === undefined) { - node.risk = riskNode.risk_score - node.analyzers = riskNode.analyzer_name + node.risk = riskNode.risk_score; + node.analyzers = riskNode.analyzer_name; } else { - node.risk += riskNode.risk_score + node.risk += riskNode.risk_score; if (node.analyzers.indexOf(riskNode.analyzer_name) === -1) { - node.analyzers += ', ' + riskNode.analyzer_name + node.analyzers += ', ' + riskNode.analyzer_name; } } } @@ -345,7 +775,6 @@ const dgraphNodesToD3Format = (dgraphNodes) => { const nodes = []; for (const node of graph.nodes.values()) { - console.log('node', node); if (node.risk_score || node.analyzer_name) { continue } @@ -353,6 +782,7 @@ const dgraphNodesToD3Format = (dgraphNodes) => { const nodeLabel = getNodeLabel(nodeType, node); nodes.push({ name: node.uid, + id: node.uid, ...node, nodeType, nodeLabel, @@ -398,9 +828,6 @@ const dgraphNodesToD3Format = (dgraphNodes) => { } } - console.log('links', links); - - return { nodes, links, @@ -447,20 +874,25 @@ const getNodeType = (node) => { return 'Connect'; } - if (node.scope !== undefined) { + if (node.scope !== undefined || node.lens !== undefined) { return 'Lens'; } - console.warn('Unable to find type for node'); + // Dynamic nodes + if (node.node_type) { + return node.node_type + } + + console.warn('Unable to find type for node ', node); return 'Unknown'; }; -const nodeToTable = (node) => { - const hidden = new Set(['risks','uid', 'scope', 'name', 'nodeType', 'nodeLabel', 'x', 'y', 'index', 'vy', 'vx', 'fx', 'fy']); - mapEdgeProps(node, (edgeName, neighbor) => { +const nodeToTable = (node, Graph) => { + const hidden = new Set(['id', '__indexColor', 'risks','uid', 'scope', 'name', 'nodeType', 'nodeLabel', 'x', 'y', 'index', 'vy', 'vx', 'fx', 'fy']); + mapEdgeProps(node, (edgeName, _neighbor) => { hidden.add(edgeName) - }) + }); let header = ''; let output = ''; @@ -473,13 +905,26 @@ const nodeToTable = (node) => { header += `${field}`; if (field.includes('_time')) { - output += `${new Date(value).toLocaleString()}>`; + try { + output += `${new Date(value).toLocaleString()}`; + } catch (e) { + console.warn('Could not convert timestamp: ', e); + output += `${sanitizeHTML(value)}`; + } } else { - output += `${value}>`; + if (value.length > 128) { + output += `${sanitizeHTML(value.slice(0, 25))}`; + } else { + output += `${sanitizeHTML(value)}`; + } } - } + header += `risk %`; + const nodes = [...Graph.graphData().nodes].map(node => node.risk); + const riskPercentile = calcNodeRiskPercentile(node.risk, nodes); + output += `${sanitizeHTML(riskPercentile)}`; + return `${header}` + `${output}`; }; @@ -554,7 +999,7 @@ const retrieveGraph = async (graph, lens) => { }); const json_res = await res.json(); - console.info('jsonres ' + json_res); + // console.info('jsonres ' + json_res); const updated_nodes = json_res['updated_nodes']; const removed_nodes = json_res['removed_nodes']; @@ -569,9 +1014,11 @@ const updateLoop = async (graphManager, lens) => { graphManager.graph, lens ); - console.log('removed_nodes', removed_nodes); + console.log('updated_nodes ', updated_nodes); + if (updated_nodes.length !== 0) { - graphManager.updateGraph(dgraphNodesToD3Format(updated_nodes)); + const update = dgraphNodesToD3Format(updated_nodes); + graphManager.updateGraph(update); } if (removed_nodes.length !== 0) { @@ -581,11 +1028,9 @@ const updateLoop = async (graphManager, lens) => { console.warn("Failed to fetch updates ", e) } - graphManager.update(); - // setTimeout(async () => { - // await updateLoop(graphManager, lens); - // graphManager.update(); - // }, 1000) + setTimeout(async () => { + await updateLoop(graphManager, lens); + }, 1000) }; function randomInt(min, max) // min and max included @@ -602,9 +1047,11 @@ document.addEventListener('DOMContentLoaded', async (event) => { return; } + document.getElementById('LensHeader').innerText = `Lens ${lens}`; + // console.log("Initializing graphManager with, ", initGraph); const graphManager = new GraphManager( - {nodes: [], links: []}, + {nodes: [], links: []}, '2d' ); console.log("Starting update loop"); await updateLoop(graphManager, lens);