Add Rapid7 InsightVM source #1010

juju4 · 2022-10-08T21:57:02Z

Add Ingestion of Rapid7 InsightVM assets as source. (follow-up of #1000)
Include initial test data and draft schema

Bugs

fix import of array and dict, or needed values (addresses, configurations...) to get instance_id, subscription_d and resource_id which are required to make relationship with AzureVirtualMachine.
fix mypy issues.

Pending

AzureVirtualMachine relationship.
import vulnerabilities' detections.
may want cleanup retention to be configurable. Typically, EDR keep inventories for 30-45 days.

Reviewed with pylint and black

ramonpetgrave64 · 2022-10-11T15:55:49Z

This is really cool, and thank you!
I can see that this is still draft from you, but I'll try to comment more closely this week.

juju4 · 2022-10-15T19:58:53Z

Like mde, good for review and merge.

cartography/data/jobs/rapid7_import_cleanup.json

.github/workflows/test_suite.yml

cartography/intel/rapid7/util.py

docs/schema/rapid7.md

ramonpetgrave64 · 2022-10-26T15:39:00Z

docs/schema/rapid7.md

+TBD
+* Azure Tenant contains one or more Rapid7 Hosts.
+```
+(AzureTenant)-[RESOURCE]->(Rapid7Host)
+```
+* Azure Subscription contains one or more Rapid7 Hosts.
+```
+(AzureSubscription)-[RESOURCE]->(Rapid7Host)
+```
+* Azure Virtual Machine is one single Rapid7 Host.
+* Similarly for other cloud providers and Onpremises.


Still TBD? Your changes do not create these relationships.

On relationships, I would be interested into some advices.
It does not seem most of the sources of cartography are doing cross-sources relations.
But this is an important point to me.

link Cloud VM with tools present on it.

problem, tool may have not always the information that makes relation certain. In case of Azure

best case is to have resourceid (mde, rapid7 but does not seem always retrieved)

crowdstrike gets only subscriptionid, resourcegroup and hostname

worst case, only have hostname

I started to make AzureVM relation based on resourceid and where not available AzureSubscription based on subscriptionid.
I removed the latter as I found it making graph less readable.
I was thinking to use PRESENT_IN if relation based on resourceid and MAYBE_PRESENT_IN if relation based just on hostname.

What do you think?

feedback on this?

I don't have much context on how Azure works. But our preference is to only be certain about the relationships we create.

the problem is always the same when dealing with multiple tools. is there a reliable key to use to join?
Azure resourceid could be but sadly not fetched reliably by most tools, at least in my context.
hostname/short_hostname is the default join key but hardly reliable (hostname reused, different case depending on tool, different length enforcement depending on tool - some enforce 15-chars windows limit...)

At this point, I'm plan to make relation base on resourceid or hostname/short_hostname.

more to add here?

cartography/intel/rapid7/util.py

This reverts commit 990e547.

juju4 · 2022-10-29T20:17:52Z

As said, would need a bit of help on mypy error as less familiar with the tool.

Also, for some reason, CLA check is not green, but when going to https://oss.lyft.com/cla, it is marked signed and current. I did it a few weeks ago.

juju4 · 2022-10-29T20:18:07Z

Ah,CLA check is green now

juju4 · 2022-11-05T15:25:59Z

on my side, linter looks all good.
only pyupgrade is failing but not related to code and did recent change to 2.31. 0 (3.2.0 works ok for me)

cartography/intel/rapid7/util.py

cartography/cli.py

cartography/data/jobs/cleanup/rapid7_import_cleanup.json

cartography/intel/rapid7/util.py

ramonpetgrave64 · 2022-11-07T00:55:12Z

cartography/intel/rapid7/util.py

+    Input: dataframe row configurations column
+    """
+
+    if configurations is None or not isinstance(configurations, list):


Does the data returned from rapid7s api sometimes have empty fields?

fine to resolve?

cartography/intel/rapid7/util.py

ramonpetgrave64 · 2022-11-07T01:00:21Z

cartography/intel/rapid7/util.py

+                azure = json.loads(line["value"])
+            else:
+                azure = line["value"]
+            instance_id = azure["instanceId"]


Will and "azure" value always be an instance?

if azure type, I believe there will be instanceid
but not all azure resources are correctly marked azure type

ramonpetgrave64

Please do try to address the pylint comments.

juju4 · 2022-11-13T00:21:26Z

I also have a mypy error that I'm not sure how to address

cartography/intel/rapid7/util.py:171: error: Unsupported target for indexed assignment ("Tuple[str, str, str, bool]")  [index]

cartography/intel/rapid7/util.py

…l through local file (dirpath) or downloadReport API (report-id), s/list/List/

ramonpetgrave64 · 2022-12-01T23:09:16Z

I was on vacation for the past month, but I'll be taking another look at this PR again next week.

cartography/intel/rapid7/util.py

ramonpetgrave64 · 2022-12-29T15:55:52Z

cartography/intel/rapid7/util.py

+    Even w 48GB RAM, only up to 13k...
+    """
+
+    if authorization[2] and authorization[4]:


Since we're checking for positions in this 6-tuple, It's better to convert the authorization to a class or named tuple
https://docs.python.org/3/library/collections.html?highlight=namedtuple#collections.namedtuple

cartography/intel/rapid7/util.py

ramonpetgrave64 · 2022-12-29T15:58:11Z

tests/integration/cartography/rapid7/test_endpoints.py

+    nodes = neo4j_session.run(
+        """
+        MATCH (n:Rapid7Host)
+        RETURN n.id


Try to add one more test to check for the relationship to the Azure hosts.

did a quick attempt. testing needed

ramonpetgrave64 · 2022-12-29T15:59:53Z

docs/root/modules/rapid7/rapid7.md

+|-------|--------------|
+| firstseen| Timestamp of when a sync job first discovered this node  |
+| lastupdated |  Timestamp of the last time the node was updated |
+| r7_id | id |


Since we know this is a Rapid7Host node, there's no need to prefix each of the properties with "r7_".

IMHO, it depends on each product which may have different output.
Typically for IP/MAC address, if multiple cards, there is no certainty that each product will pick the same. same for os product, version... no shared taxonomy sadly.

if still want to remove prefix, please say if everywhere or only part of fields:

ramonpetgrave64 · 2022-12-29T16:03:50Z

cartography/intel/rapid7/util.py

+    size_interval = 500
+    result_array: List[Any] = []
+    df_rapid7_tmp = pandas.DataFrame()
+    while count < limit:


Why do we have a limit? Is it because of API ratelimitting? If that's the case, consider using the @backoff decorator.
https://github.com/lyft/cartography/blob/master/cartography/util.py#L206

No API rate-limiting AFAIK, more response body size and experience that it is more likely to fail if more depending on context. at least, not on doc https://help.rapid7.com/insightvm/en-us/api/index.html#operation/findAssets

Another reason to use the report option.

ramonpetgrave64 · 2022-12-29T16:05:28Z

cartography/intel/rapid7/util.py

+    else:
+        nexpose_verify_cert2 = True
+
+    count = total_resources = 0


It looks like count is unused.

cartography/intel/rapid7/util.py

… add tags to downloadreport, cleaning

juju4 · 2023-01-07T17:24:47Z

Currently, this can get data from

API getAssets
local report file
remote report file directly from vulnerability management server

I don't know if should keep the API one which has limitations mostly for large inventory.
After, there are probably some cases where it is better than report.
On my side, using remote report file now in a satisfying manner.

Mypy typing errors that I didn't solve (or was impacting functionalities), all related to extract_rapid7_configurations_* functions.
https://github.com/juju4/cartography/actions/runs/3862847617/jobs/6584667101#step:4:67

cartography/intel/rapid7/util.py:51: error: Function is missing a type annotation for one or more arguments  [no-untyped-def]
cartography/intel/rapid7/util.py:80: error: Function is missing a type annotation for one or more arguments  [no-untyped-def]
cartography/intel/rapid7/util.py:104: error: Function is missing a type annotation for one or more arguments  [no-untyped-def]
cartography/intel/rapid7/util.py:128: error: Function is missing a type annotation for one or more arguments  [no-untyped-def]
Found 4 errors in 1 file (checked 441 source files)

juju4 · 2023-02-25T15:49:07Z

is there something that I can help to move this forward?
outside of the points left above for which I would like inputs too.

juju4 added 2 commits October 8, 2022 21:36

Add Rapid7 source - work in progress

6a05ade

test_suite workflow: include devel*

990e547

juju4 added 5 commits October 15, 2022 19:16

test/data/rapid7: rename file

fbb034f

fix incorrect typing

2c26e76

add rapid7 cleanup file

2df1b8d

rapid7 source update

3422631

Merge branch 'devel' into devel-rapid7

eb5dd17

ramonpetgrave64 suggested changes Oct 26, 2022

View reviewed changes

juju4 added 3 commits October 29, 2022 19:43

removing duplicate file

20b5057

Revert "test_suite workflow: include devel*"

4ff30be

This reverts commit 990e547.

rapid7 code review

487fa0c

juju4 added 2 commits November 5, 2022 15:10

code review, fix most pre-commit

ec1d738

Merge branch 'master' into devel-rapid7

f73ae4d

juju4 added 2 commits November 5, 2022 15:48

fix mypy misc

e6fe2a8

tests/integration: fix filename changed

00f5901

ramonpetgrave64 reviewed Nov 7, 2022

View reviewed changes

cartography/intel/rapid7/util.py Outdated Show resolved Hide resolved

ramonpetgrave64 reviewed Nov 7, 2022

View reviewed changes

cartography/cli.py Outdated Show resolved Hide resolved

ramonpetgrave64 reviewed Nov 7, 2022

View reviewed changes

cartography/cli.py Outdated Show resolved Hide resolved

ramonpetgrave64 reviewed Nov 7, 2022

View reviewed changes

cartography/data/jobs/cleanup/rapid7_import_cleanup.json Show resolved Hide resolved

ramonpetgrave64 reviewed Nov 7, 2022

View reviewed changes

cartography/intel/rapid7/util.py Outdated Show resolved Hide resolved

ramonpetgrave64 reviewed Nov 7, 2022

View reviewed changes

cartography/intel/rapid7/util.py Outdated Show resolved Hide resolved

ramonpetgrave64 reviewed Nov 7, 2022

View reviewed changes

cartography/intel/rapid7/util.py Outdated Show resolved Hide resolved

ramonpetgrave64 reviewed Nov 7, 2022

View reviewed changes

ramonpetgrave64 suggested changes Nov 7, 2022

View reviewed changes

rapid7 code review

42d30fd

fix mypy: separate nexpose_verify_cert var

fce2c1c

ramonpetgrave64 suggested changes Nov 15, 2022

View reviewed changes

juju4 added 3 commits November 19, 2022 17:58

switch s/logger.warning/logger.debug/

c3713b4

remove resp.content logging

5beffa9

fix rapid7-verify-cert (using str instead of bool), add data retrieve…

d66152c

…l through local file (dirpath) or downloadReport API (report-id), s/list/List/

ramonpetgrave64 suggested changes Dec 29, 2022

View reviewed changes

juju4 added 8 commits January 7, 2023 15:27

Merge branch 'devel' into devel-rapid7

739c2cc

black autoformatter

5def4a2

remove typing for extract_rapid7_configurations_* as break functions,…

d5ee84b

… add tags to downloadreport, cleaning

fix rapid7 test: int vs str

9f71413

add get_ prefix to functions

5625402

more review requests

714d441

add rapid7 relationship test - work in progress

da573a8

fix test_load_host_data()

02fb890

juju4 added 4 commits January 7, 2023 18:40

fix test_load_host_data() (2)

84f72d4

Merge branch 'devel' into devel-rapid7

d0b762d

Merge branch 'master' into devel-rapid7

aac4fe4

Merge branch 'master' into devel-rapid7

286a02e

Merge branch 'master' into devel-rapid7

ed902a5

juju4 mentioned this pull request Aug 26, 2023

Least-privileged data collection? JupiterOne/graph-rapid7#99

Open

Merge branch 'master' into devel-rapid7

303ec3a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Rapid7 InsightVM source #1010

Add Rapid7 InsightVM source #1010

juju4 commented Oct 8, 2022

ramonpetgrave64 commented Oct 11, 2022

juju4 commented Oct 15, 2022

ramonpetgrave64 Oct 26, 2022

juju4 Oct 29, 2022

juju4 Nov 5, 2022

ramonpetgrave64 Nov 15, 2022

juju4 Nov 19, 2022

juju4 Jan 7, 2023

juju4 commented Oct 29, 2022

juju4 commented Oct 29, 2022

juju4 commented Nov 5, 2022

ramonpetgrave64 Nov 7, 2022

juju4 Nov 13, 2022

juju4 Jan 7, 2023

ramonpetgrave64 Nov 7, 2022

juju4 Nov 13, 2022

ramonpetgrave64 left a comment

juju4 commented Nov 13, 2022

ramonpetgrave64 commented Dec 1, 2022

ramonpetgrave64 Dec 29, 2022

ramonpetgrave64 Dec 29, 2022

juju4 Jan 7, 2023

ramonpetgrave64 Dec 29, 2022

juju4 Jan 7, 2023

ramonpetgrave64 Dec 29, 2022

juju4 Jan 7, 2023

ramonpetgrave64 Dec 29, 2022

juju4 commented Jan 7, 2023

juju4 commented Feb 25, 2023

Add Rapid7 InsightVM source #1010

Are you sure you want to change the base?

Add Rapid7 InsightVM source #1010

Conversation

juju4 commented Oct 8, 2022

ramonpetgrave64 commented Oct 11, 2022

juju4 commented Oct 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juju4 commented Oct 29, 2022

juju4 commented Oct 29, 2022

juju4 commented Nov 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ramonpetgrave64 left a comment

Choose a reason for hiding this comment

juju4 commented Nov 13, 2022

ramonpetgrave64 commented Dec 1, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juju4 commented Jan 7, 2023

juju4 commented Feb 25, 2023