Old machine bootloader #527

VakarisZ · 2020-01-14T08:54:25Z

What is this?

Fixes #479

Add any further explanations here.

TODO

Make pyinstaller bootloader launch custom C code
Create C file that would communicate with island on Windows
FIX Windows XP requests support #528
Create C file that would communicate with island on Linux
Make island recognize an display machines where bootloader communicated, but monkey did not
Refactor pyinstaller alterations into a patch, which we can apply to any incoming new pyinstaller version
Alter deployment script to include bootloader and pyinstaller dev version with patch
Test on OS compat env

TDD

Prerequisites

Old machine bootloader program is incorporated into pyinstaller bootloader. Pyinstaller bootloader is the first thing that starts when a built monkey is launched, thus our code is launched before monkey and has access to monkey flags. This allows us to write code that would inspect the current machine and send requests to island directly or via tunnel and decide to continue with launching monkey code or to quit.

This is a multilevel problem so TDD is separated into these sections that represent corresponding layers:

Bootloader program
Connections and tunneling
Island server and data gathered
Data representation

Bootloader program

How to setup bootloader development env.

We have pyinstaller forks, one for linux and one for windows. Pull them and get relevant bootloader dev env.

To update pyinstaller we'll have to pull from pyinstaller git, merge and rebuild pyinstaller bootloader. Then do a release and add built pyinstaller bootloader.
To update bootloader we'll have to push to our forks and rebuild pyinstaller bootloader. Then do a release and add built pyinstaller bootloader.

Monkey build process

Building the monkey will require custom pyinstaller bootloader binary. Depending on OS this binary should be downloaded from our pyinstaller for release and placed into a pyinstaller install folder. Maybe this can be automated via deployment scripts.

Bootloader program workflow

Monkey binary is ran.
Bootloader starts and parses monkey flags to extract tunnel and server.
Bootloader collects information about system: OS, IP, Hostname.
Bootloader looks at current OS info and determines whether to run the monkey or to quit after it finishes.
Bootloader changes server port from x to x+1 OR we can have a dedicated flag.
Bootloader forms http request with OS, IP, Hostname and will monkey be ran info.
Bootloader sends HTTP request where a proxy server is listening.
If bootloader didn't get proper response it tries to use tunnel.

Connections and tunneling

Monkey

Tunneling on monkey is expanded to tunnel http traffic.

Island

Island starts a basic http server on a separate thread. It connects to the mongodb and gets port to serve on. It's either island's port +1 or a dedicated config value. When this HTTP proxy server gets a request it passes it to the /api/bootloader endpoint on the main server and also forwards back the response.

Island server and data gathered

Island receives data on an api/bootloader endpoint and stores it on a separate collection. Each bootloader telemetry has the following fields: OS, IP list, Hostname and if monkey will be run. Upon receiving bootloader telemetry island finds and updates bootloader_state variable on corresponding node.
bootloader_state can be one of three values:

Bottloader didn't run (default)
Bootloader ran, monkey will run ( if monkey_will_run value is True in bootloader telem )
Bootloader ran, monkey will not run ( if monkey_will_run value is False in bootloader telem )

Data representation

Map

States of node will be the following:

Scanned node (exploited: False bootloader_state: not_started on node)
Exploited node (exploited: True bootloader_state: not_started on node)
Machine too old (exploited: True bootloader_state: monkey_won't_run on node)
OR
Monkey will run (exploited: True bootloader_state: monkey_will_run on node)
Monkey is running.
Monkey died.

To recap 2 states with corresponding UI will be added: Machine too old to run monkey and Monkey will run.

Report

When generating report island will query mongodb to find if there are any entries of bootloader telemetry with bootloader_state: monkey_won't_run. If so, outdated machine issue is generated.

Testing

Bootloader binary

Manual tests on some old windows and linux machines. Also, on some new ones.

Tunneling

Windows machine will be added to tunneling env in monkeyzoo. New blackbox test will run tunneling test as usual, but will also verify that each bootloader communicated with island.

Data gathering and UI

Tested manually

Dev. env. deployment

This feature changes development environment in the following way:

We need custom pyinstaller bootloader to build monkeys

So deployment scripts will have to be altered to do:

Download pyinstaller bootloader from our pyinstaller fork release
Replace default bootloader in pyinstaller folder

…o pyinstaller

…there still is a timeout

… to old and more

danielguardicore · 2020-02-02T09:40:36Z

Some notes on the design.

For building the bootloader, I suggest doing the following
1 - Creating a patch file with all the changes, that can automatically be applied to the Pyinstaller source code.
2 - Making sure the building can be automated.
I suggest having this code be part of the Monkey repository, but under a different top level folder than monkey.

Regarding libcurl and openssl. If we are connecting to the Monkey, then we probably only need libcurl. I don't think we need our version of openssl.

I think the open port should not be hard coded to be 5001, but rather +1 on the current server port. This is a simple change and will also help in the Monkey Island side. We will also need to update documentation on the Island.

Regarding data collected. I suggest the following fields

All IP addresses (IPv4, IPv6)
Computer hostname
OS version string (compatible with current Monkey)

Regarding saving the bootloader message. I suggest turning it into a telemetry message then it will be saved in that collection automatically. This doesn't require us to really include JSON parsing in the bootloader, since it's a pretty hard coded message.

VakarisZ · 2020-02-04T09:08:04Z

As @ShayNehmad suggested, it's probably best to just make forks of pyinstaller. Those forks will store the code and binaries.
Automated as in run a command -> get bootloader? That is already automated in pyinstaller. If by automated you mean run a command -> get bootloader and replace current bootloader binary, then IDK if that's necessary.
Regarding proxy server port there's two options: current server port +1 or a separate config field and a separate flag for monkey. How is island port configuration happening right now, how can you change it?
Regarding telemetries, now island expects to get telemetries from monkey, so it looks for guid field. To be less error prone we should save bootloader telem on a separate collection, since it has different structure, no?

danielguardicore · 2020-02-04T13:14:25Z

1 - Ok.
2 - Ok
3 - I think it should be +1 and then we don't need to pass more flags.

danielguardicore · 2020-02-04T13:26:42Z

Regarding pyinstaller env, I think the best for the dev enviornment is just building from git source rather than Pypi. Pip supports that. We should explicitly list our repo and then we don't need to copy/override things.

danielguardicore · 2020-02-04T13:28:04Z

Regarding basic Island http server. Is this also a tornado/flask server or something else?

VakarisZ · 2020-02-05T08:57:33Z

ATM it's vanilla python BaseHTTPServer

…ng and icon placeholders

… windows bootloaders

ShayNehmad

Haven't finished my review yet, but some comments are here already. Will continue tomorrow

ShayNehmad · 2020-02-24T13:06:31Z

monkey/infection_monkey/monkey.py

+
+        #TODO change back
+        time.sleep(WormConfiguration.keep_tunnel_open_time)
+


TODO left in the code

ShayNehmad · 2020-02-24T13:12:53Z

monkey/infection_monkey/transport/http.py

@@ -108,7 +110,20 @@ def log_message(self, format_string, *args):
 class HTTPConnectProxyHandler(http.server.BaseHTTPRequestHandler):
    timeout = 30  # timeout with clients, set to None not to make persistent connection
    proxy_via = None  # pseudonym of the proxy in Via header, set to None not to modify original Via header
-    protocol_version = "HTTP/1.1"
+    #protocol_version = "HTTP/1.1"


Don't leave commented out code

ShayNehmad · 2020-03-03T16:21:43Z

monkey/monkey_island/cc/bootloader_server.py

+            self.send_response(200)
+        self.end_headers()
+        self.wfile.write(r.content)
+        self.connection.close()


Should we wrap this function with try except finally and then close the connection on the finally?

Not a necessity, but we can

ShayNehmad · 2020-03-03T16:22:04Z

monkey/monkey_island/cc/main.py

+    logger.info("Starting bootloader server")
+    mongo_url = os.environ.get('MONGO_URL', env.get_mongo_url())
+    bootloader_server_thread = Thread(target=BootloaderHttpServer(mongo_url).serve_forever, daemon=True)
+    # island_server_thread = Thread(target=start_island_server)


Commented out code

ShayNehmad · 2020-03-03T16:22:47Z

monkey/monkey_island/cc/main.py

+    bootloader_server_thread = Thread(target=BootloaderHttpServer(mongo_url).serve_forever, daemon=True)
+    # island_server_thread = Thread(target=start_island_server)
+
+    bootloader_server_thread.start()
+    #island_server_thread.start()
+    start_island_server()
+    bootloader_server_thread.join()


Extract all this to function

What do you mean? It's already the only lines lines in this function.

ShayNehmad · 2020-03-03T16:23:55Z

monkey/monkey_island/cc/resources/bootloader.py

+    @staticmethod
+    def parse_bootloader_request_linux(request_data: bytes) -> Dict[str, str]:
+        parsed_data = json.loads(request_data.decode().replace("\n", "")
+                                                      .replace("NAME=\"", "")
+                                                      .replace("\"\"", "\"")
+                                                      .replace("\":\",", "\":\"\","))
+        return parsed_data
+
+    @staticmethod
+    def parse_bootloader_request_windows(request_data: bytes) -> Dict[str, str]:
+        return json.loads(request_data.decode("utf-16", "ignore"))


Cover this with UTs - the tests will be good documentation as well

ShayNehmad · 2020-03-03T16:24:30Z

monkey/monkey_island/cc/resources/bootloader.py

+        else:
+            return make_response({"status": "OS_NOT_FOUND"}, 404)
+
+        resp = BootloaderService.parse_bootloader_telem(data)


rename resp

monkey/monkey_island/cc/services/bootloader.py

ShayNehmad

Added more comments

ShayNehmad · 2020-03-04T10:58:36Z

monkey/monkey_island/cc/ui/src/components/map/preview-pane/PreviewPane.js

+          info = this.props.item.group.includes('monkey', 'manual') ? this.infectedAssetInfo(this.props.item) :
+            this.props.item.group !== 'island' ? this.assetInfo(this.props.item) : this.islandAssetInfo();


Can we re-write this conditinal to be more explicit and clearer? it's very hard to understand even upon second reading

I'll rewrite it, even though this is not mine.

ShayNehmad · 2020-03-04T10:59:27Z

monkey/monkey_island/cc/ui/src/components/map/MapOptions.js

+// This list must correspond to the one on back end in cc/services/utils/node_groups.py
 const groupNames = ['clean_unknown', 'clean_linux', 'clean_windows', 'exploited_linux', 'exploited_windows', 'island',
-  'island_monkey_linux', 'island_monkey_linux_running', 'island_monkey_windows', 'island_monkey_windows_running',
-  'manual_linux', 'manual_linux_running', 'manual_windows', 'manual_windows_running', 'monkey_linux',
-  'monkey_linux_running', 'monkey_windows', 'monkey_windows_running'];
+  'island_monkey_linux', 'island_monkey_linux_running', 'island_monkey_linux_starting', 'island_monkey_windows',
+  'island_monkey_windows_running', 'island_monkey_windows_starting', 'manual_linux', 'manual_linux_running',
+  'manual_windows', 'manual_windows_running', 'monkey_linux', 'monkey_linux_running', 'monkey_windows',
+  'monkey_windows_running', 'monkey_windows_starting', 'monkey_linux_starting', 'monkey_windows_old',
+  'monkey_linux_old' ];


Perhaps a joined JSON file in common somewhere is a better solution?

I exported it to a file, although it would be better to just send it from backend.

On a second thought lets refactor

ShayNehmad · 2020-03-04T11:34:18Z

monkey/monkey_island/cc/services/utils/node_groups_test.py

+        tst1 = NodeGroups.get_group_by_keywords(['island']) == NodeGroups.ISLAND
+        tst2 = NodeGroups.get_group_by_keywords(['running', 'linux', 'monkey']) == NodeGroups.MONKEY_LINUX_RUNNING
+        tst3 = NodeGroups.get_group_by_keywords(['monkey', 'linux', 'running']) == NodeGroups.MONKEY_LINUX_RUNNING
+        tst4 = False
+        try:
+            NodeGroups.get_group_by_keywords(['bogus', 'values', 'from', 'long', 'list', 'should', 'fail'])
+        except NoGroupsFoundException:
+            tst4 = True
+        self.assertTrue(tst1 and tst2 and tst3 and tst4)


Cool test :)
It would be better to assert each statement separately I think

…requests

…loader

danielguardicore · 2020-03-15T10:03:56Z

@VakarisZ I think you need to update the pip requirement to aim at our pyinstaller fork

monkey/monkey_island/cc/services/node.py

ShayNehmad · 2020-03-15T13:10:39Z

monkey/monkey_island/cc/services/utils/node_states_test.py

+
+class TestNodeGroups(TestCase):
+
+    def test_get_group_by_keywords(self):


Use self.assertEqual(NodeStates.get_by_keywords(['island']), NodeStates.ISLAND) instead of assertTrue for tests 1,2 and 3
Use self.assertRaises(ExpectedException, afunction) for test 4

…ng machine used in tests

…loader # Conflicts: # monkey/monkey_island/cc/main.py

codecov · 2020-03-17T10:55:41Z

Codecov Report

Merging #527 into develop will increase coverage by 0.56%.
The diff coverage is 64.46%.

@@             Coverage Diff             @@
##           develop     #527      +/-   ##
===========================================
+ Coverage    56.15%   56.72%   +0.56%     
===========================================
  Files          105      112       +7     
  Lines         3588     3820     +232     
===========================================
+ Hits          2015     2167     +152     
- Misses        1573     1653      +80

Impacted Files	Coverage Δ
monkey/infection_monkey/transport/http.py	`21.15% <14.28%> (-1.19%)`	⬇️
monkey/monkey_island/cc/services/edge.py	`26.88% <20.00%> (-0.90%)`	⬇️
monkey/monkey_island/cc/services/node.py	`30.73% <35.13%> (+0.94%)`	⬆️
monkey/monkey_island/cc/services/bootloader.py	`47.16% <47.16%> (ø)`
monkey/monkey_island/cc/utils.py	`42.00% <60.00%> (+2.00%)`	⬆️
monkey/monkey_island/cc/resources/bootloader.py	`60.86% <60.86%> (ø)`
...key/monkey_island/cc/services/utils/node_states.py	`97.67% <97.67%> (ø)`
...nkey/monkey_island/cc/resources/bootloader_test.py	`100.00% <100.00%> (ø)`
...onkey/monkey_island/cc/services/bootloader_test.py	`100.00% <100.00%> (ø)`
...nkey_island/cc/services/utils/bootloader_config.py	`100.00% <100.00%> (ø)`
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8eab25e...1ff6a91. Read the comment docs.

… for bootloader resources

As per #527, this code was added for the bootloader. Now that we're removing the bootloader, this is no longer needed.

VakarisZ added 3 commits January 10, 2020 14:31

Added cpp file that creates a file (for testing)

f46635a

Added bootloader c file

4e150ea

Added bootloader endpoint, fixed c code to be able to be included int…

c8618e9

…o pyinstaller

ShayNehmad self-requested a review January 20, 2020 11:27

ShayNehmad assigned VakarisZ Jan 20, 2020

ShayNehmad added Feature Issue that describes a new feature to be implemented. Monkey labels Jan 20, 2020

ShayNehmad added this to the 1.8.0 milestone Jan 20, 2020

VakarisZ added 4 commits January 27, 2020 17:44

Code changes that loosely implement the prototype of bootloader, but …

d349f26

…there still is a timeout

Small style improvement, disabled explicit http 1.1

04c09b5

updated windows old machine bootloader code to use tunnel, quit if os…

aed7e3f

… to old and more

added linux wscript that includes libcurl into pyinstaller bootloader

0c15798

VakarisZ added 9 commits February 12, 2020 11:09

Added node and edge based on bootloader telemetry.

72d266a

Improved hostname by parsing out irrelevant data

1ebbb83

Adjusted island to parse changed host info

f792572

Added the logic of assigning states to nodes, state(node group) parsi…

a0d4b82

…ng and icon placeholders

Improved responses to linux bootloader, removed unnecessary files

c2b1250

Island adjusted to differentiate parsing data incoming from linux and…

056c260

… windows bootloaders

Fixed windows bootloader telem parsing for windows

d8aa63d

Fixed bug that treated bootloader node as infected node

8621843

Fixed bugs related to bootloader on island

1e7775a

ShayNehmad suggested changes Mar 3, 2020

View reviewed changes

ShayNehmad reviewed Mar 4, 2020

View reviewed changes

VakarisZ added 6 commits March 6, 2020 17:22

Refactored to have node state list only on backend and more CR fixes

7475cff

Fixed bootloader creating multiple nodes bug

e5201a2

Removed unnecessary comments, unused file, improved parsing of linux …

d75299b

…requests

Merge remote-tracking branch 'upstream/develop' into old_machine_boot…

2fff8d3

…loader

Fixed UI bug where security report map didn't show propper images

f07d058

Small bugfix: allow monkey run without network exploitation

586057d

ShayNehmad reviewed Mar 15, 2020

View reviewed changes

VakarisZ added 3 commits March 16, 2020 12:07

Improved bootloader telem logging, added docs about monkeyzoo tunneli…

f82b56f

…ng machine used in tests

Merge remote-tracking branch 'upstream/develop' into old_machine_boot…

a22cd89

…loader # Conflicts: # monkey/monkey_island/cc/main.py

Added proxies to post request going towards the island

a979f21

VakarisZ added 3 commits March 18, 2020 09:32

Added our version of pyinstaller to requirements and added unit tests…

69a4f30

… for bootloader resources

Improved node state tests

a0a2311

Fixed bootloader networking bug and improved exception handling

1ff6a91

ShayNehmad merged commit 4da6a6a into guardicore:develop Mar 30, 2020

VakarisZ deleted the old_machine_bootloader branch October 15, 2020 08:29

shreyamalviya added a commit that referenced this pull request Feb 1, 2022

Agent: Remove do_POST in HTTPConnectProxyHandler

76dee88

As per #527, this code was added for the bootloader. Now that we're removing the bootloader, this is no longer needed.

ilija-lazoroski pushed a commit that referenced this pull request Feb 1, 2022

Agent: Remove do_POST in HTTPConnectProxyHandler

9f11ee7

As per #527, this code was added for the bootloader. Now that we're removing the bootloader, this is no longer needed.

ilija-lazoroski pushed a commit that referenced this pull request Feb 1, 2022

Agent: Remove do_POST in HTTPConnectProxyHandler

db965e1

As per #527, this code was added for the bootloader. Now that we're removing the bootloader, this is no longer needed.

snyk-bot mentioned this pull request Aug 15, 2022

[Snyk] Upgrade react-spinners from 0.9.0 to 0.13.3 SarmadBytes/monkey#15

Open

snyk-bot mentioned this pull request Aug 24, 2022

[Snyk] Upgrade react-spinners from 0.9.0 to 0.13.4 SarmadBytes/monkey#22

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Old machine bootloader #527

Old machine bootloader #527

VakarisZ commented Jan 14, 2020 •

edited

Loading

danielguardicore commented Feb 2, 2020

VakarisZ commented Feb 4, 2020

danielguardicore commented Feb 4, 2020 •

edited

Loading

danielguardicore commented Feb 4, 2020

danielguardicore commented Feb 4, 2020

VakarisZ commented Feb 5, 2020

ShayNehmad left a comment

ShayNehmad Feb 24, 2020

ShayNehmad Feb 24, 2020

ShayNehmad Mar 3, 2020

VakarisZ Mar 6, 2020

ShayNehmad Mar 3, 2020

ShayNehmad Mar 3, 2020

VakarisZ Mar 6, 2020

ShayNehmad Mar 3, 2020

ShayNehmad Mar 3, 2020

ShayNehmad left a comment

ShayNehmad Mar 4, 2020

VakarisZ Mar 6, 2020

ShayNehmad Mar 4, 2020

VakarisZ Mar 6, 2020

VakarisZ Mar 6, 2020

ShayNehmad Mar 4, 2020

danielguardicore commented Mar 15, 2020

ShayNehmad Mar 15, 2020

codecov bot commented Mar 17, 2020 •

edited

Loading


		#TODO change back
		time.sleep(WormConfiguration.keep_tunnel_open_time)

		info = this.props.item.group.includes('monkey', 'manual') ? this.infectedAssetInfo(this.props.item) :
		this.props.item.group !== 'island' ? this.assetInfo(this.props.item) : this.islandAssetInfo();


		class TestNodeGroups(TestCase):

		def test_get_group_by_keywords(self):

Old machine bootloader #527

Old machine bootloader #527

Conversation

VakarisZ commented Jan 14, 2020 • edited Loading

What is this?

TODO

TDD

Prerequisites

This is a multilevel problem so TDD is separated into these sections that represent corresponding layers:

Bootloader program

How to setup bootloader development env.

Monkey build process

Bootloader program workflow

Connections and tunneling

Monkey

Island

Island server and data gathered

Data representation

Map

Report

Testing

Bootloader binary

Tunneling

Data gathering and UI

Dev. env. deployment

danielguardicore commented Feb 2, 2020

VakarisZ commented Feb 4, 2020

danielguardicore commented Feb 4, 2020 • edited Loading

danielguardicore commented Feb 4, 2020

danielguardicore commented Feb 4, 2020

VakarisZ commented Feb 5, 2020

ShayNehmad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShayNehmad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielguardicore commented Mar 15, 2020

Choose a reason for hiding this comment

codecov bot commented Mar 17, 2020 • edited Loading

Codecov Report

VakarisZ commented Jan 14, 2020 •

edited

Loading

danielguardicore commented Feb 4, 2020 •

edited

Loading

codecov bot commented Mar 17, 2020 •

edited

Loading