Current dev

.gitignore

@@ -1,4 +1,10 @@
-
-.idea/
 *.iml
-java/
+*.exe
+*.log
+*.json
+crackers
+prince
+files
+hashlists
+__pycache__
+*.zip

.travis.yml

@@ -0,0 +1,9 @@
+language: python
+python:
+  - "3.5"
+  - "3.6"
+install:
+  - pip install -r requirements.txt
+script:
+  - ./build.sh
+  - python hashtopolis.zip --version

README.md

@@ -1,11 +1,98 @@
-# Hashtopolis Agents
+# Hashtopolis Python Agent
 
-Official agents for using the distributed hashcracker [Hashtopolis](https://github.com/s3inlc/hashtopolis).
+[![CodeFactor](https://www.codefactor.io/repository/github/s3inlc/hashtopolis-agent-python/badge)](https://www.codefactor.io/repository/github/s3inlc/hashtopolis-agent-python)
+[![LoC](https://tokei.rs/b1/github/s3inlc/Hashtopolis-Agent-Python?category=code)](https://github.com/s3inlc/Hashtopolis-Agent-Python)
+[![Build Status](https://travis-ci.org/s3inlc/hashtopolis-agent-python.svg?branch=master)](https://travis-ci.org/s3inlc/hashtopolis-agent-python)
 
-Currently available agents:
-- [C#](csharp)
-- [Python](python)
+This agent is used with [Hashtopolis](https://github.com/s3inlc/hashtopolis), read the wiki or create issues there, visit the [Forum](https://hashtopolis.org).
+This Hashtopolis agent is only compatible with Hashtopolis versions 0.5.0 and higher.
 
-## Protocol Documentation
+## Prerequisites
 
-A documentation to the protocol which is used to communicate between the agent and the server can be found here: [Protocol.pdf](https://github.com/s3inlc/hashtopolis/blob/master/doc/protocol.pdf)
+You need python3 installed on your agent system. 
+Following python packages are required:
+
+* requests
+
+## Manual
+
+You can either download the agent from the Hashtopolis new agent page or you can use the url shown there to download the agent with 
+wget/curl.
+
+### Run
+
+To run the agent you simply need to call `python3 hashtopolis.zip`. There are no command line options accepted, all 
+settings/configurations are done via the config file, described in the following section.
+
+Please note that the client does not correctly recognize the OS when you are running in Cygwin or similar on Windows. You need to run it in Windows command line.
+
+### Config
+
+When you run the client for the first time it will ask automatically for all the requirement settings and then saves it automatically to a config file called `config.json`. This could for example look like this:
+
+```
+{
+  "url": "https://example.org/hashtopolis/api/server.php", 
+  "token": "ABCDEFGHIJ", 
+  "uuid": "49dcd31c-3637-4f2a-8df1-b545202df5b3"
+}
+```
+
+### Overview
+
+| field                 | type    | default | description                                                                |
+|-----------------------|---------|---------|----------------------------------------------------------------------------|
+| voucher               | string  |         | Used for agent registration (will be prompted on first start)              |
+| url                   | string  |         | The hashtopolis API endpoint (will be prompted on first start)             |
+| token                 | string  |         | The access token for the API (sent by server on registration)              |
+| uuid                  | string  |         | Unique identifier of the agent (generated on registration)                 |
+| debug                 | boolean | false   | Enables debug output                                                       |
+| allow-piping          | boolean | false   | Allows hashcat to read password candidates from stdin                      |
+| piping-threshold      | integer | 95      | Restarts chunk in piping mode when GPU UTIL is below this value            |
+| rsync                 | boolean | false   | Enables download of wordlists and rules via rsync                          |
+| rsync-path            | string  |         | Remote path to hashtopolis files directory                                 |
+| multicast-device      | string  | eth0    | Device which is used to retrieve UDP multicast file distribution           |
+| file-deletion-disable | boolean | false   | Disable requesting the server for files to delete                          |
+| file-deletion-interval| integer | 600     | Interval time in seconds in which the agent should check for deleted files |
+| proxies               | object  |         | Specify proxies e.g. `"proxies": {"https": "localhost:8433"}`              |
+| auth-user             | string  |         | HTTP Basic Auth user                                                       |
+| auth-password         | string  |         | HTTP Basic AUth password                                                   |
+
+### Debug example
+
+```
+{
+  "url": "https://coray.org/htp-test/src/api/server.php", 
+  "token": "7RNDqtnPxm",
+  "uuid": "49dcd31c-3637-4f2a-8df1-b545202df5b3",
+  "debug": true
+}
+```
+
+### rsync
+
+You need a user on the server which can automatically login (e.g. SSH keys) and has read access to the files directory of hashtopolis. On the client side you need rsync installed and set the following lines in your agent config.
+
+```
+  "rsync": true,
+  "rsync-path": "user@yourserver:/path/to/hashtopolis/files"
+```
+
+### Multicast
+
+In order to use the multicast distribution for files, please make sure that the agents and server are prepared according to this:https://github.com/s3inlc/hashtopolis-runner
+
+## Hashcat Compatibility
+
+The list contains all Hashcat versions with which the client was tested and is able to work with (other versions might work):
+
+* 5.0.0
+* 4.2.1
+* 4.2.0
+* 4.1.0
+* 4.0.1
+* 4.0.0
+
+## Generic Crackers
+
+This client is able to run generic Hashtopolis cracker binaries which fulfill the minimal functionality requirements, described [here](https://github.com/s3inlc/hashtopolis/tree/master/doc/README.md). An example implementation can be found [here](https://github.com/s3inlc/hashtopolis-generic-cracker)

python/__main__.py → __main__.py

@@ -1,3 +1,5 @@
+import sys
+import time
 from time import sleep
 
 from htpclient.binarydownload import BinaryDownload
@@ -6,7 +8,7 @@
 from htpclient.generic_cracker import GenericCracker
 from htpclient.hashcat_cracker import HashcatCracker
 from htpclient.hashlist import Hashlist
-from htpclient.helpers import start_uftpd
+from htpclient.helpers import start_uftpd, file_get_contents
 from htpclient.initialize import Initialize
 from htpclient.jsonRequest import *
 from htpclient.dicts import *
@@ -16,26 +18,99 @@
 
 CONFIG = None
 binaryDownload = None
+current_cracker = None
+
+
+def run_health_check():
+    global CONFIG, binaryDownload
+    logging.info("Health check requested by server!")
+    logging.info("Retrieving health check settings...")
+    query = copy_and_set_token(dict_getHealthCheck, CONFIG.get_value('token'))
+    req = JsonRequest(query)
+    ans = req.execute()
+    if ans is None:
+        logging.error("Failed to get health check!")
+        sleep(5)
+        return
+    elif ans['response'] != 'SUCCESS':
+        logging.error("Error on getting health check: " + str(ans))
+        sleep(5)
+        return
+    binaryDownload.check_version(ans['crackerBinaryId'])
+    check_id = ans['checkId']
+    logging.info("Starting check ID " + str(check_id))
+
+    # write hashes to file
+    hash_file = open("hashlists/health_check.txt", "w")
+    hash_file.write("\n".join(ans['hashes']))
+    hash_file.close()
+
+    # delete old file if necessary
+    if os.path.exists("hashlists/health_check.out"):
+        os.unlink("hashlists/health_check.out")
+
+    # run task
+    cracker = HashcatCracker(ans['crackerBinaryId'], binaryDownload)
+    start = int(time.time())
+    [states, errors] = cracker.run_health_check(ans['attack'], ans['hashlistAlias'])
+    end = int(time.time())
+
+    # read results
+    if os.path.exists("hashlists/health_check.out"):
+        founds = file_get_contents("hashlists/health_check.out").replace("\r\n", "\n").split("\n")
+    else:
+        founds = []
+    num_gpus = len(states[0].get_temps())
+    query = copy_and_set_token(dict_sendHealthCheck, CONFIG.get_value('token'))
+    query['checkId'] = check_id
+    query['start'] = start
+    query['end'] = end
+    query['numGpus'] = num_gpus
+    query['numCracked'] = len(founds) - 1
+    query['errors'] = errors
+    req = JsonRequest(query)
+    ans = req.execute()
+    if ans is None:
+        logging.error("Failed to send health check results!")
+        sleep(5)
+        return
+    elif ans['response'] != 'OK':
+        logging.error("Error on sending health check results: " + str(ans))
+        sleep(5)
+        return
+    logging.info("Health check completed successfully!")
 
 
 def init():
     global CONFIG, binaryDownload
-    logformat  = '[%(asctime)s] [%(levelname)-5s] %(message)s'
-    dateformat = '%Y-%m-%d %H:%M:%S'
-    logfile  = 'client.log'
-    loglevel = logging.INFO
+    log_format = '[%(asctime)s] [%(levelname)-5s] %(message)s'
+    print_format = '%(message)s'
+    date_format = '%Y-%m-%d %H:%M:%S'
+    log_level = logging.INFO
+    logfile = open('client.log', "a", encoding="utf-8")
 
     logging.getLogger("requests").setLevel(logging.WARNING)
 
     CONFIG = Config()
     if CONFIG.get_value('debug'):
-        loglevel = logging.DEBUG
+        log_level = logging.DEBUG
         logging.getLogger("requests").setLevel(logging.DEBUG)
-    logging.basicConfig(filename=logfile, level=loglevel, format=logformat, datefmt=dateformat)
-    logging.getLogger().addHandler(logging.StreamHandler())
+    logging.basicConfig(level=log_level, format=print_format, datefmt=date_format)
+    file_handler = logging.StreamHandler(logfile)
+    file_handler.setFormatter(logging.Formatter(log_format))
+    logging.getLogger().addHandler(file_handler)
 
     logging.info("Starting client '" + Initialize.get_version() + "'...")
 
+    session = Session(requests.Session()).s
+    session.headers.update({'User-Agent': Initialize.get_version()})
+
+    if CONFIG.get_value('proxies'):
+        session.proxies = CONFIG.get_value('proxies')
+
+    if CONFIG.get_value('auth-user') and CONFIG.get_value('auth-password'):
+        session.auth = (CONFIG.get_value('auth-user'), CONFIG.get_value('auth-password'))
+
     # connection initialization
     Initialize().run()
     # download and updates
@@ -46,7 +121,7 @@ def init():
 
 
 def loop():
-    global binaryDownload, CONFIG
+    global binaryDownload, CONFIG, current_cracker
 
     logging.debug("Entering loop...")
     task = Task()
@@ -62,7 +137,11 @@ def loop():
         if task.get_task() is not None:
             last_task_id = task.get_task()['taskId']
         task.load_task()
-        if task.get_task() is None:
+        if task.get_task_id() == -1:
+            run_health_check()
+            task.reset_task()
+            continue
+        elif task.get_task() is None:
             task_change = True
             continue
         else:
@@ -85,16 +164,23 @@ def loop():
             logging.info("Got cracker binary type " + binaryDownload.get_version()['name'])
             if binaryDownload.get_version()['name'].lower() == 'hashcat':
                 cracker = HashcatCracker(task.get_task()['crackerId'], binaryDownload)
+                current_cracker = cracker
             else:
                 cracker = GenericCracker(task.get_task()['crackerId'], binaryDownload)
+                current_cracker = cracker
         task_change = False
         chunk_resp = chunk.get_chunk(task.get_task()['taskId'])
         if chunk_resp == 0:
             task.reset_task()
             continue
         elif chunk_resp == -1:
             # measure keyspace
-            cracker.measure_keyspace(task.get_task(), chunk)
+            if not cracker.measure_keyspace(task.get_task(), chunk):  # failure case
+                task.reset_task()
+            continue
+        elif chunk_resp == -3:
+            run_health_check()
+            task.reset_task()
             continue
         elif chunk_resp == -2:
             # measure benchmark
@@ -126,6 +212,12 @@ def loop():
                 logging.info("Server accepted benchmark!")
                 continue
 
+        # check if we have an invalid chunk
+        if chunk.chunk_data() is not None and chunk.chunk_data()['length'] == 0:
+            logging.error("Invalid chunk size (0) retrieved! Retrying...")
+            task.reset_task()
+            continue
+
         # run chunk
         logging.info("Start chunk...")
         cracker.run_chunk(task.get_task(), chunk.chunk_data())
@@ -137,8 +229,11 @@ def loop():
 
 if __name__ == "__main__":
     try:
+        if len(sys.argv) > 1 and sys.argv[1] == '--version':
+            print(Initialize.get_version())
+            sys.exit()
         init()
         loop()
     except KeyboardInterrupt:
         logging.info("Exiting...")
-        exit()
+        sys.exit()

python/build.sh → build.sh

@@ -1,4 +1,6 @@
 #!/usr/bin/env bash
 
-rm hashtopolis.zip
+if [ -f hashtopolis.zip ]; then
+  rm hashtopolis.zip
+fi
 zip -r hashtopolis.zip __main__.py htpclient -x "*__pycache__*"

python/changelog.md → changelog.md

@@ -1,3 +1,19 @@
+## 0.2.0 -> 0.x.x
+
+### Features
+
+* Agents can run health checks requested from the server.
+
+### Bugfixes
+
+* Fixed benchmark results when having many GPUs in one agent.
+
+### Enhancements
+
+* Added check for chunk length 0 sent from the server to avoid full agent crash on running.
+* Using requests session
+* Added option for using proxies
+* Added HTTP Basic Auth support
 
 ## 0.1.8 -> 0.2.0