The usage of the unsafe library in Go allows developers to circumvent its memory
protection and can introduce security vulnerabilities. go-geiger helps developers
to spot usages of unsafe in their code. Machine learning can be used to classify
the reason and context of this usage.
This toolkit should provide a wrapper / Docker container for https://github.com/Cortys/unsafe-go-classifier. Snippets of Go code should be given as a parameter for a container. The container will then analyze the code for unsafe usages and try to classify it.
- Installation (local)
- Installation (Docker)
- Running the Runner Script for Docker
- Arguments for the Evaluation
- Arguments for the Visualizer
- Usage Examples
- Testing
- Audit Examples for Unsafe Usages
- In-depth Resources about Unsafe Usages
- Classifier
You should have the unsafe-go-classifier image downloaded and tagged as usgoc/pred:latest. Pull the unsafe-go-classifier from https://github.com/Cortys/unsafe-go-classifier.
Also, install go-geiger and make sure it's located in one of your path variables.
To install the Python dependencies, run the following command pip install -r requirements.txt.
You may want to install these packages in a local environment instead of global: $python3 -m venv .venv $source .venv/bin/activate.
If you want to pull SSH repositories with this tool, make sure you have working SSH access.
You should have the unsafe-go-classifier image downloaded and tagged as usgoc/pred:latest. Pull the unsafe-go-classifier from https://github.com/Cortys/unsafe-go-classifier. Also, make sure you execute the script with a user which has access to Docker.
Execute the following command to build the image:
sudo docker build . -t unsafe-go-toolkit
Run the run.py file with the following arguments to export analysis data from a file/project:
usage: run.py [-h] -p PROJECT [-o OUTPUT] [-v VISUALIZER_ARGS] [-d]
optional arguments:
-h, --help show this help message and exit
-p PROJECT, --project PROJECT
Project path
-o OUTPUT, --output OUTPUT
Output path
-v VISUALIZER_ARGS, --visualizer-args VISUALIZER_ARGS
Arguments for the visualizer as a string, use of input argument is not recommended
-d, --debug Verbose mode
The visualizer args should be given in quotes and will then be passed in the container. Note that the output should be in the mounted output directory, because the run.py script mounts only that directory to the host files system.
Example usage:
./run.py -p https://github.com/jlauinger/go-safer.git
Run the evaluate.py file with the following arguments to export analysis data from a file/project:
usage: evaluate.py [-h] [-p PROJECT] [-o OUTPUT] [-m MODE] [-d] [-c CONCURRENT_THREADS]
optional arguments:
-h, --help show this help message and exit
-p PROJECT, --project PROJECT
Path of package where the Go file lies in
-o OUTPUT, --output OUTPUT
Output file of JSON file
-m MODE, --mode MODE Mode of output file, choose between the strings readable or machine
-d, --debug Debug mode
-c CONCURRENT_THREADS, --concurrent-threads CONCURRENT_THREADS
Number of concurrent evaluation containers the script should run
Example usage:
./evaluate.py -p https://github.com/jlauinger/go-safer.git
This command analyzes the passed repository (-p) for unsafe usages, passes each unsafe usage into the classifier, and write the classification results for the complete project in the file output.json in the folder output.
Run the visualize.py with the following arguments to visualize your acquired analysis:
usage: visualize.py [-h] -i INPUT [-o OUTPUT] [-t TYPE]
optional arguments:
-h, --help show this help message and exit
-i INPUT, --input INPUT
Path of input JSON file
-o OUTPUT, --output OUTPUT
Path of output visualized folder
-t TYPE, --type TYPE File type of output graphs
./visualize.py -i output/output.json /output
This example runs the analysis on the go-safer repository and saves the data on a custom file location.
./run.py -p https://github.com/jlauinger/go-safer.git -o output/go_safer.json
This example runs the same analysis as above, but with custom visualizer args.
./run.py -p https://github.com/jlauinger/go-safer.git -o output/go_safer.json --visualizer-args "-t svg"
This project can be tested using the tests.py file and the following command:
python3 -m unittest tests.py
You can also run the tests in Visual Studio Code, the test settings have been preconfigured.
Some tests are version and package specific, so the paths for the tests should be updated to the corresponding packages.
Our tool can guide the process of auditing unsafe usages by categorizing usages.
Existing linter can identify an unsafe usage while their lack to provide more detailed information about their porpuse.
One linter in this category is gosec that provides an option to flag false positives with #nosec.
Optinally, one can add the rule to the comment, such as G103 for the rule that identify unsafe usages.
Thus, this comment helps to identify examples of unsafe usages that have been analyzed by a linter and manually verified.
A simple and fast query to github results in about 370 different Go-files that make use of #nosec G103: https://github.com/search?l=&q=%2F%2F%23nosec+G103+language%3AGo&type=code.
Note, that this query is via the GitHub Search API and result in incomplete and may differentiating results.
Several members of the Go community are engaged in sharing their knowledge and insights about the unsafe* API. Below, we list a few of these resources. In case, you think we missed one worthwhile reading or watching, feel free to open an issue/pull request to get it merged. The Table is ordered alphabetical.
| Author | Where | Title | URL | Date | Last visit |
|---|---|---|---|---|---|
| Bowes, J. | dotGo 2019 | Shattered Mirror: An Introduction to Reflect and Unsafe | YouTube | Mar, 25 2019 | Jan, 13 2023 |
| Gopher Academy Blog | Blog | Safe use of unsafe.Pointer | Blog | Dec, 5 2019 | Jan, 13 2023 |
| Kochetkov, A. | Hackernoon | Golang Unsafe Type Conversions and Memory Access | Hackernoon | Mar, 15 2020 | Jan, 13 2023 |
| Lauinger, J. | dev.to | Exploitation Exercise with unsafe.Pointer in Go: Information Leak (Part 1) | dev.to | May, 13 2020 | Jan, 13 2023 |
| Walker, J. | GopherCon 2020 | Safety Not Guaranteed: Calling Windows APIs using Unsafe & Syscall | YouTube | Dec, 22 2020 | Jan, 13 2023 |
| Wickert, A. | BSides Berlin 2020 | Go is memory safe isn't it? | YouTube | Sep, 20 2020 | Jan, 13 2023 |
- The implementation of classifier is available in this Cortys/unsafe-go-classifier GitHub repository and archived via figshare. The Docker container is available via GitHub and archived via figshare.
- Fork of
unsafe_go_study_resultthat includes our CFG generation implementation along with the data used for labelling is available in this Cortys/unsafe_go_study_results GitHub repository.