Skip to content
Permalink
Browse files

check.py and friends

  • Loading branch information
Collin Mulliner
Collin Mulliner committed Dec 6, 2019
1 parent 7a77fdd commit 8f05f2dd23ec82476f23970a994fcf6b5956b32a
Showing with 352 additions and 2 deletions.
  1. +1 −0 .gitignore
  2. +42 −0 Changelog.md
  3. +4 −2 Readme.md
  4. +66 −0 devices/Readme.md
  5. +29 −0 devices/android/unpack.sh
  6. +210 −0 devices/check.py
@@ -0,0 +1 @@
build/**
@@ -0,0 +1,42 @@
# Change Log

## Unreleased

### Added
- NEW _cpiofs_ for cpio as filesystem
- NEW universal _check.py_ (so you just need to write a custom unpacker)
- NEW _android/unpack.sh_ (for _check.py_)
- better options for scripts (FileContent and DataExtract)

### Fixed
- $PATH in makefile
- FileContent file iterator
- _squashfs_ username parsing

## [v1.2.0] - 2019-11-19

### Changed
- moved to go 1.13
- only store _current_file_treepath_ if filetree changed

## [v.1.1.0] - 2019-10-15

### Added
- NEW FileCmp check for full file diff against 'old' version
- allow multiple matches for regex based DataExtract

### Fixed
- squashfs username parsing

## [v.1.0.1] - 2019-09-19

### Fixed
- filename for BadFiles check output

## [v.1.0.0] - 2019-08-15

### Added
- CI
- Build instructions

## [initial] - 2019-08-05
@@ -80,10 +80,12 @@ Example for using custom scripts stored in the `scripts` directory:
PATH=$PATH:./scripts fwanalyzer -cfg system_fwa.toml -in system.img -out system_check_output.json
```

The [devices](devices/) folder contains helper scripts for unpacking and dealing with specific devices types and firmware package formats such as [Android](devices/android).
The [devices/](devices/) folder contains helper scripts for unpacking and dealing with specific device types and firmware package formats such as [Android](devices/android).
It also includes general configuration files that can be included in target specific FwAnalyzer configurations.

The [scripts](scripts/) folder contains helper scripts that can be called from FwAnalyzer for file content analysis and data extraction.
Check.py in the [devices/](devices) folder provides a universal script to effectively use FwAnalyzer, see [devices/Readme.md](devices/Readme.md) for details. This likely is how most people will invoke FwAnalyzer.

The [scripts/](scripts/) folder contains helper scripts that can be called from FwAnalyzer for file content analysis and data extraction.

## Config Options

@@ -1,3 +1,69 @@
# Devices

This directory contains support tools and popular checks that can be included in FwAnalyzer configs for multiple targets.

- [Android](android)
- [generic Linux](generic)

## Check.py

check.py is a universal script to run FwAnalyzer. It will unpack (with the help of a unpacker; see below) firmware
and run fwanalyzer against each of the target filesystems, it will combine all of the reports
into one big report. In addition it will do some post processing of the filetree files (if present) and
append the result to the report.

Using check.py is straight forward (the example below is for an Android OTA firmware - make sure you have the required Android unpacking tools installed and added to your PATH, see: [Android](android/Readme.md)):

```sh
check.py --unpacker android/unpack.sh --fw some_device_ota.zip --cfg-path android --cfg-include android --fwanalyzer-bin ../build/fwanalyzer
```

The full set of options is described below:
```
usage: check.py [-h] --fw FW --unpacker UNPACKER --cfg-path CFG_PATH
[--cfg-include-path CFG_INCLUDE_PATH] [--report REPORT]
[--keep-unpacked] [--fwanalyzer-bin FWANALYZER_BIN]
[--fwanalyzer-options FWANALYZER_OPTIONS]
optional arguments:
-h, --help show this help message and exit
--fw FW path to firmware file OR path to unpacked firmware
--unpacker UNPACKER path to unpacking script
--cfg-path CFG_PATH path to directory containing config files
--cfg-include-path CFG_INCLUDE_PATH
path to config include files
--report REPORT report file
--keep-unpacked keep unpacked data
--fwanalyzer-bin FWANALYZER_BIN
path to fwanalyzer binary
--fwanalyzer-options FWANALYZER_OPTIONS
options passed to fwanalyzer
```

The _--keep-unpacked_ option will NOT delete the temp directory that contains the unpacked files.
Once you have the unpacked directory you can pass it to the _--fw_ option to avoid unpacking the
firmware for each run (e.g. while you test/modify your configuration files). See the example below.

```sh
check.py --unpacker android/unpack.sh --fw /tmp/tmp987689123 --cfg-path android --cfg-include android --fwanalyzer-bin ../build/fwanalyzer
```

### unpacker

The unpacker is used by check.py to _unpack_ firmware.
The unpacker needs to be an executable file, that takes two parameters first the `file` to unpack
and second the `path to the config files` (the path that was provided via --cfg-path).

The unpacker needs to output a set of targets, the targets map a config file to a filesystem image (or directory).
The targets are specified as a JSON object.

The example below specifies two targets:

- system : use _system.toml_ when analyzing _system.img_
- boot: use _boot.toml_ when analyzing the content of directory _boot/_

```json
{ "system": "system.img" , "boot": "boot/" }
```

See [Android/unpack.sh](android/unpack.sh) for a real world example.
@@ -0,0 +1,29 @@
#!/bin/sh

# -- unpack android OTA --

if [ -z "$1" ]; then
echo "syntax: $0 <android_ota.zip>"
exit 1
fi
OTAFILE=$1

# tmpdir should contained 'unpacked' as last path element
TMPDIR=$(pwd)
if [ "$(basename $TMPDIR)" != "unpacked" ]; then
echo "run script in directory named 'unpacked'"
exit 1
fi

# unpack
unzip $OTAFILE >../unpack.log 2>&1
extract_android_ota_payload.py payload.bin >>../unpack.log 2>&1
mkboot boot.img boot_img >>../unpack.log 2>&1

# output targets, targets are consumed by check.py
# key = name of fwanalyzer config file without extension
# e.g. 'system' => will look for 'system.toml'
# value = path to filesystem image (or directory)

# analyze system.img using system.toml
echo -n '{ "system": "unpacked/system.img" }'
@@ -0,0 +1,210 @@
#!/usr/bin/env python3

# Copyright 2019 GM Cruise LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import argparse
import hashlib
import json
import os
import sys
import subprocess
import tempfile

class CheckFirmware:
def __init__(self, fwanalyzer="fwanalyzer"):
self._tmpdir = ""
self._unpackdir = ""
self._fwanalyzer = fwanalyzer
self._unpacked = False

def get_tmp_dir(self):
return self._tmpdir

def run_fwanalyzer_fs(self, img, cfg, cfginc, out, options=""):
cfginclude = ""
if cfginc:
cfginclude = " -cfgpath {0}".format(cfginc)
cmd = "{0} -in {1} {2} -cfg {3} -out {4} {5}".format(self._fwanalyzer, img, cfginclude, cfg, out, options)
return subprocess.check_call(cmd, shell=True)

def unpack(self, fwfile, unpacker, cfgpath):
TARGETS_FILE = "targets.json"
try:
if os.path.exists(os.path.join(fwfile, "unpacked")) and os.path.exists(os.path.join(fwfile, TARGETS_FILE)):
self._tmpdir = fwfile
self._unpackdir = os.path.join(self._tmpdir, "unpacked")
print("{0}: is a directory containing an 'unpacked' path, skipping".format(fwfile))
cmd = "cat {0}".format(os.path.join(fwfile, TARGETS_FILE))
self._unpacked = True
else:
self._tmpdir = tempfile.mkdtemp()
self._unpackdir = os.path.join(self._tmpdir, "unpacked")
os.mkdir(self._unpackdir)
cmd = "{0} {1} {2}".format(unpacker, fwfile, cfgpath)
res = subprocess.check_output(cmd, shell=True, cwd=self._unpackdir)
targets = json.loads(res.decode('utf-8'))
with open(os.path.join(self._tmpdir, TARGETS_FILE), "w") as fp:
fp.write(res.decode('utf-8'))
return targets
except Exception as e:
print("Exception: {0}".format(e))
print("can't load targets from output of '{0}' check your script".format(unpacker))
return None

def del_tmp_dir(self):
if not self._unpacked:
cmd = "rm -rf {0}".format(self._tmpdir)
return subprocess.check_call(cmd, shell=True)

def files_by_ext_stat(self, data):
allext = {}
for i in data["files"]:
fn, ext = os.path.splitext(i["name"])
if ext in allext:
count, ext = allext[ext]
allext[ext] = count + 1, ext
else:
allext[ext] = (1, ext)
return (len(data["files"]), allext)

def analyze_filetree(self, filetreefile):
with open(filetreefile) as fp:
data = json.load(fp)
num_files, stats = self.files_by_ext_stat(data)
out = {}
percent = num_files / 100
# only keep entries with count > 1% and files that have an extension
for i in stats:
(count, ext) = stats[i]
if count > percent and ext != "":
out[ext] = (count, ext)

return {
"total_files": num_files,
"file_extension_stats_inclusion_if_more_than": percent,
"file_extension_stats": sorted(out.values(), reverse=True)
}

# check result and run post analysis
def check_result(self, result):
with open(result) as read_file:
data = json.load(read_file)

if "offenders" in data:
status = False
else:
status = True

CURRENT_FILE_TREE = "current_file_tree_path"

if CURRENT_FILE_TREE in data:
if os.path.isfile(data[CURRENT_FILE_TREE]):
data["file_tree_analysis"] = self.analyze_filetree(data[CURRENT_FILE_TREE])

return (status, json.dumps(data, sort_keys=True, indent=2))


def hashfile(fpath):
m = hashlib.sha256()
with open(fpath, "rb") as f:
while True:
data = f.read(65535)
if not data:
break
m.update(data)
return m.hexdigest()


# make report from image reports
def make_report(fwfile, data):
report = {}
status = True
for key in data:
img_status, img_report = out[key]
if not status:
status = img_status
report[key] = json.loads(img_report)
report["firmware"] = fwfile
if os.path.isfile(fwfile):
report["firmware_digest"] = hashfile(fwfile)
report["status"] = status
return json.dumps(report, sort_keys=True, indent=2)


if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--fw", action="store", required=True, help="path to firmware file OR path to unpacked firmware")
parser.add_argument("--unpacker", action="store", required=True, help="path to unpacking script")
parser.add_argument("--cfg-path", action="store", required=True, help="path to directory containing config files")
parser.add_argument("--cfg-include-path", action="store", help="path to config include files")
parser.add_argument("--report", action="store", help="report file")
parser.add_argument("--keep-unpacked", action="store_true", help="keep unpacked data")
parser.add_argument("--fwanalyzer-bin", action="store", default="fwanalyzer", help="path to fwanalyzer binary")
parser.add_argument("--fwanalyzer-options", action="store", default="", help="options passed to fwanalyzer")
args = parser.parse_args()

fw = os.path.realpath(args.fw)
cfg = os.path.realpath(args.cfg_path)

check = CheckFirmware(args.fwanalyzer_bin)
targets = check.unpack(fw, os.path.realpath(args.unpacker), cfg)
print("using tmp directory: {0}".format(check.get_tmp_dir()))
if not targets:
print("no targets defined")
sys.exit(1)

# target file system images, a fwanalyzer config file is required for each of those
for tgt in targets:
cfg_file_name = "{0}.toml".format(tgt)
if not os.path.isfile(os.path.join(args.cfg_path, cfg_file_name)):
print("skipped, config file '{0}' for '{1}' does not exist\n".format(
os.path.join(args.cfg_path, cfg_file_name), targets[tgt]))
sys.exit(0)
else:
print("using config file '{0}' for '{1}'".format(
os.path.join(args.cfg_path, cfg_file_name), targets[tgt]))

out = {}
all_checks_ok = True
for tgt in targets:
cfg_file_name = "{0}.toml".format(tgt)
out_file_name = "{0}_out.json".format(tgt)
check.run_fwanalyzer_fs(os.path.join(check.get_tmp_dir(), targets[tgt]),
os.path.join(cfg, cfg_file_name), args.cfg_include_path, out_file_name,
options=args.fwanalyzer_options)
ok, data = check.check_result(out_file_name)
out[tgt] = ok, data
if not ok:
all_checks_ok = False

if args.keep_unpacked:
print("unpacked: {0}\n".format(check.get_tmp_dir()))
else:
check.del_tmp_dir()

report = make_report(args.fw, out)
if args.report != None:
with open(args.report, "w+") as fp:
fp.write(report)
print("report written to '{0}'".format(args.report))
else:
print(report)

if not all_checks_ok:
print("Firmware Analysis: checks failed")
sys.exit(1)
else:
print("Firmware Analysis: checks passed")
sys.exit(0)

0 comments on commit 8f05f2d

Please sign in to comment.
You can’t perform that action at this time.