Skip to content
The concepting self hosted executable binary search engine
Branch: master
Clone or download
ChiChou workaround: r2 does not yield JSON format
radare2 yields en empty string, which is expcted to be '{}'
Latest commit feaea57 May 26, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
agent workaround: r2 does not yield JSON format May 27, 2019
docker public release May 10, 2019
presets public release May 10, 2019
web public release May 10, 2019
.gitignore public release May 10, 2019
LICENSE public release May 10, 2019
Pipfile bugfix: missing r2pipe dependency May 17, 2019
Pipfile.lock bugfix: missing r2pipe dependency May 17, 2019
README.md Update README.md May 18, 2019
docker-compose.yml public release May 10, 2019
kibana.yml public release May 10, 2019
scan.py public release May 10, 2019
screenshot.png public release May 10, 2019

README.md

Wiggle

Screenshot

An executable binary metadata search engine. Currently MachO only.

Prerequisite

  • docker and docker-compose
  • Python3 (no support for Python2)
  • radare2 (on collector)

Usage

  1. Run the collector

    Install dependencies first:

    pipenv install

    Note: docker is not required to run the collector.

    python3 scan.py /System/Library/Frameworks /System/Library/PrivateFrameworks /usr/lib /usr/bin /usr/sbin /sbin /usr/libexec

    Or specifying a file that includes the directories you want to scan

    python3 scan.py --rule presets/app.txt

    For mounted filesystem, you can set the filesystem root by specifying --sysroot.

    To skip some the directories, use --block block_list.txt (wildcard not supported yet)

  2. Start the server

    docker-compose up -d
  3. Migrate the data to the full text search

    # todo: docker mount read only
    docker-compose run web python /agent/indexer.py 10.14.2 /agent/archive.db
  4. Open localhost:8000 in the browser

FAQ

Why doesn't the collector directly save the document, but using a SQLite database pipeline instead?

So the collector and the server don't have to be the same machine.

pipenv install failed to install LIEF

Follow the instruction here: https://lief.quarkslab.com/doc/stable/installation.html

Download the zip package from https://github.com/lief-project/packages/tree/lief-master-latest

pipenv shell; pip install [pylief-x.x.x.dev.zip]

TODO

  • Backend
    • Support other formats: ELF, PE etc
    • Support dyld_shared_cache
    • Consider moving to postgres
  • UI
    • Autocomplete
    • AJAX

Known Issue

Stability

The collector is extermely unstable now, the scanning process can be interrupted any time

Nested documents limitation

The number of nested documents has exceeded the allowed limit of [10000]. This limit can be set by changing the [index.mapping.nested_objects.limit] index level setting.

Need to improve the database design the overcome this instead of just changing the value

Numeric value -1

Numeric value (12727174745972277247) out of range of long (-9223372036854775808 - 9223372036854775807)

Some of the values from radare2 are 0xffffffffffffffff that cause the exception

Performance

Single node ElasticSearch sucks. Really. Should consider migrating to postgreSQL.

Acknowledgement

Wiggle is based on these awesome open source projects:

You can’t perform that action at this time.