An executable binary metadata search engine. Currently MachO only.
- docker and docker-compose
- Python3 (no support for Python2)
- radare2 (on collector)
Run the collector
Install dependencies first:
Note: docker is not required to run the collector.
python3 scan.py /System/Library/Frameworks /System/Library/PrivateFrameworks /usr/lib /usr/bin /usr/sbin /sbin /usr/libexec
Or specifying a file that includes the directories you want to scan
python3 scan.py --rule presets/app.txt
For mounted filesystem, you can set the filesystem root by specifying
To skip some the directories, use
--block block_list.txt(wildcard not supported yet)
Start the server
docker-compose up -d
Migrate the data to the full text search
# todo: docker mount read only docker-compose run web python /agent/indexer.py 10.14.2 /agent/archive.db
Open localhost:8000 in the browser
Why doesn't the collector directly save the document, but using a SQLite database pipeline instead?
So the collector and the server don't have to be the same machine.
pipenv install failed to install LIEF
Follow the instruction here: https://lief.quarkslab.com/doc/stable/installation.html
Download the zip package from https://github.com/lief-project/packages/tree/lief-master-latest
pipenv shell; pip install [pylief-x.x.x.dev.zip]
- Support other formats: ELF, PE etc
- Support dyld_shared_cache
- Consider moving to postgres
The collector is extermely unstable now, the scanning process can be interrupted any time
Nested documents limitation
The number of nested documents has exceeded the allowed limit of . This limit can be set by changing the [index.mapping.nested_objects.limit] index level setting.
Need to improve the database design the overcome this instead of just changing the value
Numeric value -1
Numeric value (12727174745972277247) out of range of long (-9223372036854775808 - 9223372036854775807)
Some of the values from radare2 are
0xffffffffffffffff that cause the exception
Single node ElasticSearch sucks. Really. Should consider migrating to postgreSQL.
Wiggle is based on these awesome open source projects: