Initial support for macOS

[offlinemark]

we've gotten a few requests from users to be able to have Manticore run on OS X. this shouldn't be too hard as nothing we do inherently ties us to linux. we'd need to

• eliminate all uses of /dev (i think there are a few)
• make sure all python stdlib calls we make work uniformly on linux/osx
• make sure python dependencies work on both
• add features to allow users to specify locations for ld.so and libraries (Add LD_LIBRARY_PATH type feature #93)

[tdeebswihart]

What should be done to verify proper execution? I've some small fixes to the tests that ensure they work on Mac OS, and I'm working through the linux examples and they seem to work just fine.

[offlinemark]

The example binaries in examples/linux are configured to be statically linked, but dynamic binaries will not work on macOS. I also just added that to the comment above

[catenacyber]

I am using MacOS and managing to run manticore, but maybe not all the features.
The only hack I did is the pull request above.

What is left to be done here ?

[offlinemark]

@catenacyber check out the top comment, primarily #93 is the big one :)

[catenacyber]

Thanks @Mossberg : I see that I will need to improve much my pull request then

[catenacyber]

After closing #93, what should be next ? support Mach-O format ?

[offlinemark]

at this point since #93 is closed, i think the main thing is making sure all of those checkboxes in the top comment are completed, and doing testing of running manticore on mac. analyzing mach-o binaries is out of scope of this issue, we are simply looking to have mac users able to use manticore on their computers :)

[catenacyber]

Ok so first one

eliminate all uses of /dev (i think there are a few)

doing grep -r "/dev" manticore
I only see one : manticore/platforms/decree.py: data = file("/dev/urandom", "r").read(count)

But I have my /dev/urandom on macOS as well

Is this ok to you ?

[offlinemark]

ah right, macos is unixy 😅 that looks fine to me

[catenacyber]

Second one

make sure all python stdlib calls we make work uniformly on linux/osx

Maybe #897 is part of this, what do you think @Mossberg ?

[offlinemark]

yup, i'd say so

[catenacyber]

Any ideas on how to test all stdlib calls ?

[offlinemark]

hm, i'm not totally sure. i think it might just be good enough to do a cursory review of the codebase and document any stdlib calls we make that are particularly suspect for non-portability.

[catenacyber]

Hmm I googled "site:https://docs.python.org/3/library/ macos" and looked out in python documentation which calls would be concerned.
I found functions such as os.getgroups but none used in manticore...
So, I guess that we can tick the second box.
How we can tick the third one (dependencies) ? trying a fresh install is enough ?

[offlinemark]

Hm, i'm thinking that if our goal is to have beta/experimental support for macos it doesn't have to work absolutely perfectly. If tests pass on macos maybe that would be enough to announce beta macos support and resolve this issue. then we can fix individual macos bugs as they are reported.

[catenacyber]

Hmm...

manticore/utils/event.py", line 38
    class Eventful(object, metaclass=EventsGatherMetaclass):
                                    ^
SyntaxError: invalid syntax

I will look into it

[offlinemark]

You need to run with py 3.6 now :)

…

On Mon, Aug 6, 2018 at 12:04 PM Catena cyber ***@***.***> wrote: Hmm... manticore/utils/event.py", line 38 class Eventful(object, metaclass=EventsGatherMetaclass): ^ SyntaxError: invalid syntax I will look into it — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#110 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACWmG24WXLuE54dHHFznOI4Jmq-q45iJks5uOGkjgaJpZM4MqgQL> .

[catenacyber]

Ok, now I am getting

manticore/core/workspace.py", line 52, in serialize
    resource.setrlimit(resource.RLIMIT_STACK, [0x100 * maxlim, resource.RLIM_INFINITY])
ValueError: not allowed to raise maximum limit

[defunctio]

i was in the process of cleaning that stuff up, you can remove the setrlimit.

see; dev-pickle-rick

[catenacyber]

Thanks @defunctio I was trying to keep it, but make it work with reasonable values on Mac

[catenacyber]

Next error is

manticore.ethereum.EthereumError: Solidity compiler not installed.

Maybe this can be added to the wiki
https://github.com/trailofbits/manticore/wiki/Hacking-on-Manticore

[defunctio]

I was thinking not too long ago maybe we should consider using py-solc instead of our own wrapper. This does come with an added benefit of solc version management for both linux and osx

[catenacyber]

maybe we should consider using py-solc instead of our own wrapper.

indeed but I think that deserves another issue

As for Mac, here is the next error :

======================================================================
ERROR: testCreating (tests.test_driver.ManticoreDriverTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "manticore/tests/test_driver.py", line 27, in testCreating
    m = Manticore('/bin/ls')
  File "manticore/manticore/manticore.py", line 188, in __init__
    self._initial_state = make_initial_state(path_or_state, argv=argv, **kwargs)
  File "manticore/manticore/manticore.py", line 138, in make_initial_state
    raise NotImplementedError("Binary {} not supported.".format(binary_path))
NotImplementedError: Binary /bin/ls not supported.

Indeed, my /bin/ls is a Mach-O file
Should a linux /bin/ls binary be added to the manticore tests ?

We come close to the end
(+ one path patch for the tests)

[defunctio]

Unless that was intended to test dynamically linked binaries (which I don't think it is?) you could probably just switch that to use tests/binaries/basic_linux_amd64 or something

[catenacyber]

Unless that was intended to test dynamically linked binaries (which I don't think it is?)

This comment shows that this may be the case :

'''
    TODO(mark): these tests assumes /bin/ls is a dynamic x64 binary
    '''

[defunctio]

I would probably just do some platform checks for now and if the test environment is OSX and skip tests that rely on dynamically linked ELF targets and leave a #TODO/#FIXME behind. The CI will end up testing for it anyways.

Anyone else may have a more elegant solution?

[catenacyber]

@Mossberg you wrote the above comment (these tests assumes /bin/ls is a dynamic x64 binary)

Do you have a better proposal than @defunctio ?

PS : the following tests use /bin/ls, even if only test_linux.py has the comment :

• tests/test_driver.py
• tests/test_linux.py
• tests/test_models.py
• tests/test_state.py
• tests/test_unicorn.py
• tests/test_workspace.py

[offlinemark]

for all of these, /bin/ls was probably just chosen as a convenient linux ELF binary to create an initial Linux state. i don't think there was any particular requirement for it to be dynamically linked, and we have a few binaries in the repo that should do the job like defunct mentioned (tests/binaries/). i think we should prob just change all these /bin/ls references to one of these tests/binaries, as opposed to doing platform specific checks. i'd like to avoid those :)

[catenacyber]

And now it works with #1032 :-)

Ran 5407 tests in 459.774s

OK (SKIP=5)

[offlinemark]

@catenacyber a question about your environment with those tests running; are you on python 3.6? and also was unicorn installed? i am having issues getting tests to pass, and it's mostly because unicorn does not seem to install in my macOS python 3.6 environment

specifically, i hit this error when installing unicorn

https://github.com/unicorn-engine/unicorn/blob/0109cd6c8a2b268d0aaf9e597752682508c7ffed/qemu/configure#L555-L561

i can install unicorn in a Linux python3.6 environment because it seems there is a wheel for it, so there is no need to compile.

[catenacyber]

I am running Python 3.7.0
And I seem to have unicorn module installed

> pip freeze | grep unicorn
unicorn==1.0.1

I do not remember installing it though

[disconnect3d]

Regarding dependencies, on latest Mac OS X (Mojave) - to install capstone and unicorn one has to do:

brew install capstone && export MACOS_UNIVERSAL=no && pip install capstone
brew install unicorn && UNICORN_QEMU_FLAGS="--python=`whereis python`" pip install unicorn`

[1] re capstone: capstone-engine/capstone#1235 - the solution proposed by Inndy works fine
[2] re unicorn: unicorn-engine/unicorn#206 - somehow setting the variable as a python 3 interpreter works fine, lol

[disconnect3d]

I added the description to https://github.com/trailofbits/manticore#installation , we probably can't enforce this on setup.py so I assume this is all we can do here.

[catenacyber]

so I assume this is all we can do here

Could we make that unicorn supports python3 ? (without needing the trick)