Ray/Cloudpickle/dis issue with Pyarmor

[PidgeyBE]

Hi

I'm trying to run obfuscated python packages via ray (https://github.com/ray-project/ray/).
I got however stuck on this line: https://github.com/ray-project/ray/blob/master/python/ray/function_manager.py#L118 which uses the python dis package.
I was trying to create a minimal example to reproduce the issue, but then I got stuck on https://github.com/ray-project/ray/blob/master/python/ray/cloudpickle/cloudpickle.py#L440.
So it seems both Ray and Cloudpickle are incompatible with Pyarmor because they are using dis.

Method to reproduce the dis issue with cloudpickle:

simple_example.py

import ray
@ray.remote
class Counter(object):
    def __init__(self):
        self.value = 0
    def increment(self):
        self.value += 1
        return self.value

-> pyarmor obfuscate simple_example.py

dist/run_example.py

import ray
from simple_example import Counter

ray.init()
c = Counter.remote()
c.increment.remote()

python dist/run_example.py

Error traceback

Traceback (most recent call last):
  File "run_example.py", line 5, in <module>
    c = Counter.remote()
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/actor.py", line 378, in remote
    return self._remote(args=args, kwargs=kwargs)
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/actor.py", line 524, in _remote
    meta.method_meta.methods.keys())
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/function_manager.py", line 358, in export_actor_class
    "class": pickle.dumps(Class),
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 72, in dumps
    cp.dump(obj)
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 617, in dump
    return Pickler.dump(self, obj)
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 559, in reducer_override
    return self._function_reduce(obj)
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 588, in _function_reduce
    return self._dynamic_function_reduce(obj)
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 569, in _dynamic_function_reduce
    state = _function_getstate(func)
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 123, in _function_getstate
    f_globals_ref = _extract_code_globals(func.__code__)
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle.py", line 212, in _extract_code_globals
    out_names = {names[oparg] for _, oparg in _walk_global_ops(co)}
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle.py", line 212, in <setcomp>
    out_names = {names[oparg] for _, oparg in _walk_global_ops(co)}
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle.py", line 440, in _walk_global_ops
    for instr in dis.get_instructions(code):
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/dis.py", line 308, in _get_instructions_bytes
    argval, argrepr = _get_name_info(arg, names)
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/dis.py", line 272, in _get_name_info
    argval = name_list[name_index]
IndexError: tuple index out of range

Method to reproduce the dis issue with ray:

simple_example.py

class Counter(object):
    def __init__(self):
        self.value = 0
    def increment(self):
        self.value += 1
        return self.value

-> pyarmor obfuscate simple_example.py

dist/run_example.py

import ray
from simple_example import Counter

ray.init()
c = ray.remote(Counter).remote()
c.increment.remote()

python dist/run_example.py

Error traceback

Traceback (most recent call last):
  File "run_example.py", line 5, in <module>
    c = ray.remote(Counter).remote()
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/actor.py", line 378, in remote
    return self._remote(args=args, kwargs=kwargs)
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/actor.py", line 524, in _remote
    meta.method_meta.methods.keys())
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/function_manager.py", line 360, in export_actor_class
    "collision_identifier": self.compute_collision_identifier(Class),
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/function_manager.py", line 118, in compute_collision_identifier
    dis.dis(function_or_class, file=string_file)
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/dis.py", line 55, in dis
    dis(x1, file=file)
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/dis.py", line 55, in dis
    dis(x1, file=file)
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/dis.py", line 60, in dis
    disassemble(x, file=file)
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/dis.py", line 335, in disassemble
    co.co_consts, cell_names, linestarts, file=file)
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/dis.py", line 346, in _disassemble_bytes
    line_offset=line_offset):
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/dis.py", line 308, in _get_instructions_bytes
    argval, argrepr = _get_name_info(arg, names)
  File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/dis.py", line 272, in _get_name_info
    argval = name_list[name_index]
IndexError: tuple index out of range

Is there a way to overcome this?

Best regards, Pieterjan

[jondy]

Thanks for your report in details, it's great. I found the problem in the traceback,

File "/home/pieterjan/miniconda3/envs/sdk-test/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle.py", line 440, in _walk_global_ops
  for instr in dis.get_instructions(code):

Here dis try to get the byte codes from the obfuscated code objects. It's what pyarmor prevents.

One solution is to obfuscate the scripts with obf_code=0, but it's less security. Refer to obfuscate scripts with different modes by project

[PidgeyBE]

Hi @jondy

Thanks for you quick reply!
We are using https://pyarmor.readthedocs.io/en/latest/advanced.html#call-pyarmor-from-python-script.
Is there a way to add the obf_code argument to pyarmor obfuscate?

[jondy]

One simple way is to patch utils.py in the pyarmor source package (about line 800)

- def encrypt_script(pubkey, filename, destname, wrap_mode=1, obf_code=1,
+ def encrypt_script(pubkey, filename, destname, wrap_mode=1, obf_code=0, 

[PidgeyBE]

Are there other ways?
Because I've made a script that packages multiple python packages, but I only want obf_code=0 for one of them. The others should stay 1...

[jondy]

It also could create a project to build this package in the script:

from pyarmor.pyarmor import main as call_pyarmor
pkgpath = '/path/to/package'
call_pyarmor(['init', '--src', pkgpath, '--entry', '__init__.py', pkgpath])
call_pyarmor(['config', '--obf-code', '0', pkgpath])
call_pyarmor(['build', '-B', '--output', '/path/to/dist', pkgpath])

[PidgeyBE]

I'm trying to use the above snippet, but seems my code is not obfuscated at all. I assume I need an alternative for the --resursive flag using in pyarmor obfuscate but I don't manage to find it...
Is there a way to pyarmor recursively using the method above?

I've tried different ways of setting srcand entry in the .pyarmor_config, my manifest states global-include *.py, but still the output returns

INFO 0 scripts has been obfuscated

[jondy]

Maybe it's better to apply this patch d5ad20d
After that pass --obf-code 0 to command obfuscate as before.

[PidgeyBE]

Looks great! Will this be in a release anytime soon? Then I rather wait for a release instead of hacking around now :)

[jondy]

It may releases in a couple of weeks, I'm developing another feature, and plan to release a minor version with this new feature, I'm not sure the exact date.

[PidgeyBE]

Ok thanks a lot!!