AttributeError: 'PreProcessedElements' object has no attribute 'mixins'

[sbellem]

Happens when running the experiment on EC2, for the commands:

• python -m apps.asynchromix.powermixing
• python -m apps.asynchromix.butterfly_network
• python -m honeybadgermpc.ipc

$ python -m aws.run-on-ec2
2020-01-13 20:01:12,935:[run-on-ec2.py:231]:[INFO]: Run Id: 63a3da1df4a349f687d3558e44eb9b3d
2020-01-13 20:01:18,640:[ec2Manager.py:45]:[INFO]: VM creation started.
2020-01-13 20:05:35,462:[ec2Manager.py:89]:[INFO]: VMs successfully booted up.
2020-01-13 20:05:38,128:[run-on-ec2.py:267]:[INFO]: Uploading config file to S3 in 'honeybadgermpc-20200113' bucket.
2020-01-13 20:05:39,159:[run-on-ec2.py:270]:[INFO]: Config file upload complete.
2020-01-13 20:05:39,160:[run-on-ec2.py:272]:[INFO]: Triggering config update on instances.
[Errno None] Unable to connect to port 22 on 3.83.40.79
[Errno None] Unable to connect to port 22 on 35.183.0.59
[Errno None] Unable to connect to port 22 on 18.185.124.164
[Errno 110] Connection timed out
2020-01-13 20:07:50,052:[run-on-ec2.py:278]:[INFO]: Config update completed successfully.
2020-01-13 20:07:50,131:[run-on-ec2.py:177]:[INFO]: Starting to create preprocessing files.
2020-01-13 20:07:50,132:[run-on-ec2.py:182]:[INFO]: Preprocessing files created in 0.0014934539794921875
2020-01-13 20:07:50,132:[run-on-ec2.py:186]:[INFO]: Uploading input files to AWS S3.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/src/HoneyBadgerMPC/aws/run-on-ec2.py", line 375, in <module>
    trigger_run(args.run_id, args.skip_setup, args.max_k, args.only_setup, args.cleanup)
  File "/usr/src/HoneyBadgerMPC/aws/run-on-ec2.py", line 285, in trigger_run
    max_k, run_id, s3manager, instance_ids
  File "/usr/src/HoneyBadgerMPC/aws/run-on-ec2.py", line 200, in get_powermixing_setup_commands
    prefix1 = f"{pp_elements.mixins[Constants.POWERS].file_prefix}_{j}"
AttributeError: 'PreProcessedElements' object has no attribute 'mixins'

[amiller]

We do not have any ci tests covering this AWS code, so it is not surprising if small interface changes or refactoring end up breaking these.

[sbellem]

A note to myself to keep track of the progress on fixing this issue ... currently fixed the "original" error and running into this error:

ubuntu@ip-172-31-88-6:~$ sudo docker run -p 7000:7000 -v /home/ubuntu/config:/usr/src/HoneyBadgerMPC/config/ -v /home/ubuntu/sharedata:/usr/src/HoneyBadgerMPC/sharedata/ -v /home/ubuntu/benchmark-logs:/usr/src/Ho
neyBadgerMPC/benchmark-logs/ sbellem/honeybadgermpc python -m honeybadgermpc.ipc -d -f config/config-1.json                                                                                                         
2020-01-29 22:05:54,345:[base_events.py:1604]:[ERROR]: Exception in callback ProcessProgramRunner.execute.<locals>.callback(<Task finishe...ound only 0')>) at /usr/src/HoneyBadgerMPC/honeybadgermpc/ipc.py:144    
handle: <Handle ProcessProgramRunner.execute.<locals>.callback(<Task finishe...ound only 0')>) at /usr/src/HoneyBadgerMPC/honeybadgermpc/ipc.py:144>                                                                
Traceback (most recent call last):                                                                                                                                                                                  
  File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run                                                                                                                                               
    self._context.run(self._callback, *self._args)                                                                                                                                                                  
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/ipc.py", line 145, in callback                                                                                                                                       
    program_result.set_result(future.result())                                                                                                                                                                      
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/mpc.py", line 243, in _run    
    return result.result()                                                               
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/mpc.py", line 356, in test_prog2    
    shares = [context.preproc.get_zero(context) for _ in range(1000)]                               
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/mpc.py", line 356, in <listcomp>                                                                                                                                 
    shares = [context.preproc.get_zero(context) for _ in range(1000)]                                                                               
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/preprocessing.py", line 647, in get_zero
    return self._zeros.get_value(context)                                          
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/preprocessing.py", line 101, in get_value
    to_return, used = self._get_value(context, key, *args, **kwargs)                                        
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/preprocessing.py", line 427, in _get_value
    f"Expected "                                                             
AssertionError: Expected 1 elements of zeros, but found only 0

[amiller]

Isn't this just the error message due to preprocessing underrun? Generate more zeroes?

[sbellem]

just adding a bit of logging info for the above error:

2020-01-30 18:30:44,067:[base_events.py:1604]:[ERROR]: Exception in callback print_exception_callback(<Task finishe..., 1): 0})\n")>) at /usr/src/HoneyBadgerMPC/honeybadgermpc/utils/misc.py:11
handle: <Handle print_exception_callback(<Task finishe..., 1): 0})\n")>) at /usr/src/HoneyBadgerMPC/honeybadgermpc/utils/misc.py:11>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/utils/misc.py", line 17, in print_exception_callback
    raise ex
  File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/ipc.py", line 145, in callback
    program_result.set_result(future.result())
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/mpc.py", line 243, in _run
    return result.result()
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/mpc.py", line 317, in test_batchopening
    xs = [context.preproc.get_zero(context) + context.Share(i) for i in range(100)]
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/mpc.py", line 317, in <listcomp>
    xs = [context.preproc.get_zero(context) + context.Share(i) for i in range(100)]
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/preprocessing.py", line 649, in get_zero
    return self._zeros.get_value(context)
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/preprocessing.py", line 101, in get_value
    to_return, used = self._get_value(context, key, *args, **kwargs)
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/preprocessing.py", line 427, in _get_value
    f"Expected "
AssertionError: Expected 1 elements of zeros, but found only 0
key is: (0, 4, 1)
count is: defaultdict(<class 'int'>, {(0, 5, 1): 1000, (0, 4, 1): 0})

the key/tuple (0, 4, 1) corresponds to (context.my_id, context.N, context.t) which is built from the config:

ubuntu@ip-172-31-46-44:~$ cat config/config-0.json 
{"N": 4, "t": 1, "my_id": 0, "peers": ["34.229.192.79:7000", "54.92.170.32:7000", "52.67.182.22:7000", "3.126.146.22:7000", "35.183.10.38:7000"], "reconstruction": {"induce_faults": false}, "skip_preprocessing": true, "extra": {}}

so what is not clear to me now, and what I am looking into now, is why is also the tuple (0, 5, 1) an entry, and this one has 1000 elements compared to (0, 4, 1) which has 0 ...

the contents of the sharedata dir on one of the machines:

ubuntu@ip-172-31-46-44:~$ ls sharedata/
triples_5_1-0.share  zeros_5_1-0.share
ubuntu@ip-172-31-46-44:~$ wc -l sharedata/zeros_5_1-0.share 
1003 sharedata/zeros_5_1-0.share

[sbellem]

Isn't this just the error message due to preprocessing underrun? Generate more zeroes?

Yes, generating more zeroes works ... 😄 but the value of n needs to match the number of VMs, as this number is currently used when generating triples and zeroes, e.g.:

HoneyBadgerMPC/aws/run-on-ec2.py

Lines 61 to 67 in 7335331

	n, t = AwsConfig.TOTAL_VM_COUNT, AwsConfig.MPC_CONFIG.T
	num_triples = AwsConfig.MPC_CONFIG.NUM_TRIPLES
	pp_elements = PreProcessedElements()
	pp_elements.generate_zeros(num_triples, n, t)
	pp_elements.generate_triples(num_triples, n, t)

[amiller]

That script may need to change since we may have VMs for clients in an experiment, that would not count as part of n or t. This especially occurs when running experiments for AVSS where there is a separate dealer