test_serde_simplify is flaky #3359
Description
Describe the bug
test_serde_simplify frequently fails and appears to be applying the wrong tensor serialization format (sometimes but not always.)
To Reproduce
Run the test (or full suite) until it fails.
Expected behavior
The test should reliably pass.
Screenshots
2020-04-14T18:01:22.2134777Z =================================== FAILURES ===================================
2020-04-14T18:01:22.2135402Z __________________________ test_serde_simplify[State] __________________________
2020-04-14T18:01:22.2135862Z
2020-04-14T18:01:22.2137173Z cls = <class 'syft.execution.state.State'>
2020-04-14T18:01:22.2138492Z workers = {'alice': <VirtualWorker id:alice #objects:0>, 'bob': <VirtualWorker id:bob #objects:0>, 'charlie': <VirtualWorker id:charlie #objects:0>, 'james': <VirtualWorker id:james #objects:0>, ...}
2020-04-14T18:01:22.2139188Z hook = <syft.frameworks.torch.hook.hook.TorchHook object at 0x7f343eeafe80>
2020-04-14T18:01:22.2140173Z start_remote_worker = <function start_remote_worker.<locals>._start_remote_worker at 0x7f343cc6aae8>
2020-04-14T18:01:22.2140630Z
2020-04-14T18:01:22.2141200Z @pytest.mark.parametrize("cls", samples)
2020-04-14T18:01:22.2142456Z def test_serde_simplify(cls, workers, hook, start_remote_worker):
2020-04-14T18:01:22.2143904Z """Checks that simplified structures match expected"""
2020-04-14T18:01:22.2144442Z _samples = samples[cls](
2020-04-14T18:01:22.2144954Z workers=workers,
2020-04-14T18:01:22.2145475Z hook=hook,
2020-04-14T18:01:22.2146062Z start_remote_worker=start_remote_worker,
2020-04-14T18:01:22.2146588Z port=9001,
2020-04-14T18:01:22.2147100Z id="simplify",
2020-04-14T18:01:22.2147606Z )
2020-04-14T18:01:22.2148108Z for sample in _samples:
2020-04-14T18:01:22.2148710Z obj, expected_simplified_obj = sample.get("value"), sample.get("simplified")
2020-04-14T18:01:22.2149270Z _simplify = (
2020-04-14T18:01:22.2149785Z msgpack.serde._simplify
2020-04-14T18:01:22.2150289Z if not sample.get("forced", False)
2020-04-14T18:01:22.2150818Z else msgpack.serde._force_full_simplify
2020-04-14T18:01:22.2151338Z )
2020-04-14T18:01:22.2151890Z serde_worker = syft.hook.local_worker
2020-04-14T18:01:22.2152420Z serde_worker.framework = sample.get("framework", torch)
2020-04-14T18:01:22.2153946Z simplified_obj = _simplify(syft.hook.local_worker, obj)
2020-04-14T18:01:22.2154461Z
2020-04-14T18:01:22.2154872Z if sample.get("cmp_simplified", None):
2020-04-14T18:01:22.2155283Z # Custom simplified objects comparison function.
2020-04-14T18:01:22.2155742Z assert sample.get("cmp_simplified")(simplified_obj, expected_simplified_obj) is True
2020-04-14T18:01:22.2156593Z else:
2020-04-14T18:01:22.2157000Z > assert simplified_obj == expected_simplified_obj
2020-04-14T18:01:22.2157445Z E assert (22, ((1, ((4...one, ...)))))) == (22, ((1, ((4...one, ...))))))
2020-04-14T18:01:22.2170190Z E At index 1 diff: ((1, ((48, ((56, (22214076726,)), (3, ((5, (b'state1',)),)), None, None)), (48, ((56, (96071220832,)), (3, ((5, (b'state2',)),)), None, None)))), (1, ((14, (9826749836, b'\x80\x02\x8a\nl\xfc\x9cF\xf9 j\xa8P\x19.\x80\x02M\xe9\x03.\x80\x02}q\x00(X\x10\x00\x00\x00protocol_versionq\x01M\xe9\x03X\r\x00\x00\x00little_endianq\x02\x88X\n\x00\x00\x00type_sizesq\x03}q\x04(X\x05\x00\x00\x00shortq\x05K\x02X\x03\x00\x00\x00intq\x06K\x04X\x04\x00\x00\x00longq\x07K\x04uu.\x80\x02ctorch._utils\n_rebuild_tensor_v2\nq\x00((X\x07\x00\x00\x00storageq\x01ctorch\nFloatStorage\nq\x02X\x0e\x00\x00\x0094299942048352q\x03X\x03\x00\x00\x00cpuq\x04K\tNtq\x05QK\x00K\x03K\x03\x86q\x06K\x03K\x01\x86q\x07\x89ccollections\nOrderedDict\nq\x08)Rq\ttq\nRq\x0b.\x80\x02]q\x00X\x0e\x00\x00\x0094299942048352q\x01a.\t\x00\x00\x00\x00\x00\x00\x00#\x95{?\xf6\x19\x17\xbe\x9c\xa3\xec=\x01\x8b,<\x11-[\xbf\xa1d\xea\xbf\xb8\xcd5\xbe\xac\xbd\xb9\xbf\xb0JQ?', None, None, None, None, (5, (b'torch',)), None, None)), (14, (49440533498, b"\x80\x02\x8a\nl\xfc\x9cF\xf9 j\xa8P\x19.\x80\x02M\xe9\x03.\x80\x02}q\x00(X\x10\x00\x00\x00protocol_versionq\x01M\xe9\x03X\r\x00\x00\x00little_endianq\x02\x88X\n\x00\x00\x00type_sizesq\x03}q\x04(X\x05\x00\x00\x00shortq\x05K\x02X\x03\x00\x00\x00intq\x06K\x04X\x04\x00\x00\x00longq\x07K\x04uu.\x80\x02ctorch._utils\n_rebuild_tensor_v2\nq\x00((X\x07\x00\x00\x00storageq\x01ctorch\nFloatStorage\nq\x02X\x0e\x00\x00\x0094299946743552q\x03X\x03\x00\x00\x00cpuq\x04K\tNtq\x05QK\x00K\x03K\x03\x86q\x06K\x03K\x01\x86q\x07\x89ccollections\nOrderedDict\nq\x08)Rq\ttq\nRq\x0b.\x80\x02]q\x00X\x0e\x00\x00\x0094299946743552q\x01a.\t\x00\x00\x00\x00\x00\x00\x00p=D??*'\xbf\x11\x07\x97\xbdx\xa2\xd0=\x91\xb3v?\xad\xee->\xfd\xbc\xbc?\xef\x87\xe7\xbe\xa4\x1c\x00@", None, None, None, None, (5, (b'torch',)), None, None))))) != ((1, ((48, ((56, (22214076726,)), (3, ((5, (b'state1',)),)), None, None)), (48, ((56, (96071220832,)), (3, ((5, (b'state2',)),)), None, None)))), (1, ((14, (9826749836, (6, ((6, (3, 3)), (5, (b'float32',)), (1, (0.9827443957328796, -0.1475599706172943, 0.11554643511772156, 0.010531187988817692, -0.8561564087867737, -1.8311959505081177, -0.1775425672531128, -1.4511008262634277, 0.8175458908081055)))), None, None, None, None, (5, (b'all',)), None, None)), (14, (49440533498, (6, ((6, (3, 3)), (5, (b'float32',)), (1, (0.7665624618530273, -0.6529883742332458, -0.07374394685029984, 0.10187238454818726, 0.9636774659156799, 0.16985578835010529, 1.4745174646377563, -0.4522089660167694, 2.0017480850219727)))), None, None, None, None, (5, (b'all',)), None, None)))))
2020-04-14T18:01:22.2171445Z E Full diff:
2020-04-14T18:01:22.2171865Z E (
2020-04-14T18:01:22.2172266Z E 22,
2020-04-14T18:01:22.2172660Z E ((1,
2020-04-14T18:01:22.2173053Z E ((48,
2020-04-14T18:01:22.2173436Z E ((56,
2020-04-14T18:01:22.2173834Z E (22214076726,)),
2020-04-14T18:01:22.2174238Z E (3,
2020-04-14T18:01:22.2174632Z E ((5,
2020-04-14T18:01:22.2175404Z E (b'state1',)),)),
2020-04-14T18:01:22.2175833Z E None,
2020-04-14T18:01:22.2176226Z E None)),
2020-04-14T18:01:22.2176620Z E (48,
2020-04-14T18:01:22.2177026Z E ((56,
2020-04-14T18:01:22.2177473Z E (96071220832,)),
2020-04-14T18:01:22.2177643Z E (3,
2020-04-14T18:01:22.2177828Z E ((5,
2020-04-14T18:01:22.2178170Z E (b'state2',)),)),
2020-04-14T18:01:22.2178369Z E None,
2020-04-14T18:01:22.2178554Z E None)))),
2020-04-14T18:01:22.2178739Z E (1,
2020-04-14T18:01:22.2178928Z E ((14,
2020-04-14T18:01:22.2179113Z E (9826749836,
2020-04-14T18:01:22.2179489Z E + b'\x80\x02\x8a\nl\xfc\x9cF\xf9 j\xa8P\x19.\x80\x02M\xe9\x03.\x80\x02}'
2020-04-14T18:01:22.2179912Z E + b'q\x00(X\x10\x00\x00\x00protocol_versionq\x01M\xe9\x03X\r\x00\x00\x00li'
2020-04-14T18:01:22.2180297Z E + b'ttle_endianq\x02\x88X\n\x00\x00\x00type_sizesq\x03}q\x04(X'
2020-04-14T18:01:22.2180762Z E + b'\x05\x00\x00\x00shortq\x05K\x02X\x03\x00\x00\x00intq\x06K\x04X\x04\x00'
2020-04-14T18:01:22.2181171Z E + b'\x00\x00longq\x07K\x04uu.\x80\x02ctorch._utils\n_rebuild_tensor_v2\n'
2020-04-14T18:01:22.2181542Z E + b'q\x00((X\x07\x00\x00\x00storageq\x01ctorch\nFloatStorage\nq\x02'
2020-04-14T18:01:22.2181933Z E + b'X\x0e\x00\x00\x0094299942048352q\x03X\x03\x00\x00\x00cpuq\x04K\tNtq'
2020-04-14T18:01:22.2182528Z E + b'\x05QK\x00K\x03K\x03\x86q\x06K\x03K\x01\x86q\x07\x89ccollections\nOrde'
2020-04-14T18:01:22.2182979Z E + b'redDict\nq\x08)Rq\ttq\nRq\x0b.\x80\x02]q\x00X\x0e\x00\x00\x00942999420'
2020-04-14T18:01:22.2183422Z E + b'48352q\x01a.\t\x00\x00\x00\x00\x00\x00\x00#\x95{?\xf6\x19\x17'
2020-04-14T18:01:22.2183887Z E + b'\xbe\x9c\xa3\xec=\x01\x8b,<\x11-[\xbf\xa1d\xea\xbf\xb8\xcd5'
2020-04-14T18:01:22.2184281Z E + b'\xbe\xac\xbd\xb9\xbf\xb0JQ?',
2020-04-14T18:01:22.2184668Z E - (6,
2020-04-14T18:01:22.2185014Z E - ((6,
2020-04-14T18:01:22.2185394Z E - (3,
2020-04-14T18:01:22.2185742Z E - 3)),
2020-04-14T18:01:22.2186104Z E - (5,
2020-04-14T18:01:22.2186487Z E - (b'float32',)),
2020-04-14T18:01:22.2186851Z E - (1,
2020-04-14T18:01:22.2187323Z E - (0.9827443957328796,
2020-04-14T18:01:22.2187690Z E - -0.1475599706172943,
2020-04-14T18:01:22.2188096Z E - 0.11554643511772156,
2020-04-14T18:01:22.2188465Z E - 0.010531187988817692,
2020-04-14T18:01:22.2188894Z E - -0.8561564087867737,
2020-04-14T18:01:22.2189260Z E - -1.8311959505081177,
2020-04-14T18:01:22.2189665Z E - -0.1775425672531128,
2020-04-14T18:01:22.2190034Z E - -1.4511008262634277,
2020-04-14T18:01:22.2190421Z E - 0.8175458908081055)))),
2020-04-14T18:01:22.2190649Z E None,
2020-04-14T18:01:22.2190860Z E None,
2020-04-14T18:01:22.2191066Z E None,
2020-04-14T18:01:22.2191256Z E None,
2020-04-14T18:01:22.2191485Z E (5,
2020-04-14T18:01:22.2191833Z E - (b'all',)),
2020-04-14T18:01:22.2192094Z E ? ^^^
2020-04-14T18:01:22.2192454Z E + (b'torch',)),
2020-04-14T18:01:22.2192664Z E ? ^^^^^
2020-04-14T18:01:22.2192875Z E None,
2020-04-14T18:01:22.2193084Z E None)),
2020-04-14T18:01:22.2193295Z E (14,
2020-04-14T18:01:22.2193506Z E (49440533498,
2020-04-14T18:01:22.2193939Z E + b'\x80\x02\x8a\nl\xfc\x9cF\xf9 j\xa8P\x19.\x80\x02M\xe9\x03.\x80\x02}'
2020-04-14T18:01:22.2194376Z E + b'q\x00(X\x10\x00\x00\x00protocol_versionq\x01M\xe9\x03X\r\x00\x00\x00li'
2020-04-14T18:01:22.2194829Z E + b'ttle_endianq\x02\x88X\n\x00\x00\x00type_sizesq\x03}q\x04(X'
2020-04-14T18:01:22.2195269Z E + b'\x05\x00\x00\x00shortq\x05K\x02X\x03\x00\x00\x00intq\x06K\x04X\x04\x00'
2020-04-14T18:01:22.2195729Z E + b'\x00\x00longq\x07K\x04uu.\x80\x02ctorch._utils\n_rebuild_tensor_v2\n'
2020-04-14T18:01:22.2196719Z E + b'q\x00((X\x07\x00\x00\x00storageq\x01ctorch\nFloatStorage\nq\x02'
2020-04-14T18:01:22.2197205Z E + b'X\x0e\x00\x00\x0094299946743552q\x03X\x03\x00\x00\x00cpuq\x04K\tNtq'
2020-04-14T18:01:22.2197642Z E + b'\x05QK\x00K\x03K\x03\x86q\x06K\x03K\x01\x86q\x07\x89ccollections\nOrde'
2020-04-14T18:01:22.2198217Z E + b'redDict\nq\x08)Rq\ttq\nRq\x0b.\x80\x02]q\x00X\x0e\x00\x00\x00942999467'
2020-04-14T18:01:22.2198682Z E + b"43552q\x01a.\t\x00\x00\x00\x00\x00\x00\x00p=D??*'\xbf\x11\x07\x97"
2020-04-14T18:01:22.2199134Z E + b'\xbdx\xa2\xd0=\x91\xb3v?\xad\xee->\xfd\xbc\xbc?\xef\x87\xe7'
2020-04-14T18:01:22.2199514Z E + b'\xbe\xa4\x1c\x00@',
2020-04-14T18:01:22.2199880Z E - (6,
2020-04-14T18:01:22.2200242Z E - ((6,
2020-04-14T18:01:22.2200612Z E - (3,
2020-04-14T18:01:22.2200979Z E - 3)),
2020-04-14T18:01:22.2201498Z E - (5,
2020-04-14T18:01:22.2201919Z E - (b'float32',)),
2020-04-14T18:01:22.2202312Z E - (1,
2020-04-14T18:01:22.2202708Z E - (0.7665624618530273,
2020-04-14T18:01:22.2203142Z E - -0.6529883742332458,
2020-04-14T18:01:22.2203545Z E - -0.07374394685029984,
2020-04-14T18:01:22.2203979Z E - 0.10187238454818726,
2020-04-14T18:01:22.2204406Z E - 0.9636774659156799,
2020-04-14T18:01:22.2204821Z E - 0.16985578835010529,
2020-04-14T18:01:22.2205233Z E - 1.4745174646377563,
2020-04-14T18:01:22.2205629Z E - -0.4522089660167694,
2020-04-14T18:01:22.2206066Z E - 2.0017480850219727)))),
2020-04-14T18:01:22.2206388Z E None,
2020-04-14T18:01:22.2206613Z E None,
2020-04-14T18:01:22.2206856Z E None,
2020-04-14T18:01:22.2207062Z E None,
2020-04-14T18:01:22.2207320Z E (5,
2020-04-14T18:01:22.2207701Z E - (b'all',)),
2020-04-14T18:01:22.2207959Z E ? ^^^
2020-04-14T18:01:22.2208337Z E + (b'torch',)),
2020-04-14T18:01:22.2208586Z E ? ^^^^^
2020-04-14T18:01:22.2208813Z E None,
2020-04-14T18:01:22.2209023Z E None))))),
2020-04-14T18:01:22.2209267Z E )
2020-04-14T18:01:22.2209433Z
2020-04-14T18:01:22.2209663Z test/serde/msgpack/test_msgpack_serde_full.py:166: AssertionError
Additional context
This started being an issue when the order of the test suite was randomized (specifically to shake out flaky tests like this.)