Non-reproducable test failures in bootstrapmanager

[DoronShapiro]

I got a few test failures for bootstrapmanager. When I re-ran the tests, they went away.

> py.test                                                                                                                                               16:19
==================================================================== test session starts =====================================================================
platform darwin -- Python 2.7.11, pytest-2.8.7, py-1.4.31, pluggy-0.3.1
collected 102 items

driver/tests/test_secretbox.py ..........
managers/tests/test_bootstrap_manager.py .F..FF..FF.F
managers/tests/test_distributor.py .............
managers/tests/test_file_manager.py ..........
managers/tests/test_manifest.py .................................
providers/tests/test_LocalFilesystemProvider.py ......
tools/tests/test_encryption.py ....
tools/tests/test_erasure_encoding.py .....
tools/tests/test_secret_sharing.py .....
tools/tests/test_utils.py ....

========================================================================== FAILURES ==========================================================================
_______________________________________________________________________ test_roundtrip _______________________________________________________________________

    def test_roundtrip():
        BM = BootstrapManager(providers, 2)
        BM.distribute_bootstrap(bootstrap)
>       assert BM.recover_bootstrap() == bootstrap

managers/tests/test_bootstrap_manager.py:23:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
managers/BootstrapManager.py:97: in recover_bootstrap
    bootstrap = Bootstrap.parse(string)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

string = '\x05O\x82j>\x97t[\xb3)\xaapO\x9f\xfb\xb4>\xc5\xd9\x83\xf9\xecyM\x9a&\xeeN\x14\xd1\xbd04D23BB49E1A4AF7A94B33D414648C91\x00\x00\x00\x02'

    @staticmethod
    def parse(string):
>       key, manifest_name, file_reconstruction_threshold = struct.unpack(Bootstrap.STRUCT_FORMAT, string)
E       error: unpack requires a string argument of length 68

managers/BootstrapManager.py:35: error

And then similar failures in test_multiple_sessions, test_erase_recover, test_corrupt_k_recover, test_corrupt_k_2_recover, and test_corrupt_k_but_not_fail.

[sudssm]

close your eyes, plug your nose, say three hail marys, and the demon will go away

[sudssm]

Did you get all of these at the same time? If so, I think the shared bootstrap object (the same across all those tests) might be at fault - perhaps there's something I didn't think enough about. Will take a look, thanks for noticing

[DoronShapiro]

Yes, all at once

[DoronShapiro]

Got them again:

________________________________________________________________ test_corrupt_k_but_not_fail _________________________________________________________________

    def test_corrupt_k_but_not_fail():
        BM = BootstrapManager(providers, 3)
        BM.distribute_bootstrap(bootstrap)

        providers[0].put(BootstrapManager.BOOTSTRAP_THRESHOLD_FILE_NAME, "4")
        providers[1].put(BootstrapManager.BOOTSTRAP_THRESHOLD_FILE_NAME, "4")
        providers[2].put(BootstrapManager.BOOTSTRAP_THRESHOLD_FILE_NAME, "4")
        providers[3].put(BootstrapManager.BOOTSTRAP_THRESHOLD_FILE_NAME, "4")

        with pytest.raises(exceptions.OperationFailure) as excinfo:
>               BM.recover_bootstrap()

managers/tests/test_bootstrap_manager.py:141:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
managers/BootstrapManager.py:98: in recover_bootstrap
    bootstrap = Bootstrap.parse(string)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

string = 'f\xf6\xe4\xfa\x944\x95hS\xf66\xc3\xf0\xca\xf9~\xfc\x8c\xebrH\xf9/\x00\x060\xa7\x13c\xd3\xe845A373C502BD474F8B8091E2E253EF89\x00\x00\x00\x02'

    @staticmethod
    def parse(string):
>       key, manifest_name, file_reconstruction_threshold = struct.unpack(Bootstrap.STRUCT_FORMAT, string)
E       error: unpack requires a string argument of length 68

managers/BootstrapManager.py:35: error

[sudssm]

Ran into this. This is actually a bug with @michsoch's secret sharing reconstruct.
There are cases where I share a string of length 68, but get back a string of length 67.
I thought that this problem was solved already, by the magic prepend that michelle was doing - thoughts?

[sudssm]

Ah ha! Specifically, if the secret starts with "*" (a star), when reconstructed, the star vanishes!
@michsoch

[michsoch]

hmm...distinctly recall testing for that particular case
will investigate and get back to you

[michsoch]

so if anything, the prepend "magic" is the issue since what it prepends with is a "*" (star) - I tested for leading stars in the serialization code and am trying to recreate the issue with sharing and reconstruction but no luck yet in creating a failing test case

[sudssm]

at some point i'll demo the error in person

On Mon, Mar 21, 2016 at 10:25 PM michsoch notifications@github.com wrote:

so if anything, the prepend "magic" is the issue since what is prepends
with is a "*" (star) - I tested for leading stars in the serialization code
and am trying to recreate the issue with sharing and reconstruction but no
luck yet in creating a failing test case

—
You are receiving this because you were assigned.

Reply to this email directly or view it on GitHub
#24 (comment)

[michsoch]

can you send me code that breaks?

[michsoch]

instructions for reproduction would be very helpful - I've since rewritten a lot of the code where this issue is likely coming from so it might be a solved problem but I would like to verify that the test fails on old code and passes on the new iteration

[sudssm]

from robustsecretsharing.schemes.sss import share_secret, reconstruct_secret
shares = share_secret(5, 3, 4, "*123")
reconstruct_secret(5, 4, shares)

[sudssm]

this seems to work with the latest copy of the library, so the problem should be resolved upon integration of rss. Resolved.

[michsoch]

this should be resolved now that integrate_rss has been merged in
will mark as closed if no one sees it for a while...