Investigate tests that fail when are run in unexpected order

[juniorz]

Last Friday, I almost lost my sanity when some tests under dir_handle_get/ started to fail after I reordered them. Because of that I figured out that I was not cleaning of initializing things appropriately.

Today, I've added a --random option to tinytest test runner (c147292). It shuffles the test cases before running. Not surprisingly, some other tests failed when they are run in an unexpected order.

I'm running the tests randomly in another branch and they are failing.

I'm not sure if we have time for this, but it would be helpful to investigate those tests.

[juniorz]

I suggest we start by double checking the tests we are familiar with.

By chance, rend_cache/intro_failure_note crashed when I ran it out of order (but together with other tests):

$ src/test/test --random=1440989960
seed=1440989960
onion_handshake: OK
[...]
link-handshake/recv_authchallenge/tooshort: [forking] OK
rend_cache/intro_failure_note:
============================================================ T= 1440990089
Tor 0.2.7.2-alpha-dev died: Caught signal 11
0   test                                0x000000010cda62f2 crash_handler + 82
1   libsystem_platform.dylib            0x00007fff92476f1a _sigtramp + 26
2   test                                0x000000010cdb509e strmap_impl_HT_FIND_P_ + 46
3   test                                0x000000010cdb4ee7 strmap_impl_HT_FIND + 71
4   test                                0x000000010cdb4e05 strmap_get + 69
5   test                                0x000000010cdb741c strmap_get_lc + 76
6   test                                0x000000010cd1ccfe cache_failure_intro_lookup + 94
7   test                                0x000000010cd1d036 rend_cache_intro_failure_note + 70
8   test                                0x000000010cac2c56 test_rend_cache_intro_failure_note + 70
9   test                                0x000000010cb89c08 testcase_run_bare_ + 136
10  test                                0x000000010cb89798 testcase_run_one + 344
11  test                                0x000000010cb8a85e tinytest_main + 1966
12  test                                0x000000010cb88a30 main + 816
13  libdyld.dylib                       0x00007fff862585c9 start + 1
14  ???                                 0x0000000000000002 0x0 + 2
[1]    10967 abort      src/test/test --random=1440989960

NOTE: it failed at 7a73c3b + branch changes, so you might need to comment out options_act_no_geoIP_database_found_to_mesure_entry_node to reproduce the order.

And rend_cache/init fails depending on who runs before it (when it runs together with its suite):

% src/test/test --random=1440991022 rend_cache/..
seed=1440991022
rend_cache/failure_intro_entry_new: OK
rend_cache/clean_v2_descs_as_dir: OK
rend_cache/entry_free: OK
rend_cache/clean: [forking] OK
rend_cache/store_v2_desc_as_dir: OK
rend_cache/failure_clean: OK
rend_cache/init:
  FAIL src/test/test_rendcache.c:542: rend_cache should be NULL when starting
  [init FAILED]
rend_cache/validate_intro_point_failure: OK
rend_cache/lookup_v2_desc_as_dir: OK
rend_cache/store_v2_desc_as_client: OK
rend_cache/failure_intro_add: OK
rend_cache/free_all: OK
rend_cache/store_v2_desc_as_dir_with_different_time: OK
rend_cache/purge: OK
rend_cache/failure_intro_entry_free: OK
rend_cache/failure_entry_free: OK
rend_cache/failure_intro_lookup: OK
rend_cache/entry_allocation: OK
rend_cache/increment_allocation: OK
rend_cache/decrement_allocation: OK
rend_cache/failure_purge: OK
rend_cache/intro_failure_note: OK
rend_cache/store_v2_desc_as_dir_with_different_content: OK
rend_cache/failure_remove: OK
rend_cache/store_v2_desc_as_client_with_different_time: OK
rend_cache/failure_entry_new: OK
rend_cache/lookup: OK
1/27 TESTS FAILED. (0 skipped)