Fix double test crash

[shinrich]

I was getting the double test to crash for me fairly consistently on our 9.0.1 build with the following stack. This was running with debug enabled. It appears that the captive_action (the action object returned for the open_read and open_write actions) had been canceled from a previous open attempt. Adding logic to the open_write that cleared the cancel bit up from as it does for open_read fixed the issue.

@SolidWallOfCode thinks the issue was introduced by possibility of open write retries in PR #6053

#0  0x00002aac8856e3d7 in raise () from /lib64/libc.so.6
#1  0x00002aac8856fac8 in abort () from /lib64/libc.so.6
#2  0x00002aac85ae40c8 in ink_abort (message_format=0x2aac85b51e78 "%s:%d: failed assertion `%s`") at ink_error.cc:99
#3  0x00002aac85adfd11 in _ink_assert (expression=0xa1507e "captive_action.cancelled == 0", file=0xa14ff8 "HttpCacheSM.cc", line=161) at ink_assert.cc:37
#4  0x0000000000788aaf in HttpCacheSM::state_cache_open_write (this=0x2aacb3086638, event=1109, data=0xffffffffffffb04f) at HttpCacheSM.cc:161
#5  0x000000000066b737 in Continuation::handleEvent (this=0x2aacb3086638, event=1109, data=0xffffffffffffb04f) at /home/shinrich/vtrafficserver9/iocore/eventsystem/I_Continuation.h:167
#6  0x00000000008e5c95 in Cache::open_write (this=0x2aac96cf3990, cont=0x2aacb3086638, key=0x2aac9260a790, info=0x0, apin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP, 
    hostname=0x2aac8d5a8831 "127.0.0.12009http127.0.0.1:2009http://127.0.0.1:2009/", host_len=9) at CacheWrite.cc:1829
#7  0x00000000008b26e6 in CacheProcessor::open_write (this=0xf10d80 <cacheProcessor>, cont=0x2aacb3086638, expected_size=0, key=0x2aac9260a780, request=0x2aacb3085018, old_info=0x0, pin_in_cache=0, 
    type=CACHE_FRAG_TYPE_HTTP) at Cache.cc:3254
#8  0x000000000078995d in HttpCacheSM::open_write (this=0x2aacb3086638, key=0x2aac9260a780, url=0x2aacb3085030, request=0x2aacb3085018, old_info=0x0, pin_in_cache=0, retry=true, allow_multiple=false)
    at HttpCacheSM.cc:363
#9  0x00000000007185aa in HttpSM::do_cache_prepare_action (this=0x2aacb3084860, c_sm=0x2aacb3086638, object_read_info=0x0, retry=true, allow_multiple=false) at HttpSM.cc:4723
#10 0x000000000072df04 in HttpSM::do_cache_prepare_write (this=0x2aacb3084860) at HttpSM.cc:4650
#11 0x0000000000725c15 in HttpSM::set_next_state (this=0x2aacb3084860) at HttpSM.cc:7576
#12 0x000000000072465d in HttpSM::call_transact_and_set_next_state (this=0x2aacb3084860, f=0x0) at HttpSM.cc:7301
#13 0x00000000007083d5 in HttpSM::handle_api_return (this=0x2aacb3084860) at HttpSM.cc:1614
#14 0x00000000007080cb in HttpSM::state_api_callout (this=0x2aacb3084860, event=0, data=0x0) at HttpSM.cc:1546
#15 0x000000000071b18f in HttpSM::do_api_callout_internal (this=0x2aacb3084860) at HttpSM.cc:5239
#16 0x000000000072dceb in HttpSM::do_api_callout (this=0x2aacb3084860) at HttpSM.cc:353
#17 0x00000000007246e6 in HttpSM::set_next_state (this=0x2aacb3084860) at HttpSM.cc:7335
#18 0x000000000072465d in HttpSM::call_transact_and_set_next_state (this=0x2aacb3084860, f=0x0) at HttpSM.cc:7301
#19 0x0000000000724c67 in HttpSM::set_next_state (this=0x2aacb3084860) at HttpSM.cc:7380
#20 0x000000000072465d in HttpSM::call_transact_and_set_next_state (this=0x2aacb3084860, f=0x0) at HttpSM.cc:7301
#21 0x00000000007083d5 in HttpSM::handle_api_return (this=0x2aacb3084860) at HttpSM.cc:1614
#22 0x00000000007080cb in HttpSM::state_api_callout (this=0x2aacb3084860, event=0, data=0x0) at HttpSM.cc:1546
#23 0x000000000071b18f in HttpSM::do_api_callout_internal (this=0x2aacb3084860) at HttpSM.cc:5239
#24 0x000000000072dceb in HttpSM::do_api_callout (this=0x2aacb3084860) at HttpSM.cc:353
#25 0x000000000072de74 in HttpSM::setup_cache_lookup_complete_api (this=0x2aacb3084860) at HttpSM.cc:2508
#26 0x000000000070dd0a in HttpSM::state_cache_open_read (this=0x2aacb3084860, event=1103, data=0xffffffffffffb050) at HttpSM.cc:2570
#27 0x000000000070e177 in HttpSM::main_handler (this=0x2aacb3084860, event=1103, data=0xffffffffffffb050) at HttpSM.cc:2612
#28 0x000000000066b737 in Continuation::handleEvent (this=0x2aacb3084860, event=1103, data=0xffffffffffffb050) at /home/shinrich/vtrafficserver9/iocore/eventsystem/I_Continuation.h:167
#29 0x000000000078874e in HttpCacheSM::state_cache_open_read (this=0x2aacb3086638, event=1103, data=0xffffffffffffb050) at HttpCacheSM.cc:133
#30 0x000000000066b737 in Continuation::handleEvent (this=0x2aacb3086638, event=1103, data=0xffffffffffffb050) at /home/shinrich/vtrafficserver9/iocore/eventsystem/I_Continuation.h:167
#31 0x00000000008ce55d in CacheVC::openReadFromWriterFailure (this=0x2aac991a1bd0, event=1103, e=0xffffffffffffb050) at CacheRead.cc:189
#32 0x00000000008cf1f1 in CacheVC::openReadFromWriter (this=0x2aac991a1bd0, event=2, e=0x2aac8a96d4e0) at CacheRead.cc:338
#33 0x000000000066b737 in Continuation::handleEvent (this=0x2aac991a1bd0, event=2, data=0x2aac8a96d4e0) at /home/shinrich/vtrafficserver9/iocore/eventsystem/I_Continuation.h:167
#34 0x00000000009d41a1 in EThread::process_event (this=0x2aac8d1320c0, e=0x2aac8a96d4e0, calling_code=2) at UnixEThread.cc:132
#35 0x00000000009d4765 in EThread::execute_regular (this=0x2aac8d1320c0) at UnixEThread.cc:241
#36 0x00000000009d4aa5 in EThread::execute (this=0x2aac8d1320c0) at UnixEThread.cc:332
#37 0x00000000009d32d3 in spawn_thread_internal (a=0x2aac893d9fc0) at Thread.cc:92
#38 0x00002aac878ffea5 in start_thread () from /lib64/libpthread.so.0
#39 0x00002aac886369fd in clone () from /lib64/libc.so.6

[bryancall]

@sudheerv Can you please take a look at this?

[shinrich]

I'm going to close this one. In #7807 @maskit and I reworked this captive_action logic. If there is still a problem, it is probably different by now.