Check child pid when disowning processes

[pedro-gutierrez]

This PR adds a check on the value returned by Map.pop/2 when disowning child processes given their ID. This should prevent the supervisor from crashing with errors such as:

12:41:20.080 [error] GenServer DistributedSupervisor terminating
** (MatchError) no match of right hand side value: {nil, %{44653510861120721877777239736197567562 => {{DistributedSupervisor, :"horde2@127.0.0.1"}, %{id: 44653510861120721877777239736197567562, restart: :transient, start: {<mod>, :start_link, [<args>]}}, #PID<0.714.0>}, 93025656845656942491727595806594601344 => ...}}
    (horde) lib/horde/dynamic_supervisor_impl.ex:255: Horde.DynamicSupervisorImpl.handle_cast/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_cast", {:disown_child_process, 290756678567143339301485527515936608587}}

(I removed specifics from my project).

Note: this fix feels a bit like sweeping under the carpet. I understand the root cause might be somewhere else but so far I haven't been able to reproduce this error in a consistent way.

[pedro-gutierrez]

Added a test to this PR. It does not reproduce the issue that I am tracking down, but it shows a net split that, if run in a loop, will eventually fail. When this happens, I've observed that the key for one of the workers is simply removed from all registries for no apparent obvious reasons. In other occasions, I've noticed multiple name conflicts on the same key, and at the end, all registered pids are terminated (ie, we end up in a situation where no winner pid is kept). Hope this helps to track potential race conditions in Horde.

[derekkraan]

Hi @pedro-gutierrez, thanks for this PR. I think I agree with your statement that this is sweeping it under the rug. As such I think I'd like to leave Horde.DynamicSupervisorImpl the way it is, but I would like your test so that I can investigate this further when I have some time. Could you revert the changes to the library code for me?

[derekkraan]

@pedro-gutierrez I think this might be related to #188

[derekkraan]

@pedro-gutierrez I think I have solved the problem in #190, can you please test & confirm?