Fix init(idempotent=True) / HailContext.getOrCreate

[tpoterba]

No description provided.

[tpoterba]

need to figure out how to properly stop.

@henrydavidge do you know how to totally kill the java process from pyspark?

In [2]: spark.stop()

In [3]: !jps
44688 Launcher
65412 SparkSubmit
65419 Jps
62991

[henrydavidge]

I don't think there's a pyspark API to do that. You could try calling Runtime.halt through py4j maybe? Or use py4j to get the pid and then kill through a shell command.

dismissing because I need to make some other changes

[tpoterba]

if pyspark doesn't have a way to do that, then I'd bet that there's some Python state that's not cleaned up if you try to restart it from the same Python runtime after killing the JVM.

I can fix Hail to properly clean up, I think.

[tpoterba]

@danking ready for another review. This PR has expanded because we needed to properly store/clear session-specific stuff in a bunch of places.

[danking]

rogue print

[danking]

these should probably all be collections.defaultdict([]), there's a few places in here where we have extra code to handle the key not being present but if we consistently treated "not present" as empty list, things would be simpler, i.e. the remove_function is just:

f = (param_types, ret_type)
_function_registry[name] = [b for b in _function_registry[name] if b != f]

Also removes an if check in _lookup_function_return_type.

---

not required for the PR, but for future reference, I think this is an exemplary use case for defaultdict

[danking]

It's definitely a bit unclear to met that we got everything and I'm not convinced the test checks everything. I don't think I'd feel much better unless we were randomly restarting the context mid tests.

[tpoterba]

It's definitely a bit unclear to met that we got everything and I'm not convinced the test checks everything. I don't think I'd feel much better unless we were randomly restarting the context mid tests.

We actually do that -- this test_context runs in the same python process as the other tests. I also think we don't really guarantee you can stop and restart, but this at least makes it work with our tests.

addressed?