Fix typed.Dict and typed.List crashing on parametrized types

[asodeur]

Fixes #6401 as described here

[gmarkall]

Thanks for the PR! Once it's no longer [WIP], we can queue it up for review.

[gmarkall]

Thanks very much for the PR, I've queued it for review.

[stuartarchibald]

Thanks for the patch and for debugging and fixing this! The suggested fix seems to work around the issue described. I've reviewed the changes, there's just a few things to take a look at, most of which are in relation to the way Numba does things opposed to anything fundamental. Thanks again!

[stuartarchibald]

I'm trying to understand why this is in a subprocess, is it with view of isolating a potential segfault? If the PR is fixing something that has a tendency to segfault, I'd rather it segfaults the test suite as a demonstration that it's not working than survive via isolation and report it, else we'd end up with all tests in subprocesses just-in-case?

[asodeur]

I have not found a reproducer w/o subprocess, yet. I've tried Dispatcher._make_finalizer(()(); Dispatcher._reset_overloads(); jit(...) to force re-loading from cache (which seems to do the job) but it still does not crash unless the cached function is used in another process. Is there anything more I can try and reset/delete in a single process?

[asodeur]

After a little more experimentation I came-up with the below. Does the way I am resetting the dispatchers and the target context look resonable?

    def test_parametrized_types(self):
        """https://github.com/numba/numba/issues/6401"""

        register_model(ParametrizedType)(UniTupleModel)

        @typeof_impl.register(Parametrized)
        def typeof_unit(val, c):
            return ParametrizedType(val)

        @unbox(ParametrizedType)
        def unbox_parametrized(typ, obj, context):
            return context.unbox(types.UniTuple(typ.dtype, len(typ)), obj)

        @generated_jit
        def dict_vs_cache_vs_parametrized(v):
            typ = v

            def objmode_vs_cache_vs_parametrized_impl(v):
                # typed.List shows same behaviour after fix for #6397
                d = typed.Dict.empty(types.unicode_type, typ)
                d['data'] = v

            return objmode_vs_cache_vs_parametrized_impl

        @jit(nopython=True, cache=True)
        def set_parametrized_data(x, y):
            # Has had a tendency to segfault when the compiled function
            # was loaded from cache in a different process than the one
            # it was originally compiled in.
            # The new process is simulated below by resetting the dispatchers
            # and the target context
            dict_vs_cache_vs_parametrized(x)
            dict_vs_cache_vs_parametrized(y)

        x, y = Parametrized(('a', 'b')), Parametrized(('a',))
        set_parametrized_data(x, y)

        # reset dispatchers and targetctx to force re-load from cache as a new process would jit
        set_parametrized_data._make_finalizer()()
        set_parametrized_data._reset_overloads()
        set_parametrized_data.targetctx.init()

        for ii in range(50):  # <- somtimes works a few times
            assert set_parametrized_data(x, y) is None