Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make IR function registration idempotent #8123

Merged
merged 1 commit into from
Feb 20, 2020

Conversation

karenfeng
Copy link
Contributor

Right now, IR function registration fails in the following case.

  • Multiple Python HailContexts are running on the same cluster
  • User on HailContext A calls a function (eg. create histogram) that triggers hl.experimental.define_function
    • An anonymous function is used to create a new Set in IRFunctionRegistry.irRegistry
  • User on HailContext B calls the same function
    • The anonymous function is added to create a Set of size 2 in IRFunctionRegistry.irRegistry, as anonymous functions are never considered equivalent
    • This triggers a fatal error Multiple functions found that satisfy ...

By changing the definition of IRFunctionRegistry.irRegistry from a MultiMap(functionName -> Set[argumentTypes, returnType, alwaysInline, anonymousFunction]) to a Map(functionName -> Map((argumentTypes, returnType, alwaysInline) -> anonymousFunction)) , we ensure function registration is idempotent as we do not compare on the anonymousFunction.

Signed-off-by: Karen Feng <karen.feng@databricks.com>
@tpoterba tpoterba self-assigned this Feb 20, 2020
@tpoterba
Copy link
Contributor

looks very reasonable on first scan, will have time to look more carefully tomorrow. If this passes tests I'm pretty confident it's fine.

@danking danking merged commit e3c0a51 into hail-is:master Feb 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants