Skip to content

Conversation

@timsaucer
Copy link
Member

Which issue does this PR close?

Note: This is based on top of #18672. I will rebase after that one merges to remove the parts of the code that are not relevant to this PR.

Rationale for this change

This PR solves one of the biggest problems we have in using user defined functions together via FFI. The justification is included in the README.md file in this PR. In order to de/serialize any expressions, either logical or physical, we need to know what functions have been registered. This is not possible to do if we cannot access the current list of registered functions.

This causes a circular dependency if we want to use registered functions as parts of expressions to other registered functions. The encoding and decoding of these functions will fail. This PR adds a weak reference to a trait that can access the current TaskContext to be used for the protobuf de/serialization.

What changes are included in this PR?

  • Adds the trait TaskContextProvider that is used to get the current task context.
  • Adds a FFI_TaskContext and FFI_TaskContextProvider
  • Plumbs the FFI_TaskContextProvider throughout any of the FFI code that requires protobof encoding and decoding.
  • Updates unit tests to pass around the SessionContext as a TaskContextProvider

Are these changes tested?

The unit tests have been updated. Additionally this has been tested against datafusion-python

Are there any user-facing changes?

This is an API change for FFI.

  • TODO(tsaucer) Add text in migration guide about these changes.

@github-actions github-actions bot added documentation Improvements or additions to documentation core Core DataFusion crate execution Related to the execution crate ffi Changes to the ffi crate labels Nov 13, 2025
@timsaucer timsaucer force-pushed the feat/ffi-task-ctx-provider branch from 12f317d to 6d48b25 Compare November 20, 2025 18:13
@timsaucer timsaucer closed this Nov 20, 2025
@timsaucer timsaucer deleted the feat/ffi-task-ctx-provider branch November 20, 2025 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate documentation Improvements or additions to documentation execution Related to the execution crate ffi Changes to the ffi crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant