New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add init callback to MapReducer #75

Merged
merged 4 commits into from Feb 4, 2019

Conversation

Projects
None yet
2 participants
@danxexe
Copy link
Contributor

danxexe commented Feb 4, 2019

My use case for this feature is to allow setting logger metadata of the stage processes, including copying metadata from the parent process.

Apparently, the only way right now to do this is by checking the process dictionary and conditionally setting the logger metadata inside a Flow.map call. This is far from optimal if you just want to set the metadata once at the beginning of the whole flow, specially for large or infinite streams.

I haven't documented the new option yet because I'm not sure about the name (init_map_reducer) and I want to hear other opinions about the feature first.

@danxexe danxexe force-pushed the danxexe:init-map-reducer branch from 798b144 to 8861eab Feb 4, 2019

@@ -4,6 +4,10 @@ defmodule Flow.MapReducer do

def init({type, opts, index, trigger, acc, reducer}) do
Process.flag(:trap_exit, true)

{init_map_reducer, opts} = Keyword.pop(opts, :init_map_reducer, & &1)
init_map_reducer.({type, opts, index, trigger, acc, reducer})

This comment has been minimized.

@josevalim

josevalim Feb 4, 2019

Member

I think this is exposing too much information to the user callback. We should probably pass the index at best.

This comment has been minimized.

@danxexe

danxexe Feb 4, 2019

Author Contributor

👍 My initial idea was to allow the callback to modify the options at runtime, but I definitely do not need that feature and even forgot to use the return of the callback.

end

[1, 2, 3, 4]
|> Flow.from_enumerable(stages: 2, init_map_reducer: init)

This comment has been minimized.

@josevalim

josevalim Feb 4, 2019

Member

Why do you think about calling the callback on_init?

This comment has been minimized.

@danxexe

danxexe Feb 4, 2019

Author Contributor

Fine by me.

danxexe added some commits Feb 4, 2019

@josevalim josevalim merged commit cbea6aa into plataformatec:master Feb 4, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@josevalim

This comment has been minimized.

Copy link
Member

josevalim commented Feb 4, 2019

❤️ 💚 💙 💛 💜

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment