Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move away from using global app env for pipeline module config #22

Closed
Ziinc opened this issue Nov 27, 2019 · 2 comments
Closed

Move away from using global app env for pipeline module config #22

Ziinc opened this issue Nov 27, 2019 · 2 comments
Assignees

Comments

@Ziinc
Copy link
Collaborator

Ziinc commented Nov 27, 2019

Pipeline config should be localized to the specific pipeline.
Benefits:

  1. Reduces clashes (when more middlewares/pipelines get built)
  2. makes parameter declaration clearer
  3. makes pipelines more "functional" and reusable
  4. use of Application.get_env usage within pipeline modules makes things less clear when declaring pipelines.

Related to #20 , would pave the way for adding logic into a pipeline (instead of having a fat pipeline module)

Proposed api:

  • Tuple definitions would only be required for modules that require configuration
    • at the pipeline level, they could throw an error when checking for parameters, or use default values.
  • a pipeline module can also be passed directly when no parameters are needed

For example:

pipelines: [
  ....
  MyCustom.Pipeline.CleanItem,
  {Crawly.Pipelines.Validate, item: [:title, :url] },
  {Crawly.Pipelines.DuplicatesFilter, item_id: :title },
  Crawly.Pipelines.JSONEncoder
]

Besides adjusting existing built-in pipelines, this proposed change would also require the adjustment of Crawly.Utils.pipe to check for tuple definitions.

@oltarasenko
Copy link
Collaborator

I kind of like this idea!

@Ziinc
Copy link
Collaborator Author

Ziinc commented Nov 29, 2019

I'll open a PR for this over the weekend

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants