[Alpha pipeline] Add prompt generating component #84

RobbeSneyders · 2023-05-08T09:11:53Z

Fixes #94

Split this from #71 so we can review this component by component.

This PR contains the first component, which generates captions to query LAION-5B with. This is also the only component that is specific to this example, so I would place all other components in a components/ directory.

I also think we can put examples directly under the examples/ directory instead of examples/pipelines, as the example pipelines contain example components, and the reusable components in the future components/ directory can serve as examples as well.

RobbeSneyders

Thanks @jamesbraniganml6 and @NielsRogge!

Left some comments.

examples/controlnet-interior-design/components/generate_prompts/Dockerfile

examples/controlnet-interior-design/components/generate_prompts/fondant_component.yaml

examples/controlnet-interior-design/components/generate_prompts/src/main.py

RobbeSneyders · 2023-05-08T09:26:50Z

examples/controlnet-interior-design/components/generate_prompts/src/main.py

+
+        df = dd.from_pandas(
+            df, npartitions=1
+        )  # need to decide how npartitions should be set


Should be based on the amount of cores available and data size.
https://stackoverflow.com/a/46646366/4098821

I think we should handle this as Fondant because:

This is the complexity we want to take out of the user's hands

The dask partitioning translates to parquet partitioning and so is also relevant for downstream components

We can do this after the user returns the dataframe leveraging repartition, but then the user still needs to provide some placeholder value in the meantime.

Another option is that we provide some utility functionality for this.

CC: @GeorgesLorre

I agree we should handle this in Fondant. the argument is optional so I would just not use it here.

It is optional, but only if you specify chunksize instead.
https://github.com/dask/dask/blob/980fd92a46f101f161d21e0c4f486024bafbfad3/dask/dataframe/io/io.py#L263-L264

PhilippeMoussalli

Looking good so far :) thanks!
Left a few coments

PhilippeMoussalli · 2023-05-08T09:52:28Z

examples/controlnet-interior-design/components/generate_prompts/src/main.py

+logger = logging.getLogger(__name__)
+
+
+interior_styles = [


Those prompts seem to be currently hardcoded within the main function. They should be moved to a general config and passed as parameters to the kubeflow component.

This way you avoid having to re-build your images everytime you want to change prompt lists and it also enables you to do experiment tracking by tracking the parameters that went into a certain kubeflow pipeline run

Then the question is if we even need this component, or if you shouldn't just pass your prompts to the LAION retriever directly.

I would still create a parquet file out of them to potentially link the retrieved images with a given prompt based on ID

PhilippeMoussalli · 2023-05-08T09:56:52Z

examples/controlnet-interior-design/components/generate_prompts/fondant_component.yaml

+        type: utf8
+
+args:
+  dataset_name:


Would add the prompt lists as arguments here. Although we still need to make sure lists are parsed properly. I created a ticket for it #85

If we do want to add them as arguments, maybe file-based arguments make more sense for this data size?

examples/controlnet-interior-design/components/generate_prompts/src/main.py

NielsRogge

LGTM after remaining comments are addressed

NielsRogge · 2023-05-09T09:33:36Z

Also could you clarify:

This is also the only component that is specific to this example, so I would place all other components in a components/ directory.

Normally all components of a given pipeline should be placed in a components folder, otherwise it would become hard to see which components a certain pipeline consists of. I thought the plan was to have native components directly available in the Fondant SDK, not in the examples folder?

examples/controlnet-interior-design/components/generate_prompts/fondant_component.yaml

RobbeSneyders · 2023-05-09T09:48:24Z

LGTM after remaining comments are addressed

Can you pick this up @NielsRogge? I just split this PR from the other branch.

Normally all components of a given pipeline should be placed in a components folder, otherwise it would become hard to see which components a certain pipeline consists of. I thought the plan was to have native components directly available in the Fondant SDK, not in the examples folder?

Indeed, I would add all other components as native / reusable components. So we should get the following directory structure:

- examples
  |- controlnet-interior-design
     |- components
        |- generate_prompts
     |- pipeline.py
- components
  |- prompt-based-LAION-retriever
  |- image-downloader
  |- image-resolution-filter
  ...

And the pipeline should use the reusable components.

examples/controlnet-interior-design/components/generate_prompts/Dockerfile

PhilippeMoussalli

Thanks! Left a few open comments to address

examples/controlnet-interior-design/components/generate_prompts/Dockerfile

examples/controlnet-interior-design/components/generate_prompts/src/main.py

...ples/pipelines/controlnet-interior-design/components/generate_prompts/fondant_component.yaml

examples/pipelines/controlnet-interior-design/controlnet_pipeline:dev.yaml

PhilippeMoussalli

Thanks for adding this Niels 👍 ! One small comment but otherwise good to go

RobbeSneyders

Thanks @NielsRogge!
LGTM (Can't approve my own PR)

fondant/component_spec.py

Co-authored-by: Niels Rogge <niels.rogge@ml6.eu> Co-authored-by: Robbe Sneyders <robbe.sneyders@gmail.com>

Fixes #84 Split this from #71 so we can review this component by component. This PR contains the first component, which generates captions to query LAION-5B with. This is also the only component that is specific to this example, so I would place all other components in a `components/` directory. I also think we can put examples directly under the `examples/` directory instead of `examples/pipelines`, as the example pipelines contain example components, and the reusable components in the future `components/` directory can serve as examples as well. --------- Co-authored-by: James Branigan <james.branigan@ml6.eu> Co-authored-by: Niels Rogge <niels.rogge@ml6.eu> Co-authored-by: Niels Rogge <nielsrogge@Nielss-MacBook-Pro.local>

RobbeSneyders requested review from PhilippeMoussalli, NielsRogge and j-branigan May 8, 2023 09:11

RobbeSneyders commented May 8, 2023

View reviewed changes

PhilippeMoussalli reviewed May 8, 2023

View reviewed changes

RobbeSneyders assigned NielsRogge May 9, 2023

NielsRogge approved these changes May 9, 2023

View reviewed changes

NielsRogge reviewed May 9, 2023

View reviewed changes

examples/controlnet-interior-design/components/generate_prompts/fondant_component.yaml Outdated Show resolved Hide resolved

PhilippeMoussalli reviewed May 9, 2023

View reviewed changes

examples/controlnet-interior-design/components/generate_prompts/Dockerfile Outdated Show resolved Hide resolved

PhilippeMoussalli reviewed May 9, 2023

View reviewed changes

NielsRogge force-pushed the feature/interior-prompts branch from b28038d to bc216d6 Compare May 9, 2023 15:04

NielsRogge changed the title ~~Add prompt generating component for interior design controlnet example~~ [Alpha pipeline] Add prompt generating component May 9, 2023

PhilippeMoussalli reviewed May 9, 2023

View reviewed changes

...ples/pipelines/controlnet-interior-design/components/generate_prompts/fondant_component.yaml Outdated Show resolved Hide resolved

PhilippeMoussalli reviewed May 9, 2023

View reviewed changes

examples/pipelines/controlnet-interior-design/controlnet_pipeline:dev.yaml Outdated Show resolved Hide resolved

PhilippeMoussalli approved these changes May 9, 2023

View reviewed changes

RobbeSneyders commented May 9, 2023

View reviewed changes

fondant/component_spec.py Outdated Show resolved Hide resolved

j-branigan and others added 5 commits May 10, 2023 09:52

Add prompt generating component for interior design controlnet example

421593d

Co-authored-by: Niels Rogge <niels.rogge@ml6.eu> Co-authored-by: Robbe Sneyders <robbe.sneyders@gmail.com>

Address comments

c915e98

Address comments

d976340

Add pipeline files

e856e5d

More improvements

b884936

NielsRogge force-pushed the feature/interior-prompts branch from 6507229 to b884936 Compare May 10, 2023 07:53

Update requirements

0acd64a

NielsRogge merged commit cbfd34a into main May 10, 2023

RobbeSneyders deleted the feature/interior-prompts branch May 15, 2023 16:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Alpha pipeline] Add prompt generating component #84

[Alpha pipeline] Add prompt generating component #84

RobbeSneyders commented May 8, 2023 •

edited

Loading

RobbeSneyders left a comment

RobbeSneyders May 8, 2023

GeorgesLorre May 8, 2023

RobbeSneyders May 8, 2023 •

edited

Loading

PhilippeMoussalli left a comment

PhilippeMoussalli May 8, 2023

RobbeSneyders May 8, 2023

PhilippeMoussalli May 8, 2023

PhilippeMoussalli May 8, 2023 •

edited

Loading

RobbeSneyders May 9, 2023

NielsRogge left a comment

NielsRogge commented May 9, 2023

RobbeSneyders commented May 9, 2023

PhilippeMoussalli left a comment

PhilippeMoussalli left a comment

RobbeSneyders left a comment

		logger = logging.getLogger(__name__)


		interior_styles = [

[Alpha pipeline] Add prompt generating component #84

[Alpha pipeline] Add prompt generating component #84

Conversation

RobbeSneyders commented May 8, 2023 • edited Loading

RobbeSneyders left a comment

Choose a reason for hiding this comment

RobbeSneyders May 8, 2023

Choose a reason for hiding this comment

GeorgesLorre May 8, 2023

Choose a reason for hiding this comment

RobbeSneyders May 8, 2023 • edited Loading

Choose a reason for hiding this comment

PhilippeMoussalli left a comment

Choose a reason for hiding this comment

PhilippeMoussalli May 8, 2023

Choose a reason for hiding this comment

RobbeSneyders May 8, 2023

Choose a reason for hiding this comment

PhilippeMoussalli May 8, 2023

Choose a reason for hiding this comment

PhilippeMoussalli May 8, 2023 • edited Loading

Choose a reason for hiding this comment

RobbeSneyders May 9, 2023

Choose a reason for hiding this comment

NielsRogge left a comment

Choose a reason for hiding this comment

NielsRogge commented May 9, 2023

RobbeSneyders commented May 9, 2023

PhilippeMoussalli left a comment

Choose a reason for hiding this comment

PhilippeMoussalli left a comment

Choose a reason for hiding this comment

RobbeSneyders left a comment

Choose a reason for hiding this comment

RobbeSneyders commented May 8, 2023 •

edited

Loading

RobbeSneyders May 8, 2023 •

edited

Loading

PhilippeMoussalli May 8, 2023 •

edited

Loading