Add interior design ControlNet pipeline readme #150

j-branigan · 2023-05-19T10:07:47Z

First draft done. Feedback on image is particularly welcome.

examples/pipelines/controlnet-interior-design/README.md

RobbeSneyders

Thanks @jamesbraniganml6!

Can we add a visual example of the data?
Can we add a section on how to reuse this pipeline for a different use case? They should only recreate the prompt generation component.
The pipeline is currently drawn with parallel captioning and segmentation, which is not the case. For clarity, it might be better to make these sequential.

examples/pipelines/controlnet-interior-design/README.md

@GeorgesLorre

PR that adds missing data types required for defining needed nested data types for the embedding component Changes: * The string values of the enum types have been changed to pyarrow types to make it easier to define complex schema * utf8 types defined in the components have been changed to strings to make them more intuitive We will need to make more changes in the future to handle different nested data types as suggested by @GeorgesLorre https://swagger.io/docs/specification/data-models/data-types/#:~:text=the%20null%20value.-,Arrays,-Arrays%20are%20defined Enums allow us to define nice constants that are typed but we'll need to define many in order to accommodate for all different types of nested structures. We might need to move away to dynamically typed data types with a dictionary but this will require quite some changes to the json schemas and the code so better leave it for later

PR that adds the image embedding component. Largely inspired by Niel's PR #111 (inference and batching with dask).

Added the logo svg's 🎉

This branch is based on the image-embedding branch which has a lot of changes. I would suggest to first merge that PR which will give a much smaller Diff here. This component implements the LAION image retrieval component which uses CLIP embeddings from the input subset to query the LAION database. It returns an images subset with URLs, similar to the other prompt based Clip Retrieval component. These URLs should then be downloaded by the already-made image-downloading component. --------- Co-authored-by: Philippe Moussalli <philippe.moussalli95@gmail.com>

This PR contains the Image Cropping component. The component looks for the most common color in the border. It uses this color to calculate how much of the image border can be cropped out. If the crop is not square, it will paste a border on the shortest side to make it square again. ![d4e35776-3ce1-4157-ac1f-5b2f18ff2ad4](https://github.com/ml6team/fondant/assets/92580873/314ec0d3-3ab6-418e-8051-d9f464496b0e) ![82eeae2d-c63c-42cb-881c-3707971d043c](https://github.com/ml6team/fondant/assets/92580873/6754b418-7922-4744-8ef3-59978b07ee9d) --------- Co-authored-by: Philippe Moussalli <philippe.moussalli95@gmail.com>

j-branigan · 2023-05-22T16:47:56Z

Thanks @jamesbraniganml6!

Can we add a visual example of the data?

Can we add a section on how to reuse this pipeline for a different use case? They should only recreate the prompt generation component.

The pipeline is currently drawn with parallel captioning and segmentation, which is not the case. For clarity, it might be better to make these sequential.

Thanks for the feedback @RobbeSneyders.
I've updated the image and added small docs for the components for convenience. On the other two issues:

I have added a line in the pipeline README linking to the generate_prompts file. It's very short but before adding more detail I think that the component itself should be updated to make it more general. The function names and prompt generation are very specifically related to interior design. I can do it myself but I'd want to open it as a separate PR if you agree.
What did you have in mind as a visual example of the data? Do you just mean an actual image from LAION. Are you talking about a visual representation of the structure of the dataframe?

GeorgesLorre

Nice James !

Since we focus on resuablity and want to inspire people to use fondant we could still make it more visual by adding more data examples or example picures (like the resizing, captioning etc). But that is maybe something we can still improve outside of this PR.

examples/pipelines/controlnet-interior-design/README.md

GeorgesLorre · 2023-05-22T21:41:51Z

docs/art/controlnet-interior-design-pipeline.png

The image is not telling much IMO, maybe a small sentence or 2 per step to explain how the data is being extended and enriched.

(also database is very vague, maybe call it dataset ready for fine-tuning or something)

components/prompt_based_laion_retrieval/README.md

examples/pipelines/controlnet-interior-design/README.md

NielsRogge · 2023-05-23T09:56:20Z

examples/pipelines/controlnet-interior-design/README.md

+## Introduction
+This example demonstrates an end-to-end fondant pipeline to collect and process data for the training of a [ControlNet](https://github.com/lllyasviel/ControlNet) model, focusing on images related to interior design.
+
+### What is Controlnet?


Suggested change

### What is Controlnet?

### What is ControlNet?

NielsRogge · 2023-05-23T09:58:22Z

examples/pipelines/controlnet-interior-design/README.md

+
+### What is Controlnet?
+
+Controlnet is an image generation model developed by https://arxiv.org/abs/2302.05543 that gives the user more control over the image generation process. It is based on the Stable Diffusion model, which generates images based on a caption and an image. The Controlnet model adds a third input, a conditioning image, that can be used for specifying specific wanted elements in the generated image.


Suggested change

Controlnet is an image generation model developed by https://arxiv.org/abs/2302.05543 that gives the user more control over the image generation process. It is based on the Stable Diffusion model, which generates images based on a caption and an image. The Controlnet model adds a third input, a conditioning image, that can be used for specifying specific wanted elements in the generated image.

ControlNet is an image generation model developed by [Zhang etl a., 2023](https://arxiv.org/abs/2302.05543) that gives the user more control over the image generation process. It is based on the [Stable Diffusion](https://stability.ai/blog/stable-diffusion-public-release) model, which generates images based on text and an optional image. The ControlNet model adds a third input, a conditioning image, that can be used for specifying specific wanted elements in the generated image.

examples/pipelines/controlnet-interior-design/README.md

NielsRogge · 2023-05-23T09:59:22Z

examples/pipelines/controlnet-interior-design/README.md

+Useful links:
+* https://github.com/lllyasviel/ControlNet
+* https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/controlnet
+* https://arxiv.org/abs/2302.05543


Suggested change

Useful links:

* https://github.com/lllyasviel/ControlNet

* https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/controlnet

* https://arxiv.org/abs/2302.05543

Useful links:

* https://github.com/lllyasviel/ControlNet

* https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/controlnet

* https://arxiv.org/abs/2302.05543

It might be you need to include an enter here for this to show appropriately

RobbeSneyders

Thanks @ChristiaensBert!

Looks good, left some comments. You'll also have to rebase / merge since some of the components have moved on main.

examples/pipelines/controlnet-interior-design/README.md

RobbeSneyders · 2023-05-23T10:57:34Z

examples/pipelines/controlnet-interior-design/README.md

+1. Building the images for each of the pipeline components
+```
+bash build_images.sh -c all 
+```


They should set the --namespace and --repo as well to push the images to their own github container registry.

@philippe-ml6 Do you have the full command that they have to use?

It's in the bash script's help function

bash build_images.sh --help Usage: build_images.sh [options] Options: -c, --component <value> Set the component name. Pass the component folder name to build a certain components or 'all' to build all components in the current directory (required) -n, --namespace <value> Set the namespace (default: ml6team) -r, --repo <value> Set the repo (default: fondant) -t, --tag <value> Set the tag (default: latest) -h, --help Display this help message

examples/pipelines/controlnet-interior-design/README.md

RobbeSneyders · 2023-05-23T10:58:37Z

examples/pipelines/controlnet-interior-design/components/caption_images/README.md

+# caption_images
+
+### Description
+This component captions inputted images using [BLIP](https://huggingface.co/docs/transformers/model_doc/blip).


This component takes a model id as input, so it can use any HF Hub model

PhilippeMoussalli · 2023-05-23T11:34:20Z

docs/art/controlnet-interior-design-pipeline.png

Is it a database or the hub that we should have at the end?

PhilippeMoussalli · 2023-05-23T11:36:36Z

examples/pipelines/controlnet-interior-design/README.md

+
+| Input image                                                    | Output image                                                     |
+|----------------------------------------------------------------|------------------------------------------------------------------|
+| ![input image](docs/art/interior_design_controlnet_input1.png) | ![output image](docs/art/interior_design_controlnet_output1.jpg) |


Those images are not rendered properly

RobbeSneyders

Thanks Bert!

Can you remove the images that are not used? I see you added some more in the docs/art folder.

First draft done. Feedback on image is particularly welcome. --------- Co-authored-by: Philippe Moussalli <philippe.moussalli95@gmail.com> Co-authored-by: khaerensml6 <92426912+khaerensml6@users.noreply.github.com> Co-authored-by: ChristiaensBert <92580873+ChristiaensBert@users.noreply.github.com> Co-authored-by: Bert Christiaens <bert.christiaens@ml6.eu>

j-branigan added 3 commits May 19, 2023 12:00

README first draft

1bdaf8c

Updated image of pipeline

6064286

removed line

4455843

j-branigan requested review from GeorgesLorre, janvanlooy and NielsRogge May 19, 2023 10:08

j-branigan commented May 19, 2023

View reviewed changes

examples/pipelines/controlnet-interior-design/README.md Outdated Show resolved Hide resolved

j-branigan linked an issue May 19, 2023 that may be closed by this pull request

Add ControlNet example README #120

Closed

Merge branch 'main' into add_pipeline_doc

7bf258a

RobbeSneyders reviewed May 22, 2023

View reviewed changes

examples/pipelines/controlnet-interior-design/README.md Outdated Show resolved Hide resolved

RobbeSneyders changed the title ~~Add pipeline doc~~ Add interior design ControlNet pipeline readme May 22, 2023

PhilippeMoussalli and others added 11 commits May 22, 2023 18:21

Add image embedding component (#148)

1e17e06

PR that adds the image embedding component. Largely inspired by Niel's PR #111 (inference and batching with dask).

add logo svg's (#153)

088032c

Added the logo svg's 🎉

Added READMEs for components and updated links

5e26f70

added README for LAION retrieval

a1750bd

rephrasing

1a0a369

moved pipeline image to art folder

07d2598

updated controlnet pipeline image

c624580

added ToC and info on how to adapt pipeline

8d1c8a0

Merge branch 'main' into add_pipeline_doc

f49f151

GeorgesLorre reviewed May 22, 2023

View reviewed changes

add readme documentation

a5bfff2

NielsRogge reviewed May 23, 2023

View reviewed changes

components/prompt_based_laion_retrieval/README.md Outdated Show resolved Hide resolved

NielsRogge reviewed May 23, 2023

View reviewed changes

components/prompt_based_laion_retrieval/README.md Outdated Show resolved Hide resolved

NielsRogge reviewed May 23, 2023

View reviewed changes

examples/pipelines/controlnet-interior-design/README.md Outdated Show resolved Hide resolved

NielsRogge reviewed May 23, 2023

View reviewed changes

examples/pipelines/controlnet-interior-design/README.md Outdated Show resolved Hide resolved

NielsRogge reviewed May 23, 2023

View reviewed changes

RobbeSneyders reviewed May 23, 2023

View reviewed changes

ChristiaensBert added 2 commits May 23, 2023 13:04

add documentation pipeline

4d97707

change documentation

e3725b1

PhilippeMoussalli reviewed May 23, 2023

View reviewed changes

docs/art/controlnet-interior-design-pipeline.png Outdated

Copy link

Contributor

PhilippeMoussalli May 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it a database or the hub that we should have at the end?

ChristiaensBert added 2 commits May 23, 2023 13:34

add example images

b493321

add exapmle images

7920d0c

PhilippeMoussalli reviewed May 23, 2023

View reviewed changes

ChristiaensBert added 2 commits May 23, 2023 13:38

add example images

e562def

add bash script to documentation:

fc3d646

RobbeSneyders approved these changes May 23, 2023

View reviewed changes

ChristiaensBert added 2 commits May 23, 2023 14:58

add better pipeline schema

0094e9a

documentation:

e412405

RobbeSneyders merged commit 015fd25 into main May 23, 2023

RobbeSneyders deleted the add_pipeline_doc branch May 23, 2023 13:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add interior design ControlNet pipeline readme #150

Add interior design ControlNet pipeline readme #150

j-branigan commented May 19, 2023

RobbeSneyders left a comment

j-branigan commented May 22, 2023 •

edited

Loading

GeorgesLorre left a comment

GeorgesLorre May 22, 2023

NielsRogge May 23, 2023

NielsRogge May 23, 2023

NielsRogge May 23, 2023

RobbeSneyders left a comment

RobbeSneyders May 23, 2023

ChristiaensBert May 23, 2023

PhilippeMoussalli May 23, 2023

RobbeSneyders May 23, 2023

PhilippeMoussalli May 23, 2023

PhilippeMoussalli May 23, 2023

RobbeSneyders left a comment


		### What is Controlnet?

		Controlnet is an image generation model developed by https://arxiv.org/abs/2302.05543 that gives the user more control over the image generation process. It is based on the Stable Diffusion model, which generates images based on a caption and an image. The Controlnet model adds a third input, a conditioning image, that can be used for specifying specific wanted elements in the generated image.

Add interior design ControlNet pipeline readme #150

Add interior design ControlNet pipeline readme #150

Conversation

j-branigan commented May 19, 2023

RobbeSneyders left a comment

Choose a reason for hiding this comment

j-branigan commented May 22, 2023 • edited Loading

GeorgesLorre left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RobbeSneyders left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RobbeSneyders left a comment

Choose a reason for hiding this comment

j-branigan commented May 22, 2023 •

edited

Loading