Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add colab link and notebook #15

Merged
merged 4 commits into from
Nov 17, 2022
Merged

add colab link and notebook #15

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1d78e58
add example jupyter notebook
shreydan Nov 15, 2022
b2b3d2d
update
shreydan Nov 15, 2022
c58dc37
readme and notebook update
shreydan Nov 16, 2022
2c8d063
update readme
shreydan Nov 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,12 @@ img.save(settings["output_img_path"])

There is minimal working example in `runner.py` that is self contained. Please have a look!

## Notebook

you can view the notebook [here](./contents/notebooks/paint_with_words.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1MZfGaY3aQQn5_T-6bkXFE1rI59A2nJlU?usp=sharing)

---

# Weight Scaling

In the paper, they used $w \log (1 + \sigma) \max (Q^T K)$ to scale appropriate attention weight. However, this wasn't optimal after few tests, found by [CookiePPP](https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/4406). You can check out the effect of the functions below:
Expand Down Expand Up @@ -219,12 +225,11 @@ img = paint_with_words(
)
```



---

# TODO

*I'll work on these after school exam is over*
_I'll work on these after school exam is over_

- [ ] Make extensive comparisons for different weight scaling functions.
- [ ] Create word latent-based cross-attention generations.
Expand All @@ -239,4 +244,4 @@ img = paint_with_words(
- [ ] "negative region", where we can set some region to "not" have some semantics. can be done with classifier-free guidance.
- [ ] Img2ImgPaintWithWords -> Img2Img, but with extra text segmentation map for better control
- [ ] InpaintPaintwithWords -> inpaint, but with extra text segmentation map for better control
- [ ] Support for other schedulers
- [ ] Support for other schedulers
Loading