Skip to content

Commit

Permalink
Merge pull request #15 from shreydan/master
Browse files Browse the repository at this point in the history
add colab link and notebook
  • Loading branch information
cloneofsimo committed Nov 17, 2022
2 parents 4c95a11 + 2c8d063 commit ec6974f
Show file tree
Hide file tree
Showing 2 changed files with 6,846 additions and 4 deletions.
13 changes: 9 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,12 @@ img.save(settings["output_img_path"])

There is minimal working example in `runner.py` that is self contained. Please have a look!

## Notebook

you can view the notebook [here](./contents/notebooks/paint_with_words.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1MZfGaY3aQQn5_T-6bkXFE1rI59A2nJlU?usp=sharing)

---

# Weight Scaling

In the paper, they used $w \log (1 + \sigma) \max (Q^T K)$ to scale appropriate attention weight. However, this wasn't optimal after few tests, found by [CookiePPP](https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/4406). You can check out the effect of the functions below:
Expand Down Expand Up @@ -219,12 +225,11 @@ img = paint_with_words(
)
```



---

# TODO

*I'll work on these after school exam is over*
_I'll work on these after school exam is over_

- [ ] Make extensive comparisons for different weight scaling functions.
- [ ] Create word latent-based cross-attention generations.
Expand All @@ -239,4 +244,4 @@ img = paint_with_words(
- [ ] "negative region", where we can set some region to "not" have some semantics. can be done with classifier-free guidance.
- [ ] Img2ImgPaintWithWords -> Img2Img, but with extra text segmentation map for better control
- [ ] InpaintPaintwithWords -> inpaint, but with extra text segmentation map for better control
- [ ] Support for other schedulers
- [ ] Support for other schedulers
Loading

0 comments on commit ec6974f

Please sign in to comment.