For Chinese Document
Hugging Face Home Page: Link
SkyPaint is a Chinese-English bilingual text-generated image project developed by Singularity-AI. It is still being updated and optimized.
Using our model, input some Chinese or English text, then the machine can generate images with modern artistic style just like human painters. Here are some examples:
机械狗
城堡 大海 夕阳 宫崎骏动画
花落知多少
半鸡半人，强壮
鸡你太美
Please visit SkyPaint
The SkyPaint text generation image model is mainly composed of two parts, namely the prompt word text encoder model and the diffusion model. Therefore, our optimization is also divided into two steps. First, based on OpenAI-CLIP, we optimized the prompt word text encoder model to make SkyPaint have the ability to recognize Chinese and English, and then optimized the diffusion model, so that SkyPaint has modern artistic capabilities and can produce high-quality pictures.
- Chinese and English mixed prompt word input.
- Generating high-quality images in a modern art styles.
- English prompt words for stable_diffusion_1.x official model and related fine-tuning models.
- Retain usage habits and methods of stable_diffusion prompt words.
Download model: SkyPaint-v1.0
from diffusers import StableDiffusionPipeline
device = 'cuda'
pipe = StableDiffusionPipeline.from_pretrained("path_to_our_model").to(device)
prompts = [
'机械狗',
'城堡 大海 夕阳 宫崎骏动画',
'花落知多少',
'鸡你太美',
]
for prompt in prompts:
prompt = 'sai-v1 art, ' + prompt
image = pipe(prompt).images[0]
image.save("%s.jpg" % prompt)————————————————————————————————————————————————————————————
SkyCLIP is a CLIP model obtained by using an efficient method of training Chinese-English bilingual CLIP models. This method only needs to use text data to achieve efficient distillation of the OpenAI-CLIP model, which greatly reduces the data threshold. At the same time, training requires Compared with the original CLIP model, the computing power requirement is reduced by more than 90%, which is convenient for the open source community to reproduce/fine-tune. This method only changes the text encoder of OpenAI-CLIP, and can be used with the image encoder of OpenAI-CLIP to realize the image-text retrieval function.
- Chinese-English Machine Translation Task Parallel Corpus.
- United Nations Chinese-English Parallel Corpus.
- LAION Chinese and English Corpus (Part).
- Wukong Chinese Corpus (Part).
- AI-Challenger translation task Chinese and English corpus.
- Chinese and English corpus of ancient poetry.
- A Chinese and English corpus composed of common words in the prompt word handbook/magic book.
Use the text_encoder of OpenAI-CLIP as the teacher model and freeze the parameters. The student model uses a multilingual BERT model of the same size as the teacher model. During training, the English input is obtained through the teacher model to obtain the corresponding t_en_hiddent_state, and English and Chinese are respectively obtained through the student model. The corresponding s_en_hiddent_state, s_zh_hidden_state uses l1, l2, cos distance, etc. to construct loss functions so that the Chinese and English hidden_state of the student model gradually approaches the hidden_state of the teacher model. Due to the natural unequal length of Chinese and English in the parallel corpus, in order to make the parallel Chinese and English as close as possible, we also added a Chinese decoder during the training process, and used the Chinese and English hidden_state of the student model as the hidden_state input of the decoder. The translation task is used to assist in the alignment of Chinese and English.
At present, we mainly evaluate the zero-shot performance of SkyCLIP on Flickr30K-CN, and mainly compare several related open source models with Chinese capabilities. For the L/14 size model, our evaluation process refers to the evaluation script provided by Chinese-CLIP.
Flickr30K-CN Retrieval:
| Task | Text-to-Image | Image-to-Text | MR | ||||
|---|---|---|---|---|---|---|---|
| Setup | Zero-shot | Zero-shot | |||||
| Metric | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | |
| Taiyi-326M | 53.8 | 79.9 | 86.6 | 64.0 | 90.4 | 96.1 | 78.47 |
| AltCLIP | 50.7 | 75.4 | 83.1 | 73.4 | 92.8 | 96.9 | 78.72 |
| Wukong | 51.9 | 78.6 | 85.9 | 75 | 94.4 | 97.7 | 80.57 |
| R2D2 | 42.6 | 69.5 | 78.6 | 63.0 | 90.1 | 96.4 | 73.37 |
| CN-CLIP | 68.1 | 89.7 | 94.5 | 80.2 | 96.6 | 98.2 | 87.87 |
| SkyCLIP | 58.8 | 82.6 | 89.6 | 78.8 | 96.1 | 98.3 | 84.04 |
from PIL import Image
import requests
import clip
import torch
from transformers import BertTokenizer
from transformers import CLIPProcessor, CLIPModel, CLIPTextModel
import numpy as np
query_texts = ['一个人', '一辆汽车', '两个男人', '两个女人'] # 这里是输入提示词，可以随意替换。
# 加载SkyCLIP 中英文双语 text_encoder
text_tokenizer = BertTokenizer.from_pretrained("./tokenizer")
text_encoder = CLIPTextModel.from_pretrained("./text_encoder").eval()
text = text_tokenizer(query_texts, return_tensors='pt', padding=True)['input_ids']
url = "http://images.cocodataset.org/val2017/000000040083.jpg" #这里可以换成任意图片的url
# 加载CLIP的image encoder
clip_model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
clip_text_proj = clip_model.text_projection
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
image = processor(images=Image.open(requests.get(url, stream=True).raw), return_tensors="pt")
with torch.no_grad():
image_features = clip_model.get_image_features(**image)
text_features = text_encoder(text)[0]
# sep_token对应于openai-clip的eot_token
sep_index = torch.nonzero(text == student_tokenizer.sep_token_id)
text_features = text_features[torch.arange(text.shape[0]), sep_index[:, 1]]
# 乘text投影矩阵
text_features = clip_text_proj(text_features)
image_features = image_features / image_features.norm(dim=1, keepdim=True)
text_features = text_features / text_features.norm(dim=1, keepdim=True)
# 计算余弦相似度 logit_scale是尺度系数
logit_scale = clip_model.logit_scale.exp()
logits_per_image = logit_scale * image_features @ text_features.t()
logits_per_text = logits_per_image.t()
probs = logits_per_image.softmax(dim=-1).cpu().numpy()
print(np.around(probs, 3))Our data uses the filtered Laion data set as the training data, and adds 'sai-v1 art' as the tag in front of the text so that the model can learn the style and quality we want more quickly. The pre-training model uses stable-diffusion-v1-5 as pre-training, and uses 16 A100s for 50 hours of training. The current model is still being optimized, and there will be more stable model updates in the future.
