Bonus Challenge | Generative-based Co-pilot

🚀 Calling all innovators! 🚀

💡 We're looking for your creative genius to level up the Penpot challenge.
Share your groundbreaking ideas and let's redefine what's possible here ! 🔥

🔎 Overview

Due to the limited academic and practical work done on the component and page as SVG generation, we propose the bonus challenge.

Instead of generating long SVG text files for each component, we can utilize controllable image generation to generate the components or page visualization in the existing or target styles.

💡 Feature analysis

Approach #1 | Generative-based Co-pilot

The system will allow a user to generate an image of the component or the page based on the input style or layout mockup.

Technology readiness	Risks	Complexity
🟡 Some elements exist but require adaptation	🔴 Higher than average	🟡 Moderately complex

Pipeline description

Step #1) Input mockup layout in the SVG format. It can be user-created or text-based generation using with Challenge #1 | Text-to-layout generator

Step #2) Based on the layout and text prompt, generate compatible images and texts via ControlNet [Github].

Step #3) User pick several inspiring options for further SVG re-implementation

Requirements

• ML model:
  • Text-to-layout generator: CLIP [Github] + UNet
  • Layout-to-content generator: ControlNet [Github] + UNet；GLIGEN [Github]
• Input: text prompt
• Output: layouts and various complete designs
• Dataset:
  • Text prompt ↔ layout pairs: follow UI description [Paper] to create
  • layout ↔ complete design pairs: Rico’17

Relevant works

[Research]

• Apple’21 proposed “parsing layout into components” [Website]

• LayoutDM [Github]

• BoostingGUI :first tries on controllable page-generation [Paper]

• UI description [Paper]

• Text-to-image generator:

  • Latent Diffusion Model [Paper] [Github]

  • Composable-Diffusion: support compositional text prompt [Github]

  • ControlNet: adding more control by image/sketch [Paper] [Github] [WebUI extension]

• Editing images:

  • GroundingDINO: regional image editing [Github]
  • Drag-your-GAN [Website] [Github]
  • Edit Everything [Paper] [Github]

[Business Solutions]

• Microsoft Designer [Website] [Demo]

Pros and Cons

🟢 Pros

• An existing high-quality open-source solution for image generation
• More research-active area

🔴 Cons

• Requires diverse new dataset and additional pre-processing
• Generation quality should be usable enough for the design application