Skip to content

3.19.0

Choose a tag to compare

View all tags
@fernandotonon fernandotonon released this 09 Jul 23:03
3.19.0
c30bacb

Image → 3D: 4-view AI texture bake + describe-then-generate

This release gives the TripoSG image→3D meshes a real, image-derived texture — and fixes the chain of issues that kept that texture from reaching the screen.

4-view AI texture bake (MV-Adapter slice 1, #805)

  • Generates front / back / left / right depth-ControlNet views (MV-Adapter's 6-view orthographic layout minus the two poles), projects them onto the mesh's UVs, and blends — a clear step up from the old front+back (which left the sides to stretch/blur) without the ~3× cost of all six.
  • Describe-then-generate: a local SmolVLM (Apache-2.0) vision model captions the input image in the background the moment it's selected; that caption drives every view's generation, so all sides are stylistically consistent (shared caption + locked seed) with no photo-projection registration artifact.
  • The mesh is degenerate-cleaned, decimated to a tri budget, and UV-unwrapped before the SD views run, so every view and the final bake share the same geometry.

Fixes

  • Baked texture now actually renders on geometry-only (TripoSG) meshes. The shared neutral-clay material had no texture unit to bind the diffuse to, and its stale RTSS technique survived a normal rebind — so the mesh stayed clay even though the bake succeeded. Now the diffuse binds, the shared material is cloned per-entity, vertex-colour tracking is cleared, and the RTSS technique is regenerated so it shows immediately (no manual material re-apply needed).
  • Correct image captions. SmolVLM was parroting a worked example from the prompt (a person came back described as a rabbit). Removed the example, rewrote the prompt to name the subject first then its surface detail (skin/hair/eyes/makeup/clothing), fixed the chat template to match the model exactly, and serialised captioning.
  • Progress no longer freezes at "view 4/4" — the completion signal is emitted once after the real bake, not per view.

Models

  • SmolVLM-500M-Instruct (Apache-2.0) — hosted on the fernandotonon/QtMeshEditor-models HF repo, downloads on first use.

License

Everything shipping is permissive and safe for commercial use + redistribution (TripoSG MIT, DINOv2 / SmolVLM / U²-Net Apache-2.0, xatlas / meshoptimizer MIT). No non-commercial or field-of-use-restricted components.

Full details: PR #827.

Assets 13
Loading