giftuber

Renderer-agnostic utilities for loading and rendering GIFTuber avatar manifests.

GIFTuber renders avatar assets on a Canvas using GIF animations as the main visual source. A manifest can also include still image layers, such as mouth shapes and talk overlays, so applications can combine expressions, blinking, and lip sync without requiring Live2D, VRM, or a character rig.

日本語 README

Install

npm install giftuber

Usage

Web Speech API (zero setup)

import {
  createAvatarRenderer,
  createWebSpeechController,
  loadAvatarAssets,
  loadManifest,
} from "giftuber";

const manifest = await loadManifest("/avatar-assets/manifest.json");
const assets = await loadAvatarAssets(manifest);

const renderer = createAvatarRenderer({
  canvas: document.querySelector("canvas")!,
  manifest,
  assets,
});

renderer.setEmotion("happy");
renderer.start();

const speech = createWebSpeechController({ renderer });

speech.speak({
  text: "こんにちは。ブラウザの音声合成で話しています。",
  lang: "ja-JP",
  rate: 1,
  pitch: 1,
  volume: 1,
});

createWebSpeechController uses the browser's built-in speechSynthesis, so it can speak without an API key, local TTS server, or generated AudioBuffer. It drives mouth shapes from SpeechSynthesisUtterance boundary events when the browser provides them, and falls back to an estimated text timeline otherwise.

Use listVoices({ lang: "ja" }) to get a Promise-based voice list that also handles Chrome's initially empty speechSynthesis.getVoices() result.

Core renderer

import {
  buildMouthTimeline,
  createAvatarRenderer,
  createSpeechController,
  loadAvatarAssets,
  loadManifest,
} from "giftuber";

const manifest = await loadManifest("/avatar-assets/manifest.json");
const assets = await loadAvatarAssets(manifest);

const renderer = createAvatarRenderer({
  canvas: document.querySelector("canvas")!,
  manifest,
  assets,
});

renderer.setEmotion("happy");
renderer.start();

const speech = createSpeechController({ renderer });
const timeline = buildMouthTimeline("こんにちは", 1200);

speech.speakSilent(timeline, 1200);

The core renderer is framework independent. It only draws the current state given by the host application:

• setEmotion(name | null) selects an idle expression, falling back to neutral when unset or unknown.
• setSpeaking(boolean) switches between idle and speech rendering.
• setMouthShape(shape) selects a mouth asset, falling back to mouth.neutral when the requested shape is unavailable.
• start() and stop() control the requestAnimationFrame loop.
• dispose() stops the loop and releases loaded ImageBitmap resources when possible.

loadAvatarAssets(manifest) preloads all image assets. GIF assets are decoded into timed frames before rendering.

VOICEVOX / AivisSpeech audio_query

import { buildMouthTimelineFromAudioQuery } from "giftuber";

const audioQuery = await fetch("/audio_query").then((response) =>
  response.json(),
);
const timeline = buildMouthTimelineFromAudioQuery(audioQuery);

await speech.speak({ audioBuffer, timeline });

buildMouthTimelineFromAudioQuery accepts the engine-independent shape used by VOICEVOX and AivisSpeech: accent_phrases[].moras[], pause_mora, prePhonemeLength, postPhonemeLength, and speedScale.

React

import {
  GiftuberAvatar,
  listVoices,
  useGiftuberAvatar,
} from "giftuber/react";
import { buildMouthTimeline } from "giftuber";

function Avatar() {
  const avatar = useGiftuberAvatar({
    manifestUrl: "/avatar-assets/manifest.json",
    emotion: "happy",
  });

  return (
    <>
      <canvas ref={avatar.canvasRef} />
      <button
        disabled={!avatar.ready}
        onClick={() =>
          avatar.speakSilent(buildMouthTimeline("Hello!", 900), 900)
        }
      >
        Speak
      </button>
      <button
        disabled={!avatar.ready}
        onClick={() =>
          avatar.speakWithWebSpeech({
            text: "こんにちは",
            lang: "ja-JP",
          })
        }
      >
        Speak with browser voice
      </button>
    </>
  );
}

function StaticAvatar() {
  return <GiftuberAvatar manifestUrl="/avatar-assets/manifest.json" />;
}

const japaneseVoices = await listVoices({ lang: "ja" });

Memoize rendererOptions before passing it to useGiftuberAvatar; a new object identity causes the hook to recreate the renderer.

Demo

npm run demo

Place a local avatar asset set at demo/avatar-assets/manifest.json. The demo/avatar-assets/ directory is ignored by Git so generated or private assets are not committed. The demo uses Web Speech API first, so a supported browser can speak aloud and drive mouth shapes without a local TTS server. Japanese voice availability and quality vary by browser and OS.

Manifest

The package currently supports GIFTuber Avatar Manifest v1 only. See docs/manifest-spec.md for the draft manifest contract.

License

MIT — see LICENSE.