[feat]: add MLX INT8 FastWan-QAD 1.3B inference on Apple Silicon - #12
[feat]: add MLX INT8 FastWan-QAD 1.3B inference on Apple Silicon#12aryan5v wants to merge 1 commit into
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces extensive agent infrastructure, including onboarding guides, contracts, and reusable skills under .agents/, alongside the integration of the Dreamverse real-time video generation and editing platform under apps/dreamverse/. It also updates Buildkite pipelines to support new test suites and cleans up repository configuration files. The code review feedback identifies several critical issues in the Dreamverse prompt enhancer module: first, the initialization of provider runtimes incorrectly requires all API keys to be present, breaking single-provider setups; second, saving prompt configurations erroneously overwrites default git-tracked files instead of utilizing local overrides; and finally, the requested JSON response format is not forwarded to the Cerebras and OpenAI-compatible clients, which can cause parsing failures. All of these comments are highly actionable and should be addressed.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| for provider_name in PROMPT_PROVIDER_PRIORITY: | ||
| api_key = PROMPT_API_KEYS[provider_name] | ||
| api_base_url = PROMPT_API_BASE_URLS[provider_name] | ||
| if not isinstance(api_key, str) or not api_key.strip(): | ||
| env_names = ", ".join(PROMPT_PROVIDER_API_KEY_NAMES[provider_name]) | ||
| raise RuntimeError("Missing required environment variable: one of " | ||
| f"{env_names}") | ||
| runtimes.append( | ||
| ProviderRuntime( | ||
| name=provider_name, | ||
| api_key=api_key, | ||
| api_base_url=api_base_url, | ||
| request_model=self.provider_request_models.get( | ||
| provider_name, | ||
| self.model, | ||
| ), | ||
| client=self._build_client( | ||
| provider=provider_name, | ||
| api_key=api_key, | ||
| api_base_url=api_base_url, | ||
| ), | ||
| )) | ||
| return runtimes |
There was a problem hiding this comment.
The current implementation of _build_provider_runtimes iterates over all providers in PROMPT_PROVIDER_PRIORITY (which includes both "cerebras" and "groq") and raises a RuntimeError if any of their API keys are missing. This makes both API keys mandatory at startup, completely breaking single-provider setups and forcing users to supply credentials for providers they do not intend to use.
Instead, the loop should skip any provider whose API key is not set, and only raise an error if no valid providers are configured at all.
for provider_name in PROMPT_PROVIDER_PRIORITY:
api_key = PROMPT_API_KEYS[provider_name]
api_base_url = PROMPT_API_BASE_URLS[provider_name]
if not isinstance(api_key, str) or not api_key.strip():
continue
runtimes.append(
ProviderRuntime(
name=provider_name,
api_key=api_key,
api_base_url=api_base_url,
request_model=self.provider_request_models.get(
provider_name,
self.model,
),
client=self._build_client(
provider=provider_name,
api_key=api_key,
api_base_url=api_base_url,
),
))
if not runtimes:
raise RuntimeError("No prompt providers configured. Please set at least one API key (e.g., CEREBRAS_API_KEY or GROQ_API_KEY).")
return runtimes| _save_prompt( | ||
| _resolve_prompt_save_path( | ||
| self.enhance_system_prompt_path, | ||
| self.enhance_system_prompt_fallback_path, | ||
| ), | ||
| normalized, | ||
| "next-segment", | ||
| ) | ||
|
|
||
| if auto_extension_system_prompt is not None: | ||
| normalized = auto_extension_system_prompt.strip() | ||
| if not normalized: | ||
| raise ValueError("auto_extension_system_prompt cannot be empty.") | ||
| _save_prompt( | ||
| _resolve_prompt_save_path( | ||
| self.auto_system_prompt_path, | ||
| self.auto_system_prompt_fallback_path, | ||
| ), | ||
| normalized, | ||
| "auto-extension", | ||
| ) | ||
|
|
||
| if rewrite_window_system_prompt is not None: | ||
| normalized = rewrite_window_system_prompt.strip() | ||
| if not normalized: | ||
| raise ValueError("rewrite_window_system_prompt cannot be empty.") | ||
| _save_prompt( | ||
| _resolve_prompt_save_path( | ||
| self.rewrite_all_system_prompt_path, | ||
| self.rewrite_all_system_prompt_fallback_path, | ||
| ), | ||
| normalized, | ||
| "rewrite-window", | ||
| ) | ||
|
|
||
| if rewrite_user_system_prompt is not None: | ||
| normalized = rewrite_user_system_prompt.strip() | ||
| if not normalized: | ||
| raise ValueError("rewrite_user_system_prompt cannot be empty.") | ||
| _save_prompt( | ||
| _resolve_prompt_save_path( | ||
| self.rewrite_user_system_prompt_path, | ||
| self.rewrite_user_system_prompt_fallback_path, | ||
| ), | ||
| normalized, | ||
| "rewrite-user", | ||
| ) |
There was a problem hiding this comment.
In save_prompt_config, the code calls _resolve_prompt_save_path to determine where to save the edited system prompts. However, _resolve_prompt_save_path is designed to fall back to the default tracked path (e.g., prompts/next_segment_system_prompt.md) if the local override file in prompts.local/ does not exist yet.
As a result, when a user edits a prompt in the UI for the first time, the changes will overwrite the default git-tracked files in prompts/ instead of creating the local override files in prompts.local/. This defeats the purpose of having a local overlay directory and pollutes the git repository.
To fix this, save_prompt_config should write directly to the primary paths (self.enhance_system_prompt_path, self.auto_system_prompt_path, etc.) without calling _resolve_prompt_save_path.
_save_prompt(
self.enhance_system_prompt_path,
normalized,
"next-segment",
)
if auto_extension_system_prompt is not None:
normalized = auto_extension_system_prompt.strip()
if not normalized:
raise ValueError("auto_extension_system_prompt cannot be empty.")
_save_prompt(
self.auto_system_prompt_path,
normalized,
"auto-extension",
)
if rewrite_window_system_prompt is not None:
normalized = rewrite_window_system_prompt.strip()
if not normalized:
raise ValueError("rewrite_window_system_prompt cannot be empty.")
_save_prompt(
self.rewrite_all_system_prompt_path,
normalized,
"rewrite-window",
)
if rewrite_user_system_prompt is not None:
normalized = rewrite_user_system_prompt.strip()
if not normalized:
raise ValueError("rewrite_user_system_prompt cannot be empty.")
_save_prompt(
self.rewrite_user_system_prompt_path,
normalized,
"rewrite-user",
)| cerebras_body = { | ||
| "model": body.get("model"), | ||
| "messages": body.get("messages"), | ||
| "temperature": body.get("temperature"), | ||
| } | ||
| cerebras_body = {key: value for key, value in cerebras_body.items() if value is not None} |
There was a problem hiding this comment.
The _build_body method sets "response_format": {"type": "json_object"} to request JSON output from the LLM. However, _request_content_with_body_cerebras does not pass response_format to the Cerebras client's chat.completions.create call.
Without response_format being forwarded to the API, the model is not guaranteed to return JSON, which can lead to parsing failures in _parse_json_response. We should pass response_format to the client if it is present in the body.
cerebras_body = {
"model": body.get("model"),
"messages": body.get("messages"),
"temperature": body.get("temperature"),
"response_format": body.get("response_format"),
}
cerebras_body = {key: value for key, value in cerebras_body.items() if value is not None}| openai_body = { | ||
| "model": body.get("model"), | ||
| "messages": body.get("messages"), | ||
| "temperature": body.get("temperature"), | ||
| } | ||
| max_completion_tokens = body.get("max_completion_tokens") | ||
| if max_completion_tokens is not None: | ||
| openai_body["max_completion_tokens"] = max_completion_tokens |
There was a problem hiding this comment.
The _build_body method sets "response_format": {"type": "json_object"} to request JSON output from the LLM. However, _request_content_with_body_openai_compatible does not pass response_format to the OpenAI-compatible client's chat.completions.create call.
Without response_format being forwarded to the API, the model is not guaranteed to return JSON, which can lead to parsing failures in _parse_json_response. We should pass response_format to the client if it is present in the body.
| openai_body = { | |
| "model": body.get("model"), | |
| "messages": body.get("messages"), | |
| "temperature": body.get("temperature"), | |
| } | |
| max_completion_tokens = body.get("max_completion_tokens") | |
| if max_completion_tokens is not None: | |
| openai_body["max_completion_tokens"] = max_completion_tokens | |
| openai_body = { | |
| "model": body.get("model"), | |
| "messages": body.get("messages"), | |
| "temperature": body.get("temperature"), | |
| "response_format": body.get("response_format"), | |
| } | |
| max_completion_tokens = body.get("max_completion_tokens") | |
| if max_completion_tokens is not None: | |
| openai_body["max_completion_tokens"] = max_completion_tokens |
d60d8e1 to
7ed0356
Compare
aryan5v
changed the title
7ed0356 to
7822aed
Compare
1 participant
Summary
mx.compilesupport.--fast. It generates fewer diffusion frames and uses the Apple-Silicon-nativerife-mlxbackend to interpolate back to the requested target frame count.Install
The
mlxextra installs both the MLX runtime andrife-mlxrequired by--fast.Generate with RIFE fast mode
--num-framesremains the final target length. With the default--fast-factor 2, the runtime generates approximately half the frames and RIFE interpolates the final video to the requested length. Fast mode also supports--fast-sharpenand composes with INT8 quantization,mx.compile, TAEHV, and full Wan VAE decoding.Test
Documentation
docs/getting_started/installation/mps.md: Apple Silicon installation, model setup, generation, quality mode, and troubleshooting.docs/design/apple_silicon_fast_mode.md: RIFE fast-mode behavior, installation, CLI flags, performance, and quality guidance.docs/inference/support_matrix.mdandREADME.md: runtime support and discoverability.Validation
pre-commit run --from-ref origin/main --to-ref HEAD: passed.python -m pytest fastvideo/tests/mlx fastvideo/tests/platforms/test_mps_vsa_error.py -q: 42 passed on Apple Silicon with MLX 0.31.2 and Torch 2.12.0.git diff --check: passed.pyproject.tomlparse check: passed.