Phi Silica text completions are really slow

[alejandro5042]

While embedding performance is OK (about 2,000 tokens/second), text completions are crazy slow: 5 seconds for roughly 250 prompt tokens and 20 completion tokens.

I don't think my use case is too crazy -- it's a subset of a much larger AI problem (measured in tokens & desired intelligence) we are effectively solving with GPT 4o mini today.

The answer is borderline workable and will need lots of prompt engineering. But even if it nailed it, for this to be workable in a UI and to choose Phi Silica over cloud models for this specific use case, I need it under 1 second -- ideally under 500ms.

Are these chat completion token counts in the ballpark for what this feature is designed? Am I missing some key configuration, a model selector, or something? What can I expect out of Phi Silica?

For reference, my code looks roughly like:

using Microsoft.Windows.AI.Generative;

if (!LanguageModel.IsAvailable()) await LanguageModel.MakeAvailableAsync();

using LanguageModel languageModel = await LanguageModel.CreateAsync();

var prompt = "...";

var options = new LanguageModelOptions();
options.Temp = ...;
options.Top_p = ...;

// I run the next line over 10 to 100 iterations, and average the iteration time:
var result = await languageModel.GenerateResponseAsync(options, prompt);

My setup is:

• Dell Latitude 7455 with Snapdragon X Elite (X1E80100)
• Windows 26120.3863
• Microsoft.WindowsAppSDK v1.7.250127003-experimental3
• .NET 8.0

[connorMicrosoft]

Thanks @alejandro5042, we are looking into this.

[alejandro5042]

Are there plans for KV caching? Could greatly improve performance on the NPU to cache the system prompt and the current document (e.g. the first user message) to use over and over again.

[alejandro5042]

Another data point:

Using the AI Dev Gallery app, it takes 27 seconds (warm) to process the following text (about 1500 tokens) and generate output. For context, GPT 4o takes 2 seconds.

Introduction
This blog is the first installment in a new series of technical content designed to provide insights into the AI innovation on Windows. Today we will share how the Applied Sciences team used a multi-interdisciplinary approach to achieve a breakthrough in power efficiency, inference speed and memory efficiency for a state-of-the-art small language model (SLM), Phi Silica. Integrated into Windows 11 Copilot+ PCs (starting with Snapdragon X Series), this SLM powers several features of the latest generation of Copilot+ PC experiences: Click to Do (Preview), on-device rewrite and summarize capabilities in Word and Outlook, and a turnkey pre-optimized SLM for developers to utilize.

Background
In May, we introduced Copilot+ PCs, these devices include a Neural Processing Unit (NPU) capable of over 40 trillion operations per second (TOPS). During our May announcement, we also unveiled Phi Silica, the new on-device SLM available starting with Snapdragon X Series NPUs. Phi Silica is the sister-series of Phi models that leverages the NPU on Copilot+ PCs. At Ignite in November, we also announced that developers can access Phi Silica API starting in January 2025. Developers can bring language intelligence capabilities into their apps without needing to worry about model optimization or customization as Phi Silica is pre-tuned and ships inbox.

NPU devices with all day battery life claims run sustained AI workloads over long periods of time, in the background, with minimal impact to system fundamentals and resources. Connected to, and enhanced by the cloud, Copilot+ PCs can now achieve a level of performance never seen before – they are up to 20x more powerful1 and up to 100x as efficient2 for running AI workloads, and have smaller footprints than GPUs per TOPS/Watt/dollar. NPUs can sustain AI workloads that exhibit emergent behavior (3 to 7B parameter SLMs) in a semi-continuous loop, allowing users to make limitless low-latency queries to the model, without incurring additional subscription fees. This is a paradigm shift in compute; we now have the ability to run powerful reasoning agents as part of background operating system (OS) services, unlocking the potential for innovation across the range of our applications and services.

Original floating-point model
Phi Silica is based on a Cyber-EO compliant derivative of Phi-3.5-mini, developed specifically for Windows 11. It has a 4k context length, supports multiple languages including Tier 1 languages and others [English, Chinese (Simplified), French, German, Italian, Japanese, Portuguese, Spanish] and includes key improvements necessary for in-product experiences.

Cyber-EO compliant Phi-3.5 model benchmark performance
Cyber-EO compliant Phi-3.5 model benchmark performance (measured on BabelBench)
A language model, such as Phi, consists of several components:

Language model generation process
Language model generation process
The tokenizer breaks down the input text into smaller units and maps them to an index based on a pre-specified vocabulary. Tokenization forms a mapping between the language of humans and the language of models.
The detokenizer performs the reverse operation.
The embedding model transforms every discrete input token ID into a continuous, higher dimensional vector that captures semantic information in the space of language as understood by the language model. The direction of the normalized embedding vector encodes the context and the meaning of the text.
The transformer block transforms these incoming vectors to output vectors (or output hidden states) that point in the direction of the token that should follow the current one.
The language model head computes the most likely token based on the output vectors.
Generating a response to a prompt consists of two distinct phases of operation of the transformer block:

Context processing: The language model processes input tokens to compute the key-value (KV) cache and generate hidden states and the first token. This involves intense parallel computation, mainly matrix multiplications, requiring high computational power.
Token iteration: Tokens are generated one by one (i.e. autoregressively) and each new token becomes part of the extended context to predict the next one. Generation stops when an end token is produced, or a user-defined condition is met.
Running the aforementioned stages for even SLMs such as Phi, with their billions of parameters, can place considerable strain on your device. The context processing stage requires significant computational resources, which impacts the CPU and running applications, and involves high power usage when GPUs are employed due to their efficiency in TOPS per Watt. In contrast, the token iteration stage demands substantial memory for storing and accessing the KV cache for each token generation step. While it needs less computation, efficient memory access is crucial for maintaining performance. Memory constraints make efficient token generation challenging.

NPUs within Copilot+ PCs are built to be power-efficient, capable of executing several TOPS within a single-digit Watt range. On Copilot+ PC devices with Snapdragon X Elite processors, Phi Silica’s context processing consumes only 4.8mWh of energy on the NPU, while the token iterator exhibits a 56% improvement in power consumption compared to operation on the CPU. Consequently, we can execute Phi Silica on your device without burdening the CPU and GPU, ensuring efficient memory and power consumption, thereby allowing this highly capable and versatile model to run seamlessly, with minimal impact on your primary applications and experiences.

As NPUs are domain-specific processors, we have employed various techniques to achieve an optimal balance between efficiency and performance without compromising accuracy. We are eager to share these techniques and hope they can be applied to other small language models as well. Our discussion will primarily focus on optimizing and offloading the transformer block to the NPU. The tokenizer, embedding and language model head are not compute-intensive but involve lookups; therefore, we allocate these tasks to the CPU.

Creating Phi Silica
Considering the size of the original floating-point model, memory limitations of the target hardware, as well as the desired performance metrics in terms of speed, memory usage and power efficiency, it was clear that Phi Silica should have the following characteristics:

4-bit weight quantization to ensure high speed and a low memory footprint during inferencing
Low idle memory consumption to support pinned memory and eliminate initialization costs
Rapid time to first token for shorter prompts to enhance interactivity
A context length of 2k or greater to ensure real-world usability
NPU-based operation to achieve power efficiency in sustained usage
High accuracy across multiple languages
Small model disk size to make distribution at Windows scale efficient
We designed Phi Silica with these goals in mind for the current generation NPUs. In doing so, we pushed the envelope on what’s possible today across several levels of the stack such as post-training quantization, efficient resource use in inference software and targeted silicon-specific optimizations across operator placement and model graph. The result is a model that delivers, with the bulk of compute offloaded:

Time to first token: 230ms for short prompts
Throughput rate: Up to 20 tokens/s
Context length: 2k (with support for 4k coming shortly)
Sustained NPU-based context processing and token iteration

---

Summarize the above article in 20 words.

To generate this output:

Microsoft's Windows 11 Copilot+ PCs, using NPUs, unlock new language AI potential, showcasing a multi-interdisciplinary approach to developing Phi Silica, achieving unprecedented performance and energy efficiency for state-of-the-art small language models.

While generating, the NPU starting at around 75%, dropping a few % every second while processing the input, then plateauing at 80% while generating output. (Hopefully we can peg them both at 100%?)

The laptop is plugged in.