Example scripts revamp

QEfficient/cloud/execute.py

@@ -115,7 +115,7 @@ def main(
         "--prompts_txt_file_path",
         "--prompts-txt-file-path",
         type=str,
-        help="File path for taking input prompts from txt file, sample prompts.txt file present in examples folder",
+        help="File path for taking input prompts from txt file, sample prompts.txt file present in examples/sample_prompts folder",
     )
     parser.add_argument("--generation_len", "--generation-len", type=int, help="Number of tokens to generate")
     parser.add_argument(

QEfficient/cloud/infer.py

@@ -390,7 +390,7 @@ def main(
         "--prompts_txt_file_path",
         "--prompts-txt-file-path",
         type=str,
-        help="File path for taking input prompts from txt file, sample prompts.txt file present in examples folder",
+        help="File path for taking input prompts from txt file, sample prompts.txt file present in examples/sample_prompts folder",
     )
     parser.add_argument("--generation_len", "--generation-len", type=int, help="Number of tokens to generate")
     parser.add_argument(

docs/source/quick_start.md

@@ -125,10 +125,10 @@ You can pass input prompts in single string but separate with pipe (|) symbol".
 python -m QEfficient.cloud.infer --model_name gpt2 --batch_size 3 --prompt_len 32 --ctx_len 128 --num_cores 16 --device_group [0] --prompt "My name is|The flat earth theory is the belief that|The sun rises from" --mxfp6 --mos 1 --aic_enable_depth_first
 ```
 
-You can also pass path of txt file with input prompts when you want to run inference on lot of prompts, Example below, sample txt file(prompts.txt) is present in examples folder.
+You can also pass path of txt file with input prompts when you want to run inference on lot of prompts, Example below, sample txt file(prompts.txt) is present in examples/sample_prompts folder.
 
 ```bash
-python -m QEfficient.cloud.infer --model_name gpt2 --batch_size 3 --prompt_len 32 --ctx_len 128 --num_cores 16 --device_group [0] --prompts_txt_file_path examples/prompts.txt --mxfp6 --mos 1 --aic_enable_depth_first
+python -m QEfficient.cloud.infer --model_name gpt2 --batch_size 3 --prompt_len 32 --ctx_len 128 --num_cores 16 --device_group [0] --prompts_txt_file_path examples/sample_prompts/prompts.txt --mxfp6 --mos 1 --aic_enable_depth_first
 ```
 **QNN CLI Inference Command**
 

docs/source/release_docs.md

@@ -13,7 +13,7 @@ Welcome to the official release of **Efficient Transformer Library v1.20.0**! Th
   - Text & Image+Text support
   - Chunk attention, Single/Dual QPC support
   - Multi-image prompts enabled via VLLM interface
-  - [Llama4 Example Script](https://github.com/quic/efficient-transformers/blob/main/examples/llama4_example.py)
+  - [Llama4 Example Script](https://github.com/quic/efficient-transformers/blob/main/examples/image_text_to_text/models/llama_vision/single_image.py)
 
 - **Grok-1**
   - Executable via [`QEffAutoModelForCausalLM`](#QEffAutoModelForCausalLM)
@@ -22,7 +22,7 @@ Welcome to the official release of **Efficient Transformer Library v1.20.0**! Th
   - Executable via [`QEFFAutoModelForImageTextToText`](#QEFFAutoModelForImageTextToText)
   - Text & Image+Text support
   - Sliding window support
-  - [Gemma3 Example Script](https://github.com/quic/efficient-transformers/blob/main/examples/gemma3_example/gemma3_mm.py)
+  - [Gemma3 Example Script](https://github.com/quic/efficient-transformers/blob/main/examples/image_text_to_text/models/gemma_vision/inference.py)
 
 
 - **SwiftKV (Llama-3.1-SwiftKV-8B-Instruct)**
@@ -32,7 +32,7 @@ Welcome to the official release of **Efficient Transformer Library v1.20.0**! Th
 - **GGUF Models**
   - Executable via [`QEffAutoModelForCausalLM`](#QEffAutoModelForCausalLM)
   - Execution support (non-quantized)
-  - [Example Script](https://github.com/quic/efficient-transformers/blob/main/examples/basic_gguf_models.py)
+  - [Example Script](https://github.com/quic/efficient-transformers/blob/main/examples/text_generation/gguf_models.py)
 
 - **FP8 Compressed Quantization**
   - Support for [`Llama-3.3-70B-Instruct-FP8-Dynamic`](https://huggingface.co/Infermatic/Llama-3.3-70B-Instruct-FP8-Dynamic)

docs/source/supported_features.rst

@@ -6,46 +6,48 @@ Supported Features
 
    * - Feature
      - Impact
+   * - `Compute Context Length (CCL) <https://github.com/quic/efficient-transformers/blob/main/examples/performance/compute_context_length/README.md>`_
+     - Optimizes inference by using different context lengths during prefill and decode phases, reducing memory footprint and computation for shorter sequences while maintaining support for longer contexts. Supports both text-only and vision-language models. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/performance/compute_context_length/basic_inference.py>`_ for more **details**.
    * - Sentence embedding, Flexible Pooling configuration and compilation with multiple sequence lengths
-     - Supports standard/custom pooling with AI 100 acceleration and sentence embedding. Enables efficient sentence embeddings via Efficient-Transformers. Compile with one or multiple seq_len; optimal graph auto-selected at runtime. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/embedding_model.py>`_ for more **details**.
+     - Supports standard/custom pooling with AI 100 acceleration and sentence embedding. Enables efficient sentence embeddings via Efficient-Transformers. Compile with one or multiple seq_len; optimal graph auto-selected at runtime. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/embeddings/sentence_embeddings.py>`_ for more **details**.
    * - `SpD, multiprojection heads <https://quic.github.io/efficient-transformers/source/quick_start.html#draft-based-speculative-decoding>`_
-     - Implemented post-attention hidden size projections to speculate tokens ahead of the base model. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/multiprojs_spd_inference.py>`_ for more **details**.
+     - Implemented post-attention hidden size projections to speculate tokens ahead of the base model. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/performance/speculative_decoding/multi_projection.py>`_ for more **details**.
    * - `QNN Compilation support <https://github.com/quic/efficient-transformers/pull/374>`_
      - Enabled for AutoModel classes QNN compilation capabilities for multi-models, embedding models and causal models.
    * - `Disaggregated serving <https://github.com/quic/efficient-transformers/pull/365>`_
      - It support for separate prefill and decode compilation for encoder (vision) and language models.
    * - `GGUF model execution <https://github.com/quic/efficient-transformers/pull/368>`_
-     - Supported GGUF model execution (without quantized weights). Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/basic_gguf_models.py>`_ for more **details**.
+     - Supported GGUF model execution (without quantized weights). Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/text_generation/gguf_models.py>`_ for more **details**.
    * - Replication of KV
      - Enabled FP8 model support on `replicate_kv_heads script <https://github.com/quic/efficient-transformers/tree/main/scripts/replicate_kv_head>`_.
    * - `gradient checkpointing <https://github.com/quic/efficient-transformers/pull/338>`_
      - Supports gradient checkpointing in the finetuning script
    * - Swift KV `Snowflake/Llama-3.1-SwiftKV-8B-Instruct <https://huggingface.co/Snowflake/Llama-3.1-SwiftKV-8B-Instruct>`_
      - Reduces computational overhead during inference by optimizing key-value pair processing, leading to improved throughput. Support for both `continuous and non-continuous batching execution <https://github.com/quic/efficient-transformers/pull/367>`_ in SwiftKV
    * - :ref:`Vision Language Model <QEFFAutoModelForImageTextToText>`
-     - Provides support for the AutoModelForImageTextToText class from the transformers library, enabling advanced vision-language tasks. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/image_text_to_text_inference.py>`_ for more **details**.
+     - Provides support for the AutoModelForImageTextToText class from the transformers library, enabling advanced vision-language tasks. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/image_text_to_text/basic_vlm_inference.py>`_ for more **details**.
    * - :ref:`Speech Sequence to Sequence Model <QEFFAutoModelForSpeechSeq2Seq>`
-     - Provides support for the QEFFAutoModelForSpeechSeq2Seq Facilitates speech-to-text sequence models. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/speech_to_text/run_whisper_speech_to_text.py>`_ for more **details**.
+     - Provides support for the QEFFAutoModelForSpeechSeq2Seq Facilitates speech-to-text sequence models. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/audio/speech_to_text.py>`_ for more **details**.
    * - Support for FP8 Execution
      - Enables execution with FP8 precision, significantly improving performance and reducing memory usage for computational tasks.
    * - Prefill caching
      - Enhances inference speed by caching key-value pairs for shared prefixes, reducing redundant computations and improving efficiency.
    * - On Device Sampling
      - Enables sampling operations to be executed directly on the QAIC device rather than the host CPU for QEffForCausalLM models. This enhancement significantly reduces host-device communication overhead and improves inference throughput and scalability. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/on_device_sampling.py>`_ for more **details**.
    * - Prompt-Lookup Decoding
-     - Speeds up text generation by using overlapping parts of the input prompt and the generated text, making the process faster without losing quality. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/pld_spd_inference.py>`_ for more **details**.
+     - Speeds up text generation by using overlapping parts of the input prompt and the generated text, making the process faster without losing quality. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/performance/speculative_decoding/prompt_lookup.py>`_ for more **details**.
    * - :ref:`PEFT LoRA support <QEffAutoPeftModelForCausalLM>`
-     - Enables parameter-efficient fine-tuning using low-rank adaptation techniques, reducing the computational and memory requirements for fine-tuning large models. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/peft_models.py>`_ for more **details**.
+     - Enables parameter-efficient fine-tuning using low-rank adaptation techniques, reducing the computational and memory requirements for fine-tuning large models. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/peft/single_adapter.py>`_ for more **details**.
    * - :ref:`QNN support <id-qnn-compilation-via-python-api>`
      - Enables compilation using QNN SDK, making Qeff adaptable for various backends in the future.
    * - :ref:`Embedding model support <QEFFAutoModel>`
      - Facilitates the generation of vector embeddings for retrieval tasks.
    * - :ref:`Speculative Decoding <id-draft-based-speculative-decoding>`
-     - Accelerates text generation by using a draft model to generate preliminary predictions, which are then verified by the target model, reducing latency and improving efficiency. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/draft_spd_inference.py>`_ for more **details**.
+     - Accelerates text generation by using a draft model to generate preliminary predictions, which are then verified by the target model, reducing latency and improving efficiency. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/performance/speculative_decoding/draft_based.py>`_ for more **details**.
    * - :ref:`Finite lorax <QEffAutoLoraModelForCausalLM>`
-     - Users can activate multiple LoRA adapters and compile them with the base model. At runtime, they can specify which prompt should use which adapter, enabling mixed adapter usage within the same batch. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/lora_models.py>`_ for more **details**.
+     - Users can activate multiple LoRA adapters and compile them with the base model. At runtime, they can specify which prompt should use which adapter, enabling mixed adapter usage within the same batch. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/peft/multi_adapter.py>`_ for more **details**.
    * - Python and CPP Inferencing API support
-     - Provides flexibility while running inference with Qeff and enabling integration with various applications and improving accessibility for developers. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/cpp_execution/text_inference_using_cpp.py>`_ for more **details**.
+     - Provides flexibility while running inference with Qeff and enabling integration with various applications and improving accessibility for developers. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/examples/performance/cpp_execution/text_inference_cpp.py>`_ for more **details**.
    * - :ref:`Continuous batching <id-continuous-batching>`
      - Optimizes throughput and latency by dynamically batching requests, ensuring efficient use of computational resources.
    * - AWQ and GPTQ support
@@ -56,7 +58,5 @@ Supported Features
      - A script for computing the perplexity of a model, allowing for the evaluation of model performance and comparison across different models and datasets. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/scripts/perplexity_computation/calculate_perplexity.py>`_ for more **details**.
    * - KV Heads Replication Script
      - A sample script for replicating key-value (KV) heads for the Llama-3-8B-Instruct model, running inference with the original model, replicating KV heads, validating changes, and exporting the modified model to ONNX format. Refer `sample script <https://github.com/quic/efficient-transformers/blob/main/scripts/replicate_kv_head/replicate_kv_heads.py>`_ for more **details**.
-   * - Context Length Specializations (upcoming)
-     - Increases the maximum context length that models can handle, allowing for better performance on tasks requiring long sequences of text.
    * - Block Attention (in progress)
-     - Reduces inference latency and computational cost by dividing context into blocks and reusing key-value states, particularly useful in RAG.
+     - Reduces inference latency and computational cost by dividing context into blocks and reusing key-value states, particularly useful in RAG.