This project has been created as part of the 42 curriculum by spacotto.

Description

This project introduces function calling in Large Language Models (LLMs) by building a self-contained system that translates natural language prompts into structured function calls with explicitly typed arguments. While standard applications rely on prompt engineering (which is inherently non-deterministic and prone to syntax hallucinations), this architecture implements constrained decoding at the engine level by intercepting raw token probabilities (logits) during the autoregressive generation loop. By dynamically applying a mathematical mask to eliminate any token that violates the target schema, the engine guarantees structurally valid JSON output with near-perfect reliability. This approach achieves deterministic compliance even when deploying a lightweight 0.5B parameter model in resource-constrained environments.

What is HuggingFace?

HuggingFace is a collaborative open-source platform and registry that hosts machine learning models, datasets, and AI libraries. It is essentially the "GitHub of AI," where models like Qwen and SmolLM are published and downloaded from.

What is Qwen (Qwen3-0.6B)?

A highly capable, open-source family of Large Language Models (LLMs) developed by Alibaba Cloud. The 0.6B (600 million parameter) version is used as the default in this project because its small size allows it to run entirely locally on standard CPUs, making it the perfect lightweight testbed for constrained decoding.

Instructions

The project uses uv for fast dependency management and a Makefile for execution.

Standard initialisation:

make install

Run Pipeline:

make run

Visualise the engine (runs batch size 1 with live state-machine visualisation):

make visual

Run the pipeline with the alternative model:

make run-alt ALT="HuggingFaceTB/SmolLM2-360M-Instruct"

Display the list of the available commands:

make help

Algorithm explanation

The constrained decoding pipeline operates in three phases to guarantee deterministic outputs:

• Phase 1, Zero-Shot Classification. The engine uses an O(1) masking cache to force the model to evaluate the prompt and output a valid function name from the provided JSON schemas.
• Phase 2, Argument Extraction. The engine dynamically bounds the generation length based on the target schema's shape. It traverses a SchemaTrie to mask out illegal structural characters (like misplaced brackets or quotes) while allowing the model to generate valid parameter values freely.
• Phase 3, Post-Processing & Validation. The raw generated strings are parsed into Python dictionaries. This phase serves as a final safety net, catching any JSONDecodeErrors, filtering out structural anomalies, and enforcing absolute schema compliance by injecting safe fallback objects when an edge case exceeds the extraction limits.

Important

For more detailed documentation, see the Engine Documentation.

Design decisions

• System Entry & Architecture (src/__main__.py)
• Data Ingestion (src/parser/)
• Context Preparation (src/formatter/)
• Low-Level Mechanics (src/tokenizer/)
• Constrained Decoding, Finalisation & Output (src/engine/)
• Visualisation (src/visualizer/)

Performance analysis: Accuracy, Speed, and Reliability

By replacing traditional "generate-and-pray" post-processing with pre-generation logit manipulation, the pipeline achieves:

• 100% JSON Structural Compliance: Guaranteed by the masking cache.
• High Throughput: O(1) lookups using an inverted index (string_to_ids) prevent the engine from slowing down during matrix generation.
• Memory Safety: Dynamic data chunking (BATCH_SIZE) prevents Out-Of-Memory exceptions during large processing runs.

Caution

Originally, the project sets 5 minutes on standard hardware as a reasonable speed. However, it has to be considered that the main speed bottleneck happens because the SDK's get_logits_from_input_ids() method only accepts a single 1D sequence at a time. Thus, the neural network's forward pass (the most computationally expensive part of the pipeline) had to be executed sequentially inside a for loop. Normally, in production frameworks like PyTorch or vLLM, you would pass a 2D tensor of shape (batch_size, sequence_length) directly into the model, allowing the GPU to compute the neural network layers for all 32 prompts simultaneously. This issue was countered as much as possible through Hybrid Batching: the bottleneck has been isolated by running the LLM predictions sequentially; then the results have been collected and stacked into a 2D NumPy array (logits_matrix = np.stack(batch_logits)); finally, all of the complex bitwise masking arrays and CFG validations have been applied simultaneously across the entire batch using optimised, vectorised C-math. The constrained decoding math has been parallelised, even though the project is restricted from parallelising the model itself.

Challenges faced

• System Entry & Architecture (src/__main__.py)
• Data Ingestion (src/parser/)
• Context Preparation (src/formatter/)
• Low-Level Mechanics (src/tokenizer/)
• Constrained Decoding, Finalisation & Output (src/engine/)
• Visualisation (src/visualizer/)

Testing strategy

The architecture is verified through a comprehensive, custom-built test suite (make test) that validates the project across multiple vectors:

• Input Validation: Ensures graceful failure on missing files or malformed definitions.
• Output Compliance: Strictly checks the generated JSON against the Pydantic-style schema models.
• Nested Parameters: Tests the engine's ability to retain state tracking deep within nested object dictionaries.
• Edge Cases: Throws adversarial inputs to verify the resilience of the PostProcessor and fallback mechanics.

Example usage

Run the engine against a custom schema and input file:

uv run python -m src \
  --functions_definition data/input/my_schema.json \
  --input data/input/my_prompts.json \
  --output data/output/results.json \
  --verbose

Resources

• Fast, High-Fidelity LLM Decoding with Regex Constraints
• Function Calling
• Hugging Face Transformers Documentation
• Large Language Models (LLMs)
• Logit
• Machine learning (ML)
• Natural language processing (NLP)
• Understanding Byte-Pair Encoding (BPE)
• Understand tokens
• uv Package Manager
• What are Logits in LLMs?

AI Usage

• Subject Research: Exploring and understanding the underlying mechanics, such as autoregressive generation, Byte-Level BPE tokenisation, logits manipulation, etc.
• Debugging: Assisting in diagnosing low-level errors, such as CUDA device-side assertions and byte-level BPE fragmentation issues.
• Code Quality & Strict Typing: Identifying and resolving formatting constraints and type-hinting errors (Flake8, Mypy) to ensure the codebase met required standards.
• Documentation: Drafting and structuring the README files and Python docstrings.