This project has been created as part of the 42 curriculum by moezzoub.
CallMeMaybe is a function-calling system powered by a local Large Language Model (LLM).
The goal of the project is to transform natural language prompts into structured function calls using constrained decoding and validation.
The program:
- Reads function definitions from a JSON file
- Reads natural language prompts
- Uses an LLM to select the correct function and arguments
- Extracts and validates the generated JSON
- Outputs a strictly formatted JSON file matching the required schema
This project focuses on LLM control, structured generation, and robust validation.
uv syncuv run python -m srcmake lintuv run python -m srcOR
uv run python -m src --functions_definition data/input/functions_definition.json --input data/input/function_calling_tests.json --output data/output/function_calling_results.jsonThe main challenge is forcing the LLM to produce valid structured JSON.
The approach used:
- Build a structured prompt including:
- clear instructions
- available functions and parameters
- the user request
- Generate tokens step by step using:
get_logits_from_input_ids- the generation process is guided toward structured JSON output and validated against the expected schema before writing results
- Stop generation early when a complete JSON object is detected:
- detect {
- track braces ({ / })
- stop when balanced
- Extract only the JSON part from the raw output
- Validate the result using a strict schema (FunctionCall)
This ensures:
- minimal extra text
- valid JSON extraction
- correct structure
- Manual JSON extraction
Instead of trusting the LLM output, the program:
- scans for { ... }
- extracts only valid JSON
Reason: LLMs often produce extra text or invalid formatting.
- Validation with Pydantic
All outputs are validated against a strict model:
- function name
- parameters
- types
Reason: Guarantees compliance with the subject and prevents invalid outputs.
- Environment-based configuration
The model can be changed via environment variable:
CALLMEMAYBE_MODEL=Qwen/Qwen3-0.6BReason: Supports multiple models without modifying the code.
Initial implementation:
- ~38 minutes
Optimized version:
- ~5 minutes
Optimizations:
- early stopping when JSON is complete
- reduced token generation
- minimal prompt size
Accuracy is ensured through:
- strict validation
- controlled decoding
- strict argument validation
The system is robust against:
- extra LLM text
- malformed JSON
- incorrect arguments
Invalid outputs are detected and handled safely.
- Enforcing strict JSON output
Problem: LLM generates text + JSON + noise
Solution:
- extract JSON manually
- ignore everything else
- Matching function parameters exactly
Problem: LLM may use wrong keys
Solution:
- compare expected vs generated keys
- correct specific cases when needed
- Performance issues
Problem: Generation was too slow (~38 minutes)
Solution:
- stop generation early when JSON is complete
- reduce max tokens
- LLM inconsistency
Problem: Same prompt → different outputs
Solution:
- constrained function selection
- deterministic validation
Validation is done at multiple levels:
- JSON structure validation
- Function existence check
- Parameter key matching
- Type consistency
Additionally:
- moulinette tests used as final validation
- manual tests for edge cases
- LLM tokenization and decoding concepts
- Pydantic documentation
- JSON specification
- LLM function calling concepts
AI was used to:
- understand constrained decoding concepts
- debug edge cases
- improve prompt design
Core logic and decisions were implemented and validated manually.
src/
├── __main__.py
├── generator.py
├── loader.py
├── models.py
└── writer.py
data/
├── input/
└── output/ (Generated at runtime)
- The core logic was stabilized before adding bonuses
- All bonuses were added without modifying grading behavior
- The project focuses on correctness, robustness, and control over LLM output