How Do Prompt Variations Affect Energy Consumption in On-Device LLMs?

Project page · Paper

This repository contains the prompt datasets, power collection tools, and analysis scripts for our EMNLP paper. The study examines how two prompt properties, cognitive load and phrasing pattern, shape the energy behavior of on-device LLM inference. We profile five lightweight LLMs (Gemma-2-2B, LLaMA-3.2-1B, Qwen2.5-0.5B, Qwen2.5-1.5B, SmolLM2-360M) on Pixel 7 and Pixel 8 Pro phones, separating prefill and decode energy at the phase level.

The on-device runtime used to run the models and log latency traces is our fork of MLC-LLM with LM-Meter instrumentation, available at LM-Meter. This repository covers everything after the phone: the prompts we send, the power we record, and the analysis that turns raw measurements into the tables and figures in the paper.

Repository structure

docs/                      Static paper project page for GitHub Pages

datasets/                  Prompt datasets used in the experiments
  cognitive_load/          Our constructed dataset: SVAMP, BoolQ, and AI2-ARC base prompts,
                           each with intrinsic, extraneous, and germane load variants
  phrasing_pattern/        Phrasing variants from the CLEF 2025 ELOQUENT robustness dataset,
                           covering seven sub-properties (aggressive, conversational, CoT,
                           format, persona, polite, technical)

power_related/             Host-side measurement tools
  power_collection.py      Samples battery current and voltage over ADB while the phone
                           runs inference, and saves timestamped power readings
  cpu_gpu_freq_monitor.sh  Optional live monitor for CPU and GPU frequencies

data_analysis/             Analysis notebooks, staged to mirror the paper pipeline
  stage1_power_latency_data_align.ipynb        Aligns power traces with inference latency
                                               logs into per-request energy records
  stage2_evaluate_single_run_*.ipynb           Scores response quality for one run:
                                               ai2_arc, boolq, svamp (accuracy against
                                               ground truth), phrasing_pattern (six GEval
                                               rubrics, no gold answers)
  stage3_merge_3_single_runs.ipynb             Merges the three repeated runs per model
  stage3_*_data_analysis.ipynb                 Produces the RQ1, RQ2, and RQ3 figures
  stage3_*_table_organize.ipynb                Renders the LaTeX summary tables
  ground_truth_info/                           Ground truth tables for the accuracy tasks
  processed_data/                              Measurement data at every pipeline stage

Data flow

Each experiment runs every prompt three times on the phone. The data then moves through four folders under data_analysis/processed_data/:

1. single_run/ per-request records aligning power, latency, and energy for one run, produced by stage 1
2. single_run_eval/ stage 2 output samples with response quality columns added
3. single_run_processed/ per-run files organized by dataset and model, ready for merging
4. merge_data/ one file per model and dataset, averaging the three runs; stage 3 reads only from here

The raw power and latency traces are large and are not included. The aligned per-request records in processed_data/ are sufficient to reproduce every evaluation, figure, and table.

Running the analysis

Start Jupyter inside data_analysis/ so the relative paths resolve. The stage 3 notebooks run offline on the provided data. The stage 2 notebooks for AI2-ARC, BoolQ, and phrasing patterns use Gemini-2.5-Pro wrapped in the DeepEval model interface, so set GENAI_API_KEY before running them; the SVAMP scorer is purely local.

The on-device inference runs use fixed sampling parameters across all experiments: temperature=0.6 and top_p=0.8. The Gemini judge uses a separate low-temperature setting (temperature=0.1) for stable scoring.

Main dependencies: pandas, numpy, matplotlib, seaborn, deepeval, google-generativeai.

Power collection

power_collection.py reads current_now and voltage_now from the battery driver through ADB at a 10 ms interval and computes instantaneous power. Start it on the host right before launching inference on the phone, and stop it with Ctrl+C to write the JSON trace. Phase-level energy is later obtained in stage 1 by integrating these samples over the prefill and decode windows recorded by the runtime.

Citation

@inproceedings{hu2026prompt,
  title     = {How Do Prompt Variations Affect Energy Consumption in On-Device LLMs?},
  author    = {Hu, Wei and Tu, Xiaolong and Chen, Dawei and Chen, Yitao and Han, Kyungtae and Wang, Haoxin},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026}
}