Repository navigation
Replies: 2 comments 3 replies
|
I've found running f16 or bf16 models rather than q4 helps too, but that may only be for some models. |
2 replies
|
You are right but, on a 16 GB card, it's either Q4 + high context, or f16 weights + smaller context. |
|
256k isn't bad! |
|
Heres my setup: CPU: AMD Ryzen 9 5900X Build command: compose.yaml (GPU+CPU): services:
llama-gemma:
container_name: llama-gemma
image: llama-cpp-vulkan:latest
ports:
- "8080:8080"
volumes:
- ./hf-cache:/root/.cache/huggingface
devices:
- /dev/dri
command:
- --server
- --host
- "0.0.0.0"
- --port
- "8080"
- --hf-repo
- "unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL"
#- --n-gpu-layers
#- "all"
- --spec-type
- "draft-mtp"
- --spec-draft-n-max
- "1"
- --no-mmproj-offload
- --parallel
- "1"
- --cache-type-k
- "q8_0"
- --cache-type-v
- "q8_0"
- --ctx-size
- "150016"
restart: unless-stoppedPrompt to test:
Output: {
"choices": [
{
"finish_reason": "length",
"index": 0,
"message": {
"role": "assistant",
"content": "The mountain did not breathe, but Ignis did.\n\nTo the villagers of Oakhaven, nestled in the valley far below, the mountain was a silent, jagged tooth of granite. But to those who knew the old songs, the mountain was a sleeper. And Ignis was the breath within the stone.\n\nHe was not the dragon of the nursery rhymes. He did not crave the clinking weight of gold coins, nor did he find joy in the charred remains of knights. Gold, Ignis found, was terribly boring. It was cold, it was hard, and it had no soul. It could not tell you how the wind felt on a summer evening, nor could it sing of the heartbreak of a dying star.\n\nIgnis hoarded something far more precious: Echoes.\n\nHis cavern was not a treasury of metal, but a library of light and sound. In the dim, amber glow of his own internal furnace, thousands of glass spheres rested upon shelves carved directly from the bedrock. Within each sphere lived a moment. A first kiss caught in a swirl of pink mist; the roar of a long-extinct sea beast; the precise, heartbreaking melody of a lullaby sung by a mother a thousand years dead.\n\nHe sat in the center of this silent symphony, his scales the color of cooling embers, his eyes two molten pools of ancient intelligence. He was waiting.\n\nThe sound came at twilight—the rhythmic *clack-clack* of iron on stone.\n\nIgnis did not move, but his nostrils flared. A scent drifted into the cave: sweat, wet wool, and the sharp, metallic tang of fear. A human.\n\nHe watched through slit eyes as a small figure emerged from the shadows of the tunnel. It was not a knight in shining armor, but a girl. She was small, wrapped in a cloak far too large for her, carrying a lantern that trembled in her hand.\n\n\"I know you are here,\" she whispered. Her voice was thin, cracking like dry parchment.\n\nIgnis let out a breath, a low rumble that vibrated through the floorboards of the world. \"Many know I am here,\" he replied. His voice was not a roar, but a deep, resonant hum that seemed to come from the earth itself. \"Few have the courage to seek me.\"\n\nThe girl stumbled back, her lantern swinging wildly. \"I am not seeking gold. I have none to give.\"\n\n\"I have no use for gold,\" Ignis said, shifting his massive bulk. His scales scraped against the stone like tectonic plates grinding together. \"Why have you breached my silence, little spark?\"\n\nThe girl swallowed hard. She stood her ground, though her knees visibly shook. \"My village... the Great Frost is coming. Not a normal winter, but a silence. The crops won't grow, the hearths won't stay lit. My people are forgetting how to be warm. They are forgetting how to hope.\"\n\nShe took a step forward, her eyes glistening in the amber light. \"I heard the legends. That you keep the things the world has lost. I didn't come to steal. I came to ask... if you could lend us a memory. Just one. Something of warmth. Something of summer.\"\n\nIgnis leaned his massive head down, his snout inches from the girl. He could feel the frantic drumming of her heart. It was a fast, rhythmic little thing, full of life and terrifying fragility.\n\n\"A memory is not a trinket, child,\" Ignis warned, his eyes narrowing. \"To give a memory is to lose a piece of the world’s tapestry. If I give you the warmth of a summer afternoon, that afternoon may vanish from the minds of all men. It becomes a ghost.\"\n\n\"Then let it be a ghost that keeps us alive,\" she pleaded.\n\nIgnis studied her. He saw the desperation, but he also saw something else—a fierce, burning spark of will. It was a beautiful thing. It was a story waiting to be told.\n\nHe turned his gaze to his shelves. He bypassed the spheres of triumph, the spheres of conquest, and the spheres of sorrow. He looked for something specific.\n\nWith a movement as delicate as a breeze, he extended a single, taloned claw. He touched a sphere that glowed with a soft, honeyed light. Inside, a sun-drenched meadow danced, the air thick with the scent of clover and the hum of bees.\n\n\"This,\" Ignis murmured, \"is the feeling of a sunbeam on a sleeping cat. It is the warmth of a shared meal after a long journey. It is the golden hour that lingers when the world is at peace.\"\n\nHe brought the sphere toward the girl. \"But the price is not gold. To take this, you must leave something in its place. A memory of your own. Something of equal weight.\"\n\nThe girl went still. \"I... I have nothing but my journey.\"\n\n\"The journey is the memory,\" Ignis said. \"The fear, the cold, the climb, and the moment you stood before a dragon and asked for mercy. Give me that. Give me the memory of this night.\"\n\nThe girl closed her eyes. She thought of the climb, the biting wind, and the terrifying shadow of the mountain. She felt the weight of it—the terror and the hope entwined.\n\nIgnis breathed a soft, rhythmic heat toward her. The girl felt a strange tugging in her chest, a sensation of something being gently unspooled. Her eyes flew open, and for a moment, she looked confused, as if she had forgotten why she was standing in a cave.\n\nThen, she saw the sphere in the dragon's claw. It was no longer just",
"reasoning_content": "A dragon.\nStory.\nOpen (can be epic, cozy, tragic, action-oriented, etc.).\n\n * *Option A: The Hoarder.* Classic, gold-obsessed, fighting knights. A bit cliché.\n * *Option B: The Guardian.* Protects a forest or a village. Good, but needs a twist.\n * *Option C: The Outcast/Lonely Dragon.* A dragon who doesn't fit in or is misunderstood. Emotional depth.\n * *Option D: The Last of its Kind.* Melancholic, epic scope.\n\n *Decision:* Let's go with a mix of **Option C and D**. A dragon that isn't interested in gold, but in something unexpected—like memories or stars. Let's call him Ignis... no, something more unique. *Aethelgard*. Or maybe just *Ember*. Let's go with **Aurum**, but he doesn't hoard gold; he hoards *stories*.\n\n * *Setting:* A high, lonely peak, perhaps above the clouds. A cave filled not with coins, but with objects that hold meaning.\n * *Conflict:* A young thief/scholar comes to steal \"treasure,\" but finds something else. Or, the dragon is dying and needs one last story.\n * *Plot:*\n 1. Introduce Aurum and his unique hoard.\n 2. A human intruder arrives (Elara, a chronicler).\n 3. Initial tension (threat of fire).\n 4. The realization: He doesn't want her life; he wants her tale.\n 5. The exchange: A story for safety/passage.\n 6. Ending: A sense of connection or a bittersweet departure.\n\n * *Introduction:* Describe the dragon. Not just a lizard, but an ancient force. Describe the cave. It shouldn't smell like sulfur; it should smell like old parchment and rain.\n * *The Hoard:* Instead of gold, there are trinkets. A rusted locket, a broken flute, a dried flower. This establishes character immediately.\n\n * *The Encounter:* Elara enters. She's not a warrior; she's a scholar. She's terrified. The dragon's presence should be overwhelming—the heat, the scale.\n\n * *Dialogue:* The dragon shouldn't speak like a monster. He should speak like an old professor or a weary king. Wise, slow, rhythmic.\n * *Drafting thought:* \"Why are you here, little thief?\" vs \"What heavy truth do you carry in your pockets, child?\" The latter is better.\n\n * *The Climax:* The confrontation. Elara realizes he isn't there to eat her. He asks for the \"weight\" she carries. She tells a story—maybe about her village or her journey.\n\n * *The Resolution:* The dragon is satisfied. Elara leaves, changed. The dragon remains, waiting for the next story.\n\n * *Sensory details:* The \"low hum of a furnace\" for his breath. The \"scales like obsidian glass.\" The \"clatter of a dropped lantern.\"\n * *Word Choice:* Use evocative verbs (shimmered, resonated, exhaled, etched).\n\n * *Check:* Is the dragon too friendly?\n * *Adjustment:* Keep him intimidating. Even if he's \"nice,\" he's still a predator. The tension should remain.\n * *Check:* Is the ending satisfying?\n * *Adjustment:* Make sure the \"exchange\" feels earned. She gives a memory; he gives her wisdom/passage."
}
}
],
"created": 1786429985,
"model": "unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL",
"system_fingerprint": "b10362-4801e3c56",
"object": "chat.completion",
"usage": {
"completion_tokens": 2000,
"prompt_tokens": 23,
"total_tokens": 2023,
"prompt_tokens_details": {
"cached_tokens": 0
}
},
"id": "chatcmpl-ouDA2jMOl1V6j1o17ko5eCFJ706ketYU",
"timings": {
"cache_n": 0,
"prompt_n": 23,
"prompt_ms": 467.649,
"prompt_per_token_ms": 20.332565217391306,
"prompt_per_second": 49.18218578463762,
"predicted_n": 2000,
"predicted_ms": 34225.054,
"predicted_per_token_ms": 17.112526999999996,
"predicted_per_second": 58.436722992460446,
"draft_n": 1197,
"draft_n_accepted": 802
}
}58 t/s compose.yaml (only GPU): services:
llama-gemma:
container_name: llama-gemma
image: llama-cpp-vulkan:latest
ports:
- "8080:8080"
volumes:
- ./hf-cache:/root/.cache/huggingface
devices:
- /dev/dri
command:
- --server
- --host
- "0.0.0.0"
- --port
- "8080"
- --hf-repo
- "unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL"
- --n-gpu-layers
- "all"
- --spec-type
- "draft-mtp"
- --spec-draft-n-max
- "1"
- --no-mmproj-offload
- --parallel
- "1"
- --cache-type-k
- "q8_0"
- --cache-type-v
- "q8_0"
- --ctx-size
- "150016"
restart: unless-stoppedOutput: {
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "The mountain did not breathe, but Ignis did.\n\nTo the villagers in the valley below, the mountain was a jagged tooth of basalt and ice, a place of superstition and shadow. They spoke of the \"Great Ember\" in whispers, a beast of legend that had once leveled cities and turned rivers to steam. But to Ignis, the mountain was not a home; it was a vessel.\n\nIgnis was not a creature of gold and greed. He found the clinking of coins tiresome and the glitter of gemstones distracting. No, Ignis was a collector of something far more volatile: memories.\n\nHis cavern was not filled with mounds of treasure, but with scrolls that crumbled at a touch, jars of captured starlight, and the echoes of songs long forgotten by the world of men. He sat amidst this silent library, his scales the color of cooling lava—deep crimson shot through with veins of obsidian.\n\nFor three hundred years, Ignis had been content with the silence. But silence, he was learning, was a slow death.\n\nThe sound came on a Tuesday—a rhythmic *tink, tink, tink* against the stone entrance of his sanctum. It was a small sound, fragile as a moth’s wing, but in the vacuum of his solitude, it rang like a hammer on an anvil.\n\nIgnis uncoiled. His movement was a tectonic event, a slow, grinding slide of muscle and scale that sent dust cascading from the ceiling. He didn't fly; he flowed. He moved through the darkness, his eyes glowing like twin hearths, until he reached the threshold.\n\nThere, standing in the biting wind of the high altitude, was not a knight in shining plate, nor a king demanding tribute. It was a girl.\n\nShe was perhaps twelve, wrapped in furs that were too large for her, clutching a heavy, leather-bound book to her chest. She was shivering so violently that her teeth rattled, but her eyes—wide, dark, and fiercely intelligent—were fixed on him.\n\nIgnis let a plume of smoke drift from his nostrils, a warning. **\"You are very small to be so brave,\"** he rumbled. The sound was not a voice so much as a vibration in the floorboards of the world.\n\nThe girl gasped, stumbling back, but she did not run. She gripped her book tighter. \"I'm not brave,\" she whispered, her voice cracking. \"I'm just... I'm the only one left who remembers.\"\n\nIgnis lowered his massive head, his snout stopping inches from her. He could smell the salt of her tears, the iron of her fear, and the strange, sweet scent of old ink. **\"Remembers what, little spark?\"**\n\n\"The Song of the Sun-Eater,\" she said, her voice gaining a sudden, desperate strength. \"My grandmother told me. She said the dragons didn't go away because they were defeated. They went away because the world stopped listening. And if no one listens, the stories die. And if the stories die... the magic goes out.\"\n\nIgnis felt a strange, sharp ache in his chest—a sensation he hadn't felt since the age of heroes. He looked at his cavern, at the piles of rotting parchment and the dim, fading light of his collected wonders. She was right. The world had become loud, fast, and hollow. They traded legends for ledgers and myths for machines.\n\n**\"The Song of the Sun-Eater is a dangerous thing to hold,\"** Ignis cautioned, his golden eyes narrowing. **\"It carries the heat of a thousand summers. It could burn your mind.\"**\n\n\"Then let it burn,\" the girl said. She opened her book. The pages were blank, save for the first line, written in a hand that looked ancient. \"I want to write it down. Properly. So it doesn't vanish.\"\n\nIgnis looked at the girl, and then he looked at the vast, empty expanse of his library. For centuries, he had been a tomb, a place where things went to be forgotten. He had been a hoarder of the past, waiting for an end that had already arrived.\n\nSlowly, the Great Ember shifted. He did not stand to fight, nor did he roar to intimidate. Instead, he exhaled. But it wasn't a blast of flame. It was a soft, glowing mist—a breath of pure, concentrated memory. The mist swirled around the girl, illuminating the dark corners of the cave, turning the dust into dancing motes of gold.\n\n**\"Sit, little scribe,\"** Ignis commanded, his voice softening into a low hum. **\"The mountain is cold, and the night is long. If you wish to save the world, you had best start with the beginning.\"**\n\nThe girl sat. She opened her book to a fresh page.\n\nAnd there, in the heart of the silent mountain, the dragon began to speak. He told her of the time the stars were young, of the dragons who danced in the aurora, and of the fire that once lived in the heart of every living thing. \n\nAs she wrote, the ink on her pages didn't just sit there; it glowed. And as the stories poured out of the dragon and into the girl, the mountain began to breathe once more.",
"reasoning_content": "A dragon.\nOpen (no specific genre, tone, or length specified).\nA compelling narrative, character development (even for a dragon), and a sense of atmosphere.\n\n * *Option A: The Classic Hoarder.* A dragon sitting on gold, fighting knights. (A bit cliché).\n * *Option B: The Reluctant Protector.* A dragon guarding a village or a child. (A bit more emotional).\n * *Option C: The Scholar Dragon.* A dragon interested in history/knowledge rather than gold. (Unique).\n * *Option D: The Aging Dragon.* A dragon facing its mortality or the changing world. (Poignant).\n\n *Decision:* Let's go with a mix of **Option C and D**. An ancient, scholarly dragon named Ignis (or something more unique like *Aurelius*) who lives in a library/cave of memories and is facing the \"fading\" of magic.\n\n * *Character:* Ignis the Ember-Eyed. Not just a beast, but a keeper of stories.\n * *Setting:* The Obsidian Spire, a mountain peak filled with scrolls, artifacts, and the scent of old parchment and sulfur.\n * *Conflict:* The world below is forgetting magic. The \"Silence\" is coming.\n * *Inciting Incident:* A small human enters—not a knight, but a lost scholar or child. Let's go with a young girl, a \"Seeker.\"\n\n * **Introduction:** Describe Ignis. Not the gold, but the *knowledge*. The scale of his wings, the heat of his breath, the way he reads.\n * **The Arrival:** A sound in the silence. A small, fragile human.\n * **The Encounter:** Ignis is wary but curious. The girl, Elara, isn't there to kill him; she's looking for something specific.\n * **The Interaction:** They talk. She wants a story. He realizes that as people stop believing/seeking, his world shrinks.\n * **The Climax/Turning Point:** He realizes his purpose isn't just to *keep* the stories, but to *pass them on*.\n * **Ending:** A sense of legacy. The dragon's fire isn't just destructive; it's the light of memory.\n\n * *Sensory Details:* The smell of ozone and old paper. The sound of scales scraping stone. The visual of glowing embers in a dark cavern.\n * *Drafting - Scene 1 (The Setting):* Start with the silence. Ignis is old. His scales are duller than they used to be. He's reading a scroll.\n * *Drafting - Scene 2 (The Intrusion):* A clatter. A girl. She's small, wearing a scholar's robe. She’s shivering.\n * *Drafting - Scene 3 (The Dialogue):* He speaks. His voice should sound like \"grinding stones.\" She isn't afraid; she's awestruck. She asks for \"The Song of the First Star.\"\n * *Drafting - Scene 4 (The Realization):* He sees her notebook. She's writing things down. This is how magic survives.\n\n * *Check:* Is the dragon too human? *Adjustment:* Keep some beastly traits—the smoke from his nostrils, the predatory instinct, the sheer size.\n * *Check:* Is the ending too cheesy? *Adjustment:* Make it bittersweet. He is still old and fading, but there is hope."
}
}
],
"created": 1786429697,
"model": "unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL",
"system_fingerprint": "b10320-23634783c",
"object": "chat.completion",
"usage": {
"completion_tokens": 1903,
"prompt_tokens": 23,
"total_tokens": 1926,
"prompt_tokens_details": {
"cached_tokens": 0
}
},
"id": "chatcmpl-4h4pwTllo5te01yHjwhnIfPmPlLPLoTA",
"timings": {
"cache_n": 0,
"prompt_n": 23,
"prompt_ms": 1333.389,
"prompt_per_token_ms": 57.97343478260869,
"prompt_per_second": 17.249279842566573,
"predicted_n": 1903,
"predicted_ms": 41580.454,
"predicted_per_token_ms": 21.849949553336835,
"predicted_per_second": 45.766696053871854,
"draft_n": 1123,
"draft_n_accepted": 780
}
}45 t/s |
1 reply
|
Hi, I just ran your exact test configuration on my own machine so you have a direct comparison point. {
"executable": "./llama-server",
"options": {
"-m": "/Qwen3.5-9B-The-Defiant-Fable-Uncnr-Heretic-NEO-MAX-MTP-Q4/model.gguf",
"--mmproj": "/Qwen3.5-9B-The-Defiant-Fable-Uncnr-Heretic-NEO-MAX-MTP-Q4/mmproj.gguf",
"-t": 6,
"-ngl": "all",
"--parallel": 1,
"--ctx-size": 150016,
"--cache-type-k": "q8_0",
"--cache-type-v": "q8_0",
"--spec-type": "draft-mtp",
"--spec-draft-n-max": 1,
"--no-mmproj-offload": true,
"--flash-attn": "on",
"--host": "127.0.0.1",
"--port": 8000
}
}{
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "Ignis did not hoard gold. Gold was too cold, too hard, and it didn't speak. Ignis hoarded the Echoes.\n\nEvery time a traveler died in the Whispering Peaks, their final, unspoken thought would dissolve into the wind. Ignis collected them. He tucked them into the hollows of his ribs and behind his ear scales. He was old enough that he remembered the days when dragons burned villages for sport, and young enough to understand that the smoke had finally cleared.\n\nThe valley below, known as the Hollow, had changed. The iron bridges had replaced the hanging willow roots. The blacksmiths were no longer forging armor for war, but plows for the rich harvest. And the dragons? The great red ones, the blue ones with scales like polished lapis, had retreated to the high cliffs. They were guardians, or so the old songs said. But in the Hollow, they were merely weather. The wind that rustled the crops was sometimes a dragon exhaling, and the thunder was their wings.\n\nIgnis was the oldest of the elders. His scales, once the color of burning orange, were now a faded, bruised crimson, patched with white scars from a hunting falcon he’d spared centuries ago. He spent his days perched on the highest spire of the Obsidian Ridge, watching the smoke rise from the chimneys.\n\nA knock on his cave entrance startled him. He didn't open his eyes; he knew the tread. It was soft, deliberate, and hesitant.\n\nThe door creaked open, revealing Elara. She was a cartographer, though her maps had long since stopped being used by the guild. She carried a leather satchel and a sketchbook, her eyes sharp and tired.\n\n\"Ignis,\" she said, her voice soft. \"You're quiet.\"\n\n\"I am listening,\" he rumbled. The sound vibrated through the stone floor. \"Is the wind bringing news of the south?\"\n\nElara stepped inside, leaving the door slightly ajar. She placed her satchel on a pile of ancient scrolls. \"The wind is just wind, Ignis. But I have news from the north. The first frost of the year is early this year. The harvest will be thin.\"\n\nIgnis's tail flicked, disturbing a pile of dried moss. \"And this is my problem?\"\n\n\"It is your problem because you are the only one who knows the old way of reading the sky,\" Elara said, turning to face him. \"The elders said the clouds would hold. But the cold is coming, and the people are worried. They whisper that the dragons are forgetting how to help.\"\n\nIgnis sighed, a gust that rattled the torches. \"I help by not eating them,\" he muttered.\n\nElara smiled sadly. \"That isn't enough anymore. They don't want a dragon that just sleeps in a cave. They need a dragon that understands change.\"\n\n\"I understand enough,\" Ignis said. \"I understand that gold doesn't buy food, and that hoarding Echoes doesn't stop the frost.\"\n\n\"Then what do you understand?\" she asked.\n\nIgnis closed his eyes. He reached into the hollow of his left rib, where the air felt different, heavier. He plucked out an Echo. It was small, warm, and smelled of ozone and fear.\n\n*He felt the mountain slide beneath him. The stone tasted of iron and despair.*\n\nIt was the last thought of a climber who had fallen years ago.\n\n\"It is heavy,\" Ignis said.\n\n\"It is the truth,\" Elara replied.\n\nIgnis sat up, his spine cracking audibly. \"Elara, do you know why we built the ridge? Why the peaks are so jagged?\"\n\nShe shook her head. \"To keep the wind away. To shelter the valley.\"\n\n\"No,\" Ignis said. \"We built the ridge to make a funnel. To gather the storm clouds and let them pass through, carrying the cold with them. It is an old trick. But it is a trick that still works.\"\n\nHe stood, his joints grinding like boulders in a mill. \"The frost isn't coming because the sky is angry. It is coming because the great thermal currents are shifting. The dragon-scales of the valley have been broken, and we are the only ones left who remember how to mend them.\"\n\nHe turned to her, his eyes glowing with a faint, dying ember. \"I cannot fly to the north. My wings are too stiff. But I can go to the river. There is a thermal spring beneath the waterfall. If I can warm the water with my breath, it will steam up, carrying the heat to the peaks. It will delay the frost by three days.\"\n\n\"Three days,\" Elara whispered. \"Is that enough?\"\n\n\"Ignis is never wrong,\" she said, her voice firm. \"But the valley is too small. The people will hear.\"\n\nIgnis looked at her. He knew what she was asking. She was asking if he was willing to reveal himself. To come down from the ridge. To let the children look up and see the great red dragon, not as a ghost story, but as a living thing.\n\n\"Ignis,\" she said softly. \"You are fading. The Echoes in your chest are growing heavy. If you do not find a way to share them, to pass them on, they will just weigh you down until you cannot even breathe.\"\n\nIgnis felt the truth of her words. The hoard of Echoes was becoming a burden, a stone in his gut. He had kept them too long.\n\nHe nodded slowly. \"I will come down.\"\n\nElara's eyes widened, then softened. She reached out and placed a hand on his snout, feeling the rough, cracked scales. \"Thank you.\"\n\n\"Don't thank me,\" Ignis grumbled, though there was no bite in it. \"Just tell them not to light bonfires in the river valley. I need the steam.\"\n\nShe bowed low. \"As you wish, Old One.\"\n\nShe left, and the door clicked shut behind her.\n\nIgnis remained still for a long time. He felt the weight of the Echoes, the memories of fear, of wonder, of pain. He had carried them alone for so long, believing that to forget them was to lose his history. But the wind outside was stirring. The valley below was waiting.\n\nHe stood up, stretched his wings, and felt the familiar rush of air against his feathers. It wasn't the soaring flight of youth, but it was enough.\n\nHe would fly. Not to hunt, not to guard, but to warm the river. To share the heat of his breath, to let the valley know that the old dragon was not just a memory, but still breathing.\n\nAs he took flight, Ignis let one single, small Echo slip from his rib cage and catch in the wind. It was a memory of the first time he'd seen a star, and how bright it felt. He wasn't hoarding it anymore. He was letting it go.\n\nAnd for the first time in centuries, the dragon felt light.",
"reasoning_content": "The user wants a story about a dragon.\nI should aim for something engaging with depth, not just a generic fantasy trope.\nThemes of aging, purpose, and changing perspectives could add weight to the narrative.\nI will create a dragon named Ignis whose hoard isn't gold, but memories or stories, perhaps reflecting on a changing world where dragons are no longer feared.\nThe setting will be a valley where humanity and dragons have reached a fragile truce.\nThe plot will involve Ignis confronting his own obsolescence and finding a new role.\nI will focus on sensory details: the texture of scales, the smell of rain, the weight of the past.\nThe ending should be bittersweet but hopeful.\n"
}
}
],
"created": 1786437303,
"model": "/Qwen3.5-9B-The-Defiant-Fable-Uncnr-Heretic-NEO-MAX-MTP-Q4/model.gguf",
"system_fingerprint": "b10355-dd1ea5243",
"object": "chat.completion",
"usage": {
"completion_tokens": 1654,
"prompt_tokens": 17,
"total_tokens": 1671,
"prompt_tokens_details": {
"cached_tokens": 0
}
},
"id": "chatcmpl-JxvAAPmgLMt616RIpDz0vpe3nWISMGA8",
"timings": {
"cache_n": 0,
"prompt_n": 17,
"prompt_ms": 93.542,
"prompt_per_token_ms": 5.502470588235294,
"prompt_per_second": 181.73654615039234,
"predicted_n": 1654,
"predicted_ms": 20413.518,
"predicted_per_token_ms": 12.341909310761789,
"predicted_per_second": 81.02474056652068,
"draft_n": 1035,
"draft_n_accepted": 618
}
} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hey everyone,
I wanted to share my experience with llama.cpp on AMD hardware – After a few days of tuning, I've found a setup that consistently delivers ~86 tokens/s on my RX 7800 XT with a 9B model – and that's with 102k context and ~13.5 GB VRAM usage. I'm using "Jan" and "Odysseus AI" as Frontend.
🖥️ My Hardware & Software
radvVulkan driver (Mesa)📊 Benchmark Results
My Settings
💡 Key Learnings – Why This Setup is Fast
f16cache was faster thanq8_0on my AMD card.around ~10 tok/s faster than
q8_0with MTP--spec-draft-n-max 2is the sweet spot.1→ too little speculation, GPU idle.2→ perfect balance.3→ too much overhead, actually slower.--parallel 1gives full context (102k) and avoids performance drops.With
--parallel 2, each slot only gets 51k tokens, and concurrent requests cause major speed drops (down to ~15–50 tok/s).Manual server start is crucial.
I use Jan as a client but start the server manually. This bypasses the Flatpak sandbox, which can throttle performance.
🔥 Stability Test – Deep Research & Real-World Mix
I ran two separate stress tests to validate the setup:
1️⃣ Deep Research Session (Odysseus)
A single, continuous Deep Research conversation with ~20,000 tokens of context, where the model had to process and reason over a large amount of information.
2️⃣ 10 Mixed Questions in One Chat (Jan)
I asked 10 completely different questions (cooking, coding, finance, creative writing, technical, email drafting, etc.) in the same Jan chat, letting the context grow naturally. This simulates a real-world assistant workflow.
🧪 Bonus:
llama-benchOutput (for reference)Result:
Note: The
tg128value here is without MTP.Hope this helps someone! If you've got similar hardware or found other tweaks, let me know – I'm always curious to learn more.
All reactions