Task-based evaluation for LLM usage of Hanko SDK

[gyxlucy]

Hi — thanks for building Hanko, really interesting auth framework!

We’ve been exploring how well LLMs actually use SDKs like Hanko in practice — not just from docs, but through structured, task-based evaluation.

We’ve started generating a set of representative tasks that simulate how developers (or agents) would realistically use Hanko's SDK.

Here’s an example of what those tasks look like:
👉 https://gist.github.com/gyxlucy/4c788013516f7eaa59693277b9074ebf

The goal is to capture things like:

• multi-step workflows (setup → execution → handling results)
• correct API usage under realistic constraints
• whether models fall back to outdated or incomplete patterns

Even with this early setup, we’re already seeing some interesting differences across models — especially around multi-step reliability.

Not trying to promote anything — just wanted to sanity-check whether this kind of task-level visibility would be useful from your perspective.

Happy to share more or tailor a few tasks specifically for this repo if helpful.

Curious if this resonates at all?