parakh-action

GitHub Action wrapper around parakh — the self-hosted, field-level OCR / extraction evaluation framework.

One YAML block. PR fails if accuracy regresses past your threshold.

---

Quick start

Put this in .github/workflows/eval.yml:

name: eval

on:
  pull_request:
    paths:
      - "models/**"
      - "extraction/**"
      - ".github/workflows/eval.yml"

jobs:
  parakh:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: sarcascoder/parakh-action@v1
        with:
          config: eval/config.yaml
          model: your-vlm
          threshold: 0.92
          reading-order-threshold: 0.88

That's it. On every PR that touches your extraction code, parakh runs your eval suite, fails the check if field-level or reading-order accuracy regresses, and uploads the full HTML report as a workflow artifact.

---

Inputs

Name	Required	Default	Notes
config	yes	eval/config.yaml	Path to your parakh config
model	no	default	Adapter name (built-in or defined in your config)
threshold	no	0.90	Min overall field-level accuracy to pass (0.0–1.0)
reading-order-threshold	no	0.85	Min reading-order accuracy. Set to 0 to disable
golden	no	—	Override golden-dataset path from config
python-version	no	3.11	Python version for the runner
upload-report	no	true	Upload HTML report as a workflow artifact
fail-on-regression	no	true	Set to false for advisory-only (no PR failure)

Outputs

Name	Notes
accuracy	Overall field-level accuracy (0.0–1.0)
reading-order	Reading-order accuracy (0.0–1.0)
failed-fields	Count of fields below per-field threshold
report-path	Path to the HTML report on the runner

Examples

Comment the result back on the PR

- id: eval
  uses: sarcascoder/parakh-action@v1
  with:
    config: eval/config.yaml

- uses: actions/github-script@v7
  if: github.event_name == 'pull_request'
  with:
    script: |
      const acc  = parseFloat("${{ steps.eval.outputs.accuracy }}");
      const ro   = parseFloat("${{ steps.eval.outputs.reading-order }}");
      const fail = parseInt("${{ steps.eval.outputs.failed-fields }}");
      const body = [
        "### 📋 parakh extraction eval",
        `- field-level accuracy: **${(acc * 100).toFixed(1)}%**`,
        `- reading-order accuracy: **${(ro * 100).toFixed(1)}%**`,
        `- failed fields: **${fail}**`,
        ``,
        `Full HTML report attached as a workflow artifact.`,
      ].join("\n");
      github.rest.issues.createComment({
        issue_number: context.issue.number,
        owner: context.repo.owner,
        repo: context.repo.repo,
        body,
      });

Run only on changed config / golden files

on:
  pull_request:
    paths:
      - "eval/**"
      - "extraction/**"

Advisory mode (no PR failure)

- uses: sarcascoder/parakh-action@v1
  with:
    config: eval/config.yaml
    fail-on-regression: "false"

---

What this is not

• Not a labelling tool. Bring a golden dataset — Label Studio is great for creating one.
• Not an LLM evaluator. This is for OCR / extraction / VLM. Use LangSmith / Braintrust / HumanLoop for chat eval.
• Not a runtime monitor. This runs in CI on PRs. For prod drift monitoring use parakh Cloud.

License

Apache-2.0 — matches the upstream parakh project.

Who's behind this

Anupam Deep Tripathi · Founding AI Engineer at Hashteelab · IIT Tirupati '25.

Part of the OpenExtract family: OpenExtract (drop-in Textract replacement) · parakh (extraction evals) · taul (reading-order eval) · TurboQuant (KV-cache quantization).