[dataflow-dataset] DataFlow Dataset Build Report (dataflow) — 2026-09-25

[github-actions[bot]]

Summary

Built a cleaned, deduplicated dataset from GitHub discussions and PRs.

Metric	Value
Execution mode	dataflow
Input records	600
Output (clean) records	597
Retention rate	99.5%
Average text length	3,104 chars
Operators used	CharNumberFilter, HashDeduplicateFilter, python_alpha_ratio_filter

All filtering and deduplication stages used the current DataFlow operators.

Records by Source

Source	Count
discussion	97
pull_request	500

Pipeline

The pipeline applied these processing stages:

1. Normalise — Merged discussions and PRs into unified JSONL (title + body → text)
2. Text length filter — Kept records with 50–100,000 characters (DataFlow CharNumberFilter)
3. Alpha-ratio filter — Removed records with fewer than 25% alphabetic characters using a deterministic Python stage
4. Near-duplicate removal — Eliminated exact-duplicate records using DataFlow HashDeduplicateFilter

📦 Dataset Artifact

upload_artifact is not enabled for this run's safe outputs; the cleaned dataset (dataset_clean.jsonl, 597 records, ~2.1MB) was generated at /tmp/gh-aw/agent/dataflow/output/dataset_clean.jsonl but could not be attached to this run.

Pipeline Configuration

Execution mode: dataflow
DataFlow package: open-dataflow==1.0.10
Source: https://github.com/OpenDCAI/DataFlow
Input: GitHub Discussions (100) + merged PRs (500) from github/gh-aw
Output: JSONL — one record per item, text field for LLM use

Raw pipeline statistics

{
  "input_count": 600,
  "after_length_filter": 597,
  "after_alpha_filter": 597,
  "after_dedup": 597,
  "operators_used": [
    "CharNumberFilter",
    "HashDeduplicateFilter",
    "python_alpha_ratio_filter"
  ],
  "execution_mode": "dataflow",
  "fallback_mode": false,
  "dataflow_ready": true,
  "dataflow_version": "1.0.10",
  "warnings": []
}

---

Generated by DataFlow PR & Discussion Dataset Builder — Run #36087770067 on 2026-09-25

🌊 Dataset built by DataFlow PR & Discussion Dataset Builder · copilot · auto · 38.8 AIC · ⌖ 10.3 AIC · ⊞ 15.4K · ◷

• expires on Oct 1, 2026, 6:55 PM UTC-08:00