[dataflow-dataset] DataFlow Dataset Build Report (dataflow) — 2026-09-11

[github-actions[bot]]

Summary

Built a cleaned, deduplicated dataset from GitHub discussions and PRs.

Metric	Value
Execution mode	dataflow
Input records	600
Output (clean) records	598
Retention rate	99.7%
Average text length	2,967 chars
Operators used	CharNumberFilter, HashDeduplicateFilter, python_alpha_ratio_filter

All filtering and deduplication stages used the current DataFlow operators.

Records by Source

Source	Count
discussion	98
pull_request	500

Pipeline

The pipeline applied these processing stages:

1. Normalise — Merged discussions and PRs into unified JSONL (title + body → text)
2. Text length filter — Kept records with 50–100,000 characters (DataFlow CharNumberFilter)
3. Alpha-ratio filter — Removed records with fewer than 25% alphabetic characters using a deterministic Python stage
4. Near-duplicate removal — Eliminated near-identical records using DataFlow HashDeduplicateFilter

📦 Dataset Artifact

dataset_clean.jsonl (598 records) uploaded as a workflow run artifact — see run artifacts.

Pipeline Configuration

Execution mode: dataflow
DataFlow package: open-dataflow==1.0.10
Source: https://github.com/OpenDCAI/DataFlow
Input: GitHub Discussions + merged PRs from github/gh-aw
Output: JSONL — one record per item, text field for LLM use

Raw pipeline statistics

{
  "input_count": 600,
  "after_length_filter": 598,
  "after_alpha_filter": 598,
  "after_dedup": 598,
  "operators_used": [
    "CharNumberFilter",
    "HashDeduplicateFilter",
    "python_alpha_ratio_filter"
  ],
  "execution_mode": "dataflow",
  "fallback_mode": false,
  "dataflow_ready": true,
  "dataflow_version": "1.0.10",
  "warnings": []
}

---

Generated by DataFlow PR & Discussion Dataset Builder — Run #34555978870 on 2026-09-11

🌊 Dataset built by DataFlow PR & Discussion Dataset Builder · copilot · auto · 39.3 AIC · ⌖ 10.1 AIC · ⊞ 15.6K · ◷

• expires on Sep 17, 2026, 6:54 PM UTC-08:00