Repository navigation
Data Flow
github-actions[bot] edited this page Oct 13, 2026
·
5 revisions
Language / Dil EN English · TR Türkçe
This document describes the complete data flow through the TC-SGB-API-to-List threat intelligence pipeline, from API ingestion to final output publication.
+=====================================================================+
| TC-SGB-API-to-List Data Flow |
+=====================================================================+
[1] [2] [3] [4]
+------+ +----------+ +---------+ +----------+
| | | | | | | |
| API |----->| Raw Data |---->|Validate |---->|Normalize |
| Call | | Store | | | | |
| | | | | Schema | | Canonical|
| GET | | JSON | | Check | | Format |
| /v1 | | Response | | Type Map| | |
| | | | | Null Chk| | |
+------+ +----------+ +---------+ +----------+
| |
| errors | transformed
v v
+---------+ +----------+
| Error | |Transform |
| Log | | Records |
+---------+ +----------+
|
v
[5] [6] [7] [8]
+----------+ +----------+ +----------+ +----------+
| | | | | | | |
| Dedup |-->| False |-->| Output |---->| Quality |
| | | Positive | | Generate | | Tests |
| Hash | | Control | | | | |
| Content | | | | 16 Fmt | | Valid |
| Seen Map | | Benign | | | | Schema |
| | | Whitelist| | | | Stats |
+----------+ +----------+ +----------+ +----------+
| |
| files | report
v v
+----------+ +----------+
| | | |
| Publish | | Quality |
| | | Report |
| GitHub | | |
| Release | | Metrics |
| PyPI | | Dashboard|
| | | |
+----------+ +----------+
+---------------------------------------------------+
| STAGE 1: API Download |
+---------------------------------------------------+
| |
| Input: API endpoint configuration |
| Output: Raw JSON response per page |
| |
| Process: |
| 1. Resolve API base URL |
| 2. Query total record count |
| 3. Calculate pagination (ceil(total/per_page)) |
| 4. Fetch all pages concurrently (semaphore=5) |
| 5. Store raw responses |
| |
| Rate Limiting: |
| - Max 5 concurrent requests |
| - Min 100ms between sequential requests |
| - Exponential backoff on 429/503 |
| |
| Data Volume: |
| - ~483,690 records (as of 2025) |
| - 500 records per page |
| - ~968 API calls for full fetch |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 2: Raw Data Store |
+---------------------------------------------------+
| |
| Input: Raw JSON API responses |
| Output: Parsed record list (unvalidated) |
| |
| Schema: |
| { |
| "data": [ |
| { |
| "id": int, |
| "type": str, |
| "value": str, |
| "first_seen": str, |
| "last_seen": str, |
| "status": str |
| } |
| ], |
| "meta": { |
| "total": int, |
| "page": int, |
| "per_page": int |
| } |
| } |
| |
| Retention: In-memory during processing |
| Persistence: Optional local cache (JSON) |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 3: Validation |
+---------------------------------------------------+
| |
| Input: Raw parsed records |
| Output: Validated records + error report |
| |
| Checks: |
| +------------------------------------------------+
| | Rule | Action |
| +------------------------------------------------+
| | Required fields present | Reject if missing |
| | ID is integer > 0 | Reject if invalid |
| | Type in enum set | Reject if unknown |
| | Value non-empty string | Reject if empty |
| | Value length <= 2048 | Reject if too long |
| | No null bytes | Reject if contains |
| | No control characters | Strip or reject |
| | Date format valid | Reject if malformed |
| | Status in valid set | Map unknown to default|
| +------------------------------------------------+
| |
| Output: |
| - valid_records: List[IoCRecord] |
| - invalid_records: List[ValidationError] |
| - validation_stats: ValidationReport |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 4: Normalization |
+---------------------------------------------------+
| |
| Input: Validated records |
| Output: Normalized records (canonical form) |
| |
| Transformations by type: |
| |
| domain: |
| - Lowercase |
| - Trim whitespace |
| - Punycode encode (IDN) |
| - Remove trailing dots |
| - Validate against RFC 1035/1123 |
| |
| ip: |
| - Strip whitespace |
| - Validate against ipaddress module |
| - Expand compressed IPv6 |
| - Remove zone IDs |
| |
| ip6: |
| - Compress to shortest form |
| - Lowercase hex |
| - Validate full expansion |
| |
| ip6net: |
| - Normalize prefix length |
| - Validate CIDR notation |
| - Expand network address |
| |
| url: |
| - Percent-encode special chars |
| - Normalize scheme to lowercase |
| - Remove default ports (80/443) |
| - Decode unnecessary percent-encoding |
| - Sort query parameters |
| - Remove fragment identifiers |
| - Remove tracking parameters (utm_*) |
| |
| Metadata: |
| - Standardize date format to ISO 8601 |
| - Normalize status enum |
| - Add processing timestamp |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 5: Deduplication |
+---------------------------------------------------+
| |
| Input: Normalized records |
| Output: Unique records + dedup report |
| |
| Strategy: |
| +------------------------------------------------+
| | Level 1: Exact Match |
| | - Hash(type + normalized_value) |
| | - O(1) lookup in seen set |
| +------------------------------------------------+
| | Level 2: Semantic Match |
| | - URL: ignore query params, fragments |
| | - IP: equivalent notations |
| | - Domain: www prefix equivalence |
| +------------------------------------------------+
| | Level 3: Subdomain Dedup |
| | - *.evil.com deduped to evil.com |
| | - Configurable depth |
| +------------------------------------------------+
| |
| Merge Policy: |
| - Keep earliest first_seen |
| - Keep latest last_seen |
| - Union of status flags |
| - Preserve source attribution |
| |
| Statistics: |
| - Total input records |
| - Duplicates removed |
| - Dedup ratio |
| - Per-type breakdown |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 6: False Positive Control |
+---------------------------------------------------+
| |
| Input: Deduplicated records |
| Output: Verified records + FP report |
| |
| Whitelist Checks: |
| +------------------------------------------------+
| | Check | Action |
| +------------------------------------------------+
| | Known benign domains | Flag/Exclude |
| | CDN infrastructure | Flag |
| | Major cloud providers | Flag |
| | Government domains | Flag |
| | Popular websites | Flag |
| +------------------------------------------------+
| |
| Heuristics: |
| - Newly registered domains (< 30 days) |
| - High-entropy subdomains (DGA detection) |
| - IP address in private ranges |
| - Reserved TLD (.local, .test, .example) |
| - Self-referencing values |
| |
| Configuration: |
| - Whitelist file: config/whitelist.txt |
| - Strict mode: reject all flagged |
| - Advisory mode: flag but include |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 7: Output Generation |
+---------------------------------------------------+
| |
| Input: Verified records |
| Output: 16 formatted output files |
| |
| Output Formats: |
| +------------------------------------------------+
| | Format | Extension | Use Case |
| +------------------------------------------------+
| | NextDNS | .txt | DNS blocking |
| | AdGuard | .txt | DNS blocking |
| | Pi-hole | .txt | DNS blocking |
| | dnsmasq | .conf | DNS blocking |
| | Unbound | .conf | DNS blocking |
| | RPZ | .zone | DNS RPZ zone |
| | Technitium | .zone | DNS blocking |
| | MikroTik | .rsc | RouterOS import |
| | nftables | .nft | Firewall rules |
| | ipset | .ipset | Kernel firewall |
| | Suricata | .json | IDS/IPS |
| | CrowdSec | .yaml | CrowdSec |
| | CSV | .csv | Spreadsheet |
| | JSON | .json | General |
| | YAML | .yaml | Config/CI |
| | SQLite | .db | Database query |
| +------------------------------------------------+
| |
| Each format includes: |
| - Header with metadata |
| - Processing timestamp |
| - Source attribution |
| - Record count |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 8: Quality Tests |
+---------------------------------------------------+
| |
| Input: All output files |
| Output: Quality report + pass/fail |
| |
| Test Categories: |
| +------------------------------------------------+
| | Test | Threshold |
| +------------------------------------------------+
| | Schema validity | 100% pass |
| | Format conformance | 100% pass |
| | Round-trip consistency | JSON=JSON |
| | Record count match | input == output |
| | No duplicate IoCs | 0 duplicates |
| | Field completeness | >99% non-null |
| | Date range validity | All dates parse |
| | Type distribution | Matches expected |
| | Output file sizes | >0 bytes |
| | Encoding (UTF-8) | Valid UTF-8 |
| +------------------------------------------------+
| |
| Report: |
| - Total tests run |
| - Tests passed / failed |
| - Coverage percentage |
| - Performance metrics |
| - Anomaly detection results |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 9: Publication |
+---------------------------------------------------+
| |
| Input: Quality-assured output files |
| Output: Published artifacts |
| |
| Channels: |
| +------------------------------------------------+
| | Channel | Trigger | Format |
| +------------------------------------------------+
| | GitHub Release | Tag push | All formats |
| | GitHub Actions | On success | Artifacts |
| | PyPI | Tag push | Package |
| | Local filesystem | Always | All formats |
| +------------------------------------------------+
| |
| Release Artifacts: |
| - tc-sgb-{version}.tar.gz |
| - tc_sgb-{version}-py3-none-any.whl |
| - Output/*.json |
| - Output/*.csv |
| - Output/*.stix.json |
| - Output/report.html |
| |
+---------------------------------------------------+
+----------+
| Error |
| Detected |
+----------+
|
+----------+----------+
| |
v v
+----------+ +----------+
| Recoverable| | Fatal |
| | | |
+----------+ +----------+
| |
v v
+----------+ +----------+
| Retry with| | Abort |
| backoff | | Pipeline |
| (max 3) | | |
+----------+ +----------+
| |
v v
+----------+ +----------+
| Success? | | Error |
| | | Report |
+----------+ +----------+
| | |
v v v
+------+ +------+ +----------+
| Continue| Abort| | Exit |
| | | | Code 1 |
+------+ +------+ +----------+
Every record carries provenance metadata through the pipeline:
{
"lineage": {
"source": "tc-sgb-api",
"fetch_time": "2025-01-15T10:30:00Z",
"page": 42,
"raw_hash": "sha256:abc123...",
"validation": {
"passed": true,
"checks": 8,
"timestamp": "2025-01-15T10:30:01Z"
},
"normalization": {
"transforms": ["lowercase", "trim", "punycode"],
"timestamp": "2025-01-15T10:30:01Z"
},
"dedup": {
"is_unique": true,
"hash": "sha256:def456...",
"timestamp": "2025-01-15T10:30:02Z"
},
"quality": {
"score": 0.98,
"flags": [],
"timestamp": "2025-01-15T10:30:02Z"
}
}
}Bu belge, TC-SGB-API-to-List tehdit istihbaratı hattındaki API alımından nihai çıktı yayımına kadar olan tam veri akışını tanımlamaktadır.
+=====================================================================+
| TC-SGB-API-to-List Data Flow |
+=====================================================================+
[1] [2] [3] [4]
+------+ +----------+ +---------+ +----------+
| | | | | | | |
| API |----->| Raw Data |---->|Validate |---->|Normalize |
| Call | | Store | | | | |
| | | | | Schema | | Canonical|
| GET | | JSON | | Check | | Format |
| /v1 | | Response | | Type Map| | |
| | | | | Null Chk| | |
+------+ +----------+ +---------+ +----------+
| |
| errors | transformed
v v
+---------+ +----------+
| Error | |Transform |
| Log | | Records |
+---------+ +----------+
|
v
[5] [6] [7] [8]
+----------+ +----------+ +----------+ +----------+
| | | | | | | |
| Dedup |-->| False |-->| Output |---->| Quality |
| | | Positive | | Generate | | Tests |
| Hash | | Control | | | | |
| Content | | | | 16 Fmt | | Valid |
| Seen Map | | Benign | | | | Schema |
| | | Whitelist| | | | Stats |
+----------+ +----------+ +----------+ +----------+
| |
| files | report
v v
+----------+ +----------+
| | | |
| Publish | | Quality |
| | | Report |
| GitHub | | |
| Release | | Metrics |
| PyPI | | Dashboard|
| | | |
+----------+ +----------+
+---------------------------------------------------+
| STAGE 1: API Download |
+---------------------------------------------------+
| |
| Input: API endpoint configuration |
| Output: Raw JSON response per page |
| |
| Process: |
| 1. Resolve API base URL |
| 2. Query total record count |
| 3. Calculate pagination (ceil(total/per_page)) |
| 4. Fetch all pages concurrently (semaphore=5) |
| 5. Store raw responses |
| |
| Rate Limiting: |
| - Max 5 concurrent requests |
| - Min 100ms between sequential requests |
| - Exponential backoff on 429/503 |
| |
| Data Volume: |
| - ~483,690 records (as of 2025) |
| - 500 records per page |
| - ~968 API calls for full fetch |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 2: Raw Data Store |
+---------------------------------------------------+
| |
| Input: Raw JSON API responses |
| Output: Parsed record list (unvalidated) |
| |
| Schema: |
| { |
| "data": [ |
| { |
| "id": int, |
| "type": str, |
| "value": str, |
| "first_seen": str, |
| "last_seen": str, |
| "status": str |
| } |
| ], |
| "meta": { |
| "total": int, |
| "page": int, |
| "per_page": int |
| } |
| } |
| |
| Retention: In-memory during processing |
| Persistence: Optional local cache (JSON) |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 3: Validation |
+---------------------------------------------------+
| |
| Input: Raw parsed records |
| Output: Validated records + error report |
| |
| Checks: |
| +------------------------------------------------+
| | Rule | Action |
| +------------------------------------------------+
| | Required fields present | Reject if missing |
| | ID is integer > 0 | Reject if invalid |
| | Type in enum set | Reject if unknown |
| | Value non-empty string | Reject if empty |
| | Value length <= 2048 | Reject if too long |
| | No null bytes | Reject if contains |
| | No control characters | Strip or reject |
| | Date format valid | Reject if malformed |
| | Status in valid set | Map unknown to default|
| +------------------------------------------------+
| |
| Output: |
| - valid_records: List[IoCRecord] |
| - invalid_records: List[ValidationError] |
| - validation_stats: ValidationReport |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 4: Normalization |
+---------------------------------------------------+
| |
| Input: Validated records |
| Output: Normalized records (canonical form) |
| |
| Transformations by type: |
| |
| domain: |
| - Lowercase |
| - Trim whitespace |
| - Punycode encode (IDN) |
| - Remove trailing dots |
| - Validate against RFC 1035/1123 |
| |
| ip: |
| - Strip whitespace |
| - Validate against ipaddress module |
| - Expand compressed IPv6 |
| - Remove zone IDs |
| |
| ip6: |
| - Compress to shortest form |
| - Lowercase hex |
| - Validate full expansion |
| |
| ip6net: |
| - Normalize prefix length |
| - Validate CIDR notation |
| - Expand network address |
| |
| url: |
| - Percent-encode special chars |
| - Normalize scheme to lowercase |
| - Remove default ports (80/443) |
| - Decode unnecessary percent-encoding |
| - Sort query parameters |
| - Remove fragment identifiers |
| - Remove tracking parameters (utm_*) |
| |
| Metadata: |
| - Standardize date format to ISO 8601 |
| - Normalize status enum |
| - Add processing timestamp |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 5: Deduplication |
+---------------------------------------------------+
| |
| Input: Normalized records |
| Output: Unique records + dedup report |
| |
| Strategy: |
| +------------------------------------------------+
| | Level 1: Exact Match |
| | - Hash(type + normalized_value) |
| | - O(1) lookup in seen set |
| +------------------------------------------------+
| | Level 2: Semantic Match |
| | - URL: ignore query params, fragments |
| | - IP: equivalent notations |
| | - Domain: www prefix equivalence |
| +------------------------------------------------+
| | Level 3: Subdomain Dedup |
| | - *.evil.com deduped to evil.com |
| | - Configurable depth |
| +------------------------------------------------+
| |
| Merge Policy: |
| - Keep earliest first_seen |
| - Keep latest last_seen |
| - Union of status flags |
| - Preserve source attribution |
| |
| Statistics: |
| - Total input records |
| - Duplicates removed |
| - Dedup ratio |
| - Per-type breakdown |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 6: False Positive Control |
+---------------------------------------------------+
| |
| Input: Deduplicated records |
| Output: Verified records + FP report |
| |
| Whitelist Checks: |
| +------------------------------------------------+
| | Check | Action |
| +------------------------------------------------+
| | Known benign domains | Flag/Exclude |
| | CDN infrastructure | Flag |
| | Major cloud providers | Flag |
| | Government domains | Flag |
| | Popular websites | Flag |
| +------------------------------------------------+
| |
| Heuristics: |
| - Newly registered domains (< 30 days) |
| - High-entropy subdomains (DGA detection) |
| - IP address in private ranges |
| - Reserved TLD (.local, .test, .example) |
| - Self-referencing values |
| |
| Configuration: |
| - Whitelist file: config/whitelist.txt |
| - Strict mode: reject all flagged |
| - Advisory mode: flag but include |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 7: Output Generation |
+---------------------------------------------------+
| |
| Input: Verified records |
| Output: 16 formatted output files |
| |
| Output Formats: |
| +------------------------------------------------+
| | Format | Extension | Use Case |
| +------------------------------------------------+
| | NextDNS | .txt | DNS blocking |
| | AdGuard | .txt | DNS blocking |
| | Pi-hole | .txt | DNS blocking |
| | dnsmasq | .conf | DNS blocking |
| | Unbound | .conf | DNS blocking |
| | RPZ | .zone | DNS RPZ zone |
| | Technitium | .zone | DNS blocking |
| | MikroTik | .rsc | RouterOS import |
| | nftables | .nft | Firewall rules |
| | ipset | .ipset | Kernel firewall |
| | Suricata | .json | IDS/IPS |
| | CrowdSec | .yaml | CrowdSec |
| | CSV | .csv | Spreadsheet |
| | JSON | .json | General |
| | YAML | .yaml | Config/CI |
| | SQLite | .db | Database query |
| +------------------------------------------------+
| |
| Each format includes: |
| - Header with metadata |
| - Processing timestamp |
| - Source attribution |
| - Record count |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 8: Quality Tests |
+---------------------------------------------------+
| |
| Input: All output files |
| Output: Quality report + pass/fail |
| |
| Test Categories: |
| +------------------------------------------------+
| | Test | Threshold |
| +------------------------------------------------+
| | Schema validity | 100% pass |
| | Format conformance | 100% pass |
| | Round-trip consistency | JSON=JSON |
| | Record count match | input == output |
| | No duplicate IoCs | 0 duplicates |
| | Field completeness | >99% non-null |
| | Date range validity | All dates parse |
| | Type distribution | Matches expected |
| | Output file sizes | >0 bytes |
| | Encoding (UTF-8) | Valid UTF-8 |
| +------------------------------------------------+
| |
| Report: |
| - Total tests run |
| - Tests passed / failed |
| - Coverage percentage |
| - Performance metrics |
| - Anomaly detection results |
| |
+---------------------------------------------------+
+---------------------------------------------------+
| STAGE 9: Publication |
+---------------------------------------------------+
| |
| Input: Quality-assured output files |
| Output: Published artifacts |
| |
| Channels: |
| +------------------------------------------------+
| | Channel | Trigger | Format |
| +------------------------------------------------+
| | GitHub Release | Tag push | All formats |
| | GitHub Actions | On success | Artifacts |
| | PyPI | Tag push | Package |
| | Local filesystem | Always | All formats |
| +------------------------------------------------+
| |
| Release Artifacts: |
| - tc-sgb-{version}.tar.gz |
| - tc_sgb-{version}-py3-none-any.whl |
| - Output/*.json |
| - Output/*.csv |
| - Output/*.stix.json |
| - Output/report.html |
| |
+---------------------------------------------------+
+----------+
| Error |
| Detected |
+----------+
|
+----------+----------+
| |
v v
+----------+ +----------+
| Recoverable| | Fatal |
| | | |
+----------+ +----------+
| |
v v
+----------+ +----------+
| Retry with| | Abort |
| backoff | | Pipeline |
| (max 3) | | |
+----------+ +----------+
| |
v v
+----------+ +----------+
| Success? | | Error |
| | | Report |
+----------+ +----------+
| | |
v v v
+------+ +------+ +----------+
| Continue| Abort| | Exit |
| | | | Code 1 |
+------+ +------+ +----------+
Her kayıt, hat boyunca köken meta verilerini taşır:
{
"lineage": {
"source": "tc-sgb-api",
"fetch_time": "2025-01-15T10:30:00Z",
"page": 42,
"raw_hash": "sha256:abc123...",
"validation": {
"passed": true,
"checks": 8,
"timestamp": "2025-01-15T10:30:01Z"
},
"normalization": {
"transforms": ["lowercase", "trim", "punycode"],
"timestamp": "2025-01-15T10:30:01Z"
},
"dedup": {
"is_unique": true,
"hash": "sha256:def456...",
"timestamp": "2025-01-15T10:30:02Z"
},
"quality": {
"score": 0.98,
"flags": [],
"timestamp": "2025-01-15T10:30:02Z"
}
}
}TC-SGB-API-to-List — Threat Intelligence Pipeline
Open source. Community driven.
Açık kaynak. Topluluk güdümlü.
License: MIT License Lisans: MIT Lisansı
Disclaimer: TC-SGB-API-to-List is an independent open-source project. IoC data originates from the T.C. Siber Güvenlik Başkanlığı (Cyber Security Presidency) public API. This project is not affiliated with, endorsed by, or officially connected to TC SGB in any way. Sorumluluk Reddi: TC-SGB-API-to-List bağımsız bir açık kaynak projesidir. IoC verileri T.C. Siber Güvenlik Başkanlığı (TC SGB) kamu API'sinden sağlanmaktadır. Bu proje hiçbir şekilde TC SGB ile bağlantılı, onun tarafından onaylanmış veya resmî olarak ilişkili değildir.
- Home
- Architecture
- Data Flow
- Module Architecture
- Repository Structure
- API Analysis
- Data Model
- Threat Model
- Security Analysis
- License Analysis
- Test Strategy
- Regression Strategy
- Performance Strategy
- Versioning Strategy
- Publishing Strategy
- Maintenance Plan
- Risk Analysis
- Roadmap
- Legal Notices
- Troubleshooting
- Ana Sayfa
- Mimari
- Veri Akışı
- Modül Mimarisi
- Depo Yapısı
- API Analizi
- Veri Modeli
- Tehdit Modeli
- Güvenlik Analizi
- Lisans Analizi
- Test Stratejisi
- Regresyon Stratejisi
- Performans Stratejisi
- Sürümleme Stratejisi
- Yayın Stratejisi
- Bakım Planı
- Risk Analizi
- Yol Haritası
- Yasal Bildirimler
- Sorun Giderme
TC-SGB-API-to-List — Threat Intelligence Pipeline Open source. Community driven. Açık kaynak. Topluluk güdümlü.