docs: Add DevIndex Data Factory Guides: Orchestrator & Data Hygiene (#9244)

Author: tobiu

apps/devindex/services/Manager.mjs

@@ -45,7 +45,9 @@ class Manager extends Base {
      * Entry point for the application.
      * @returns {Promise<void>}
      */
-    async main() {
+    async initAsync() {
+        await super.initAsync();
+
         const program = new Command();
 
         program

apps/devindex/services/cli.mjs

@@ -6,7 +6,7 @@ import Manager   from './Manager.mjs';
 dotenv.config({quiet: true});
 
 async function start() {
-    await Manager.main();
+    await Manager.ready();
 }
 
 start().catch(console.error);

learn/guides/devindex/data-factory/DataHygiene.md

@@ -0,0 +1,56 @@
+# Data Hygiene & Cleanup
+
+The **Cleanup Service** ([`DevIndex.services.Cleanup`](https://github.com/neomjs/neo/blob/dev/apps/devindex/services/Cleanup.mjs)) acts as the **Garbage Collector** and **State Enforcer** for the DevIndex data pipeline. 
+
+Because the Data Factory operates autonomously—discovering thousands of users and constantly writing to JSON files—data entropy is inevitable. The Cleanup service is invoked automatically by the Orchestrator before any major operation to ensure the system starts with a clean, consistent, and optimized state.
+
+---
+
+## Core Responsibilities
+
+The Cleanup service performs a strict, multi-pass filtering and formatting routine on the entire JSON dataset.
+
+### 1. Threshold Pruning (The Meritocracy Filter)
+To prevent the index from bloating with inactive or low-value data, the system enforces a strict `minTotalContributions` threshold (configured in `config.mjs`).
+
+During cleanup, every profile in the rich `users.jsonl` store is evaluated. If a user's total contributions fall below this threshold, their profile is **permanently deleted**. Subsequently, they are also purged from the `tracker.json` index, ensuring the Updater doesn't waste API quota continually re-evaluating them.
+
+### 2. Blocklist Enforcement
+Privacy is paramount. If a user's GitHub handle appears in `blocklist.json` (usually populated by the Opt-Out service), the Cleanup routine will aggressively hard-delete any trace of that user from all data files (`users.jsonl`, `tracker.json`, etc.). This is an absolute override that happens on every execution.
+
+### 3. Allowlist Protection & "Resurrection"
+Conversely, the `allowlist.json` provides an absolute protective barrier. It serves two distinct functions during cleanup:
+*   **Protection:** If a user is on the allowlist, they are completely exempt from Threshold Pruning. They will remain in the index even if they have 0 contributions.
+*   **Resurrection:** Before pruning begins, the service cross-references the allowlist against the tracker queue. If an allowlisted VIP is somehow missing from the tracker, the Cleanup service will instantly "resurrect" them, injecting a new pending entry (`lastUpdate: null`) into the queue to ensure they are scheduled for the next Updater run.
+
+### 4. The 30-Day "Penalty Box" TTL
+When the Updater encounters an error analyzing a user (e.g., a GraphQL timeout or a 404), that user is placed in `failed.json`—the "Penalty Box."
+
+Users in the Penalty Box are temporarily protected from being completely pruned from the tracker, giving them a chance to be successfully processed in a future run. However, the Cleanup service enforces a strict **30-Day Time-To-Live (TTL)**. 
+
+```javascript readonly
+// Enforce Retention Policy (Penalty Box TTL)
+const THIRTY_DAYS_MS = 30 * 24 * 60 * 60 * 1000;
+const now            = Date.now();
+
+for (const [login, timestamp] of failed) {
+    const ts = new Date(timestamp).getTime();
+    if (now - ts > THIRTY_DAYS_MS) {
+        console.log(`[Cleanup] Expiring failed user (TTL > 30d): ${login}`);
+        failed.delete(login); // Removes Tracker protection
+    }
+}
+```
+If a user remains in a failed state for more than 30 days, their protection is revoked, and the standard pruning logic will expunge them from the system.
+
+### 5. Canonical Sorting
+The final step of the Cleanup routine is purely for Developer Experience (DX) and Git repository health. 
+
+When JSON files are modified by asynchronous services, the order of keys or array elements is often unpredictable. If committed as-is, this creates massive, noisy Git diffs, making code review impossible.
+
+The Cleanup service applies **Canonical Sorting** before writing any data back to disk:
+*   `users.jsonl` is strictly sorted by `total_contributions` (Descending).
+*   `tracker.json` is sorted alphabetically by `login` (Ascending).
+*   `blocklist.json`, `allowlist.json`, and `visited.json` are sorted alphabetically (Ascending).
+
+This guarantees that any Git diff generated by the pipeline accurately reflects *meaningful changes* rather than arbitrary structural shuffling.

learn/guides/devindex/data-factory/Orchestrator.md

@@ -0,0 +1,105 @@
+# The Orchestrator (CLI & Pipeline)
+
+The DevIndex Data Factory is essentially a collection of specialized micro-services (Spider, Updater, Storage, etc.). To coordinate these services into a cohesive, automated workflow, the system relies on the **Orchestrator** layer.
+
+This layer is comprised of three distinct parts:
+1.  **The Entry Point:** `apps/devindex/services/cli.mjs`
+2.  **The Command Router:** [`DevIndex.services.Manager`](https://github.com/neomjs/neo/blob/dev/apps/devindex/services/Manager.mjs)
+3.  **The Automated Pipeline:** `.github/workflows/devindex-pipeline.yml`
+
+---
+
+## The Entry Point (`cli.mjs`)
+
+The entry point for the backend services is incredibly minimal, leaning entirely on the native Neo.mjs component lifecycle.
+
+```javascript readonly
+import Manager from './Manager.mjs';
+
+async function start() {
+    await Manager.ready();
+}
+
+start().catch(console.error);
+```
+
+Because `Manager` is a Neo.mjs singleton (`Neo.setupClass(Manager)`), simply importing the module triggers its instantiation. The `start()` function then simply awaits the native `Manager.ready()` promise, which resolves when the Manager's asynchronous initialization—including executing the requested CLI command—is complete.
+
+---
+
+## The Command Router (`Manager.mjs`)
+
+The `Manager` service uses the `commander` library to parse command-line arguments and `inquirer` to provide interactive prompts for a robust Developer Experience (DX).
+
+Its primary responsibility is mapping high-level commands to specific service executions.
+
+### Available Commands
+*   `update`: Triggers the **Updater** to process a batch of pending users.
+*   `add [username]`: Manually adds or forces an update for a specific user.
+*   `spider`: Triggers the **Spider** to discover new candidates. Offers interactive strategy selection if run without flags.
+*   `cleanup`: Manually triggers the **Data Hygiene** routine.
+*   `optin` / `optout`: Processes issue-based and star-based privacy requests.
+
+### The "Pre-Run Cleanup" Pattern
+A critical architectural pattern enforced by the Manager is the "Pre-Run Cleanup". Before executing any command that reads or modifies the index (like `spider` or `update`), the Manager automatically triggers `Cleanup.run()`.
+
+```javascript readonly
+program
+    .command('update')
+    .action(async (options) => {
+        await Cleanup.run(); // Pre-run hygiene
+        await this.runUpdate(options.limit);
+    });
+```
+This guarantees that the services always operate on valid, sorted, and pruned data, preventing dirty data from polluting the discovery or enrichment processes.
+
+### Smart Scheduling
+When the `update` command is run, the Manager doesn't just blindly pass the whole queue to the Updater. It implements a smart scheduling algorithm:
+1.  It filters out any user who has already been successfully updated *today* (based on the `lastUpdate` timestamp).
+2.  It sorts the remaining backlog, prioritizing completely new users (`lastUpdate: null`) and the oldest records first.
+3.  It slices the queue to the requested batch limit to respect API quotas.
+
+---
+
+## The Automated Pipeline (GitHub Actions)
+
+While a developer can run commands manually via the CLI, the DevIndex is designed to be fully autonomous. The ultimate orchestrator is the GitHub Actions workflow defined in `.github/workflows/devindex-pipeline.yml`.
+
+This workflow runs on an **hourly schedule** and strings the individual services together into a single, atomic "Data Factory" assembly line:
+
+```yaml readonly
+jobs:
+  run-pipeline:
+    steps:
+      # 1. Process Privacy Requests First
+      - name: Run DevIndex Opt-In
+        run: npm run devindex:optin
+        
+      - name: Run DevIndex Opt-Out
+        run: npm run devindex:optout
+
+      # 2. Aggressive Discovery (3x Loop)
+      - name: Run DevIndex Spider
+        run: |
+          for i in 1 2 3; do
+            npm run devindex:spider -- --strategy random
+          done
+
+      # 3. Enrichment & Processing
+      - name: Run DevIndex Updater
+        run: npm run devindex:update -- --limit=800
+
+      # 4. Atomic Persistence
+      - name: Commit, Rebase and Push
+        run: |
+          git add apps/devindex/resources/*.json*
+          git commit -m "chore(devindex): Hourly pipeline update [skip ci]"
+          git pull origin dev --rebase
+          git push origin dev
+```
+
+### Key Pipeline Concepts:
+
+1.  **Privacy-First Execution:** The `optin` and `optout` services run *before* any discovery or enrichment. This ensures we never accidentally index a user who requested removal in the same hour.
+2.  **The 3x Spider Loop:** Because the Spider uses a random-walk algorithm, running it multiple times consecutively (before the Updater) significantly broadens the discovery net while utilizing very little API quota.
+3.  **Atomic Commits:** Rather than each service committing its own changes independently (which would cause massive Git conflicts), the services modify the local JSON files on the runner. Only at the very end of the hourly pipeline are all changes bundled into a single, atomic commit containing the fully processed data. The `[skip ci]` flag prevents infinite loops.

learn/guides/devindex/tree.json

@@ -5,7 +5,9 @@
     {"name": "Privacy & Opt-Out",                  "parentId": null,                      "id": "OptOut"},
     {"name": "The Data Factory",                   "parentId": null,     "isLeaf": false, "id": "DataFactory"},
         {"name": "Introduction",                   "parentId": "DataFactory",             "id": "data-factory/Intro"},
+        {"name": "The Orchestrator",               "parentId": "DataFactory",             "id": "data-factory/Orchestrator"},
         {"name": "Spider Engine",                  "parentId": "DataFactory",             "id": "data-factory/Engine"},
+        {"name": "Data Hygiene & Cleanup",         "parentId": "DataFactory",             "id": "data-factory/DataHygiene"},
         {"name": "Opt-In Service Architecture",    "parentId": "DataFactory",             "id": "data-factory/OptIn"},
         {"name": "Opt-Out Service Architecture",   "parentId": "DataFactory",             "id": "data-factory/OptOut"},
     {"name": "Architecture",                       "parentId": null,                      "id": "Architecture"}