Add Knowledge Graphs dashboard pages

[kevinschaper]

Add Knowledge Graphs Dashboard Pages

Summary

• Add dynamic dashboard pages for upstream knowledge graphs (RTX-KG2, ROBOKOP, PrimeKG, etc.)
• Index page with comparison table showing nodes, edges, and normalization rates
• Detail pages with normalization pipeline funnels, knowledge source flow diagram, epistemic robustness scores, edge type validation, and ABox/TBox classification
• Dynamic SQL generation for pipeline metrics via generate-release-sql.cjs
• Consolidated SQL sources (kg_list, kg_summary, kg_categories, kg_predicates)

What's Changed

Before	After
Hardcoded rtx-kg2_normalization.md, robokop_normalization.md	Single dynamic [knowledge_graph].md
Per-KG SQL files (robokop_node_summary.sql, etc.)	Consolidated kg_summary.sql, kg_pipeline_metrics.sql
Version extraction via Makefile env vars	Version extraction from globals.yml at build time

Complexity Notes

1. RTX-KG2 Naming Mismatch

There's an inconsistency between config and BigQuery naming:

• globals.yml uses rtx_kg2 (underscore)
• BigQuery upstream_data_source uses rtx-kg2 (hyphen)

This is handled by an explicit mapping in the SQL generator:

// generate-release-sql.cjs:258
const GLOBALS_TO_BQ_NAME = {
  'rtx_kg2': 'rtx-kg2'  // globals.yml uses underscore, BigQuery uses hyphen
};

And when generating kg_versions.sql:

// generate-release-sql.cjs:337
const unionClauses = kgNames.map(kg => {
  // Use BigQuery name if different from globals.yml name
  const bqName = GLOBALS_TO_BQ_NAME[kg] || kg;
  return `  SELECT '${bqName}' as knowledge_graph, ...`;
});

2. KG Discovery via Table Pattern Matching

The script auto-discovers KG sources by pattern-matching BigQuery table names rather than hardcoding a list. New KGs are automatically picked up if their tables follow the naming convention:

// generate-release-sql.cjs:376-384
const query = `
  SELECT table_name
  FROM \`${projectId}.${datasetId}.INFORMATION_SCHEMA.TABLES\`
  WHERE (
    table_name LIKE '%_nodes_ingested%'
    OR table_name LIKE '%_nodes_transformed'
    OR table_name LIKE '%_nodes_normalized'
    OR table_name LIKE '%_edges_ingested%'
    OR table_name LIKE '%_edges_transformed'
    OR table_name LIKE '%_edges_normalized'
  )
`;

Table names are parsed to extract KG source, entity type, and stage:

// generate-release-sql.cjs:407-420
// Pattern: {kg_source}_{entity_type}_{stage}[_{version}]
// Examples: rtx_kg2_nodes_ingested_v2_10_0, robokop_edges_transformed

// Ingested tables (with version suffix)
const ingestedMatch = tableName.match(/^(.+?)_(nodes|edges)_ingested_(.+)$/);

// Transformed/normalized tables (no version suffix)
const otherMatch = tableName.match(/^(.+?)_(nodes|edges)_(transformed|normalized)$/);

3. Exclusion Lists (Repeated in Multiple Places)

Non-KG data sources are excluded to ensure the Knowledge Graphs section only shows upstream graph sources. This logic is repeated in:

SQL sources (kg_list.sql, kg_summary.sql):

WHERE upstream_data_source NOT IN (
  'disease_list',           -- EC Core Entity
  'drug_list',              -- EC Core Entity
  'ec_ground_truth',        -- Evaluation data
  'ec_clinical_trials',     -- Internal data
  'kgml_xdtd_ground_truth', -- Evaluation data
  'off_label'               -- Internal data
)

JavaScript generator (generate-release-sql.cjs:220):

const KG_EXCLUSIONS = [
  'disease_list',
  'drug_list',
  'ec_ground_truth',
  'ec_clinical_trials',
  'kgml_xdtd_ground_truth',
  'off_label'
];

Note: This duplication could be consolidated in a future refactor (e.g., single source of truth in a config file).

4. Version Extraction from globals.yml

KG versions are now extracted at build time from pipelines/matrix/conf/base/globals.yml and written to kg_versions.sql. This replaces the previous Makefile approach that parsed globals.yml via shell commands and passed versions as env vars.

// generate-release-sql.cjs:272-297
function extractKGVersions() {
  const globalsContent = fs.readFileSync(GLOBALS_YML_PATH, 'utf8');
  const globals = yaml.load(globalsContent);
  const dataSources = globals.data_sources || {};
  const kgVersions = {};

  for (const [sourceName, sourceConfig] of Object.entries(dataSources)) {
    // Only include known KG sources (not disease_list, drug_list, etc.)
    if (!KG_SOURCES.includes(sourceName)) {
      continue;
    }
    if (sourceConfig && sourceConfig.version !== undefined) {
      kgVersions[sourceName] = String(sourceConfig.version);
    }
  }
  return kgVersions;
}

The explicit KG source list ensures only actual upstream knowledge graphs get versions displayed:

// generate-release-sql.cjs:246-252
const KG_SOURCES = [
  'rtx_kg2',
  'robokop',
  'primekg',
  'spoke',
  'embiology'
];

Files Changed

Added

• pages/Knowledge Graphs/index.md - Index page with comparison table
• pages/Knowledge Graphs/[knowledge_graph].md - Dynamic detail page
• sources/bq/kg_list.sql - KG listing with display names
• sources/bq/kg_summary.sql - Aggregate metrics per KG
• sources/bq/kg_categories.sql - Top category pairs per KG
• sources/bq/kg_predicates.sql - Top predicates per KG

Removed

• pages/normalization/robokop_normalization.md
• pages/normalization/rtx-kg2_normalization.md
• sources/bq/robokop_node_summary.sql
• sources/bq/robokop_edge_summary.sql
• sources/bq/rtx_kg2_node_summary.sql
• sources/bq/rtx_kg2_edge_summary.sql

Modified

• scripts/generate-release-sql.cjs - Added KG discovery and version extraction
• Makefile - Removed version env vars, added new SQL targets
• pages/normalization/index.md - Added link to Knowledge Graphs section

[github-actions]

This pull request has been automatically marked as stale because it has had no activity in the last 14 days.

If this PR is still relevant:

• Please update it to the latest main/master branch
• Address any existing review comments
• Leave a comment to keep it open

Otherwise, it will be closed in 2 weeks if no further activity occurs.

[github-actions]

This pull request has been automatically closed because it has been stale for 90 days with no activity.

If you believe this is still relevant, please feel free to reopen it or create a new PR.

[JacquesVergine]

Looks good to me, thanks Kevin!