Bulk Resource & Publication Discovery for EA Organizations

[OAGr]

Goal

Massively expand the wiki's resource/publication coverage for EA-related organizations. Currently the wiki has 5,346 resources in PG but metadata coverage is poor (9% have authors, 9% have dates, 5% have abstracts), and there's no systematic publication discovery for the 592 tracked organizations.

Target: Go from ~5K sparse resources to ~30-60K well-enriched resources covering the major outputs of EA/AI safety organizations, forums, and academic researchers.

Current State (as of 2026-04-02)

Resources in PG

• 5,346 total resources
• 9% have authors, 9% have dates, 5% have abstracts, 42% have summaries
• 4,480 need web fetching, 185 need forum enrichment, 127 need Semantic Scholar, 61 need arXiv
• Resources are primarily page citations scraped from MDX content — not a curated research library

Literature.yaml

• 66 hand-curated papers across ~10 categories
• Only 30 attributed to orgs: Anthropic (7), DeepMind (6), OpenAI (4), GovAI (3), Google (3), MIRI (1)
• Most EA orgs have zero publications listed

Organization Coverage

• 592 organizations tracked (71 safety-orgs, 55 funders, 54 academic, 10 frontier labs)
• All major EA orgs present but publication linkage is almost empty

---

Phase 0: Enrich Existing Resources ($0 cost)

Issue: #3576

Run the existing free enrichment pipeline on all 5,346 current resources:

pnpm crux resources enrich-papers --limit 500    # Semantic Scholar API
pnpm crux resources enrich-crossref --limit 500  # Crossref API (free, no key)
pnpm crux resources enrich-forums --limit 500    # LW/AF/EAF GraphQL
pnpm crux resources fetch-all --batch 200         # URL meta tags
pnpm crux resources enrich-free                   # Convenience: all of the above

Expected outcome: >50% author coverage, >50% date coverage on existing resources.

---

Phase 1: LLM Batch Enrichment (~$115)

Issue: #3577

After Phase 0:

pnpm crux resources classify          # Haiku batch: ~$15
pnpm crux resources deep-enrich       # Sonnet batch: ~$100
pnpm crux resources cross-reference   # Link to org entities: $0

Expected outcome: >80% with summaries, >50% linked to org entities.

---

Phase 2: Bulk Resource Discovery (NEW)

This is the big push. Multiple independent discovery channels, each targeting different source types.

---

2A. LessWrong / EA Forum / Alignment Forum — Global Post Import

The single biggest win. ~19,000-37,000 high-quality posts available via free GraphQL API.

API Capabilities (verified 2026-04-02)

Both LW and EAF expose a public GraphQL API at:

• LW: https://www.lesswrong.com/graphql
• EAF: https://forum.effectivealtruism.org/graphql
• AF: https://www.alignmentforum.org/graphql

Key query: Global post listing with filtering and sorting:

query {
  posts(input: {
    terms: {
      sortedBy: "top",       # or "new"
      limit: 500,            # max page size
      offset: 0,
      karmaThreshold: 30,    # minimum karma score
      after: "2024-01-01",   # date window start
      before: "2024-07-01"   # date window end
    }
  }) {
    results {
      _id title slug postedAt baseScore voteCount
      user { displayName slug }
      coauthors { displayName slug }
    }
  }
}

Full markdown content also available (no scraping needed):

query {
  post(input: { selector: { _id: "abc123" } }) {
    result {
      title
      contents { markdown }
      user { displayName }
      coauthors { displayName }
      postedAt
    }
  }
}

API Constraints

• Max skip (offset): ~2,500 — exceeding this returns "Exceeded maximum value for skip"
• Workaround: Use half-year date windows (after/before) to keep each window under 2,500 posts
• Rate limits: No hard documented limit. 200-300ms delay between requests is sufficient.
• No auth required
• totalCount returns null on global queries — must paginate until empty
• karmaThreshold works as a server-side filter (efficient, reduces result set before pagination)

Measured Post Counts (karma >= 10)

Period	LW Posts	EAF Posts
2015 H1	414	—
2018 H1	606	—
2018 H2	617	214
2019 H1	717	393
2019 H2	922	468
2020 H1	1,145	582
2020 H2	1,192	499
2021 H1	1,239	498
2021 H2	999	917
2022 H1	1,484	499
2022 H2	1,692	997
2023 H1	2,095	999
2023 H2	1,639	1,401
2024 H1	1,501	1,193
2024 H2	1,427	1,110
2025 H1	1,762	1,169
2025 H2	2,135	1,212
2026 H1	1,102	561
TOTAL	23,584	13,388

Combined total (karma >= 10): ~36,972 posts

Cost Estimates by Karma Threshold

Threshold	Estimated Posts	Approx Percentile	LLM Enrichment Cost
karma >= 100	~5,500	Top ~5%	~$13
karma >= 50	~11,000	Top ~12%	~$26
karma >= 30	~19,000	Top ~20%	~$44
karma >= 10	~37,000	Top ~35%	~$85

Recommendation: Start with karma >= 30 (~19K posts, ~$44 LLM cost). This captures substantive posts while filtering spam and low-effort content. Can always lower the threshold later.

Full Cost Breakdown (karma >= 30)

Step	Cost	Time
Discovery (GraphQL metadata fetch)	$0	~1 minute
Resource creation (PG batch upsert)	$0	~30 seconds
Content fetching (full markdown via GraphQL)	$0	~95 minutes
LLM classification (Haiku batch)	~$6	async batch
LLM deep enrichment (Sonnet batch)	~$38	async batch
Cross-referencing (entity linking)	$0	~5 minutes
TOTAL	~$44	~2 hours

Implementation Plan

What already exists:

• crux/lib/forum-api.ts — GraphQL client for LW/EAF (needs AF added to ALL_FORUMS)
• crux/scripts/fetch-forum-posts.ts — Single-author post discovery + resource creation
• crux/resource-enrichment/enrich-forums.ts — Enriches existing forum resources with metadata
• POST /api/resources/batch — Batch upsert (200/batch)
• POST /api/resources/suggest — Creates stubs + auto-chains enrichment jobs
• resource_forum_posts sub-table with karma, commentCount, tags, curated, authorUsername fields

What needs building:

1. Add Alignment Forum to ALL_FORUMS in forum-api.ts
2. New function: getGlobalPosts(forum, { karmaThreshold, after, before, sortedBy }) — paginated global query
3. New command: crux resources discover-forums with options:
   • --karma=30 — minimum karma threshold
   • --forums=lw,eaf,af — which forums
   • --after=2020-01-01 — start date
   • --limit=5000 — max resources to create
   • --dry-run — preview without creating
   • --apply — create resources
4. Deduplication logic: normalize forum URLs, detect LW↔AF cross-posts (same _id prefix or title+author match)
5. Wire into enrichment pipeline: after creation, trigger enrich-forums for metadata + optionally chain LLM enrichment

Estimated engineering: ~4-6 hours

Cross-Post Handling

Posts are often cross-posted between LessWrong and Alignment Forum. The resource_forum_posts table has crossPostedFrom and canonicalForum fields (currently unused). The discovery command should:

• Detect cross-posts by matching _id (AF posts often share LW _id)
• Choose canonical URL (prefer AF for alignment-specific, LW for general)
• Store cross-post metadata for reference

---

2B. Forum Author Sweep — Known Researchers

Complements 2A by ensuring we have complete publication lists for known EA researchers.

What exists

• crux/lib/forum-api.ts: fetchAuthorPosts(slug) — fetches all posts by author across forums
• crux/scripts/fetch-forum-posts.ts — creates resources from author's posts
• data/entities/people.yaml — person entities with some having forum slugs

Implementation

1. Build mapping: person entity → forum username(s)
   • Some already have lesswrongSlug or similar fields
   • Could match by display name as fallback
2. Iterate known researchers, call fetchAuthorPosts() for each
3. Bulk-create resources, linking to person entities via authorEntityIds

Advantage over 2A: Directly links posts to person entities (2A creates resources but doesn't know which wiki person wrote them).

Estimated engineering: ~2-3 hours (mostly building the person→slug mapping)

---

2C. Forum Tag-Based Discovery

LW/EAF support querying posts by tag. Many orgs and topics have dedicated tags.

GraphQL Query (not yet implemented, needs verification)

query {
  posts(input: {
    terms: {
      filterSettings: {
        tags: [{ tagId: "TAG_ID", filterMode: "Required" }]
      },
      limit: 500,
      offset: 0,
      sortedBy: "top"
    }
  }) {
    results { _id title baseScore ... }
  }
}

Note: The exact terms syntax for tag filtering needs verification against the LW/EAF API. The filterSettings approach is used by the website but may not be exposed via the public GraphQL API. An alternative is to fetch tag pages directly.

Use Cases

• Find all posts tagged "miri", "anthropic", "alignment-research-center"
• Find all posts tagged "ai-governance", "interpretability", "rlhf"
• Discover topic-specific content not caught by karma threshold alone

Estimated engineering: ~3-4 hours (API exploration + command implementation)

---

2D. Semantic Scholar — Author-Chain Organization Discovery ($0)

For academic orgs, discover papers via known researchers' Semantic Scholar profiles.

What exists

• crux/lib/search/providers/semantic-scholar.ts — S2 API provider
  • searchAuthors(query) → returns authorId, affiliations
  • getAuthorPapers(authorId) → returns all papers by author
• crux/resource-enrichment/enrich-papers.ts — enriches paper resources with S2 metadata

The Pattern

For each org (Anthropic, DeepMind, MIRI, etc.):
  1. Get list of known researchers (from person entities or org relatedEntries)
  2. For each researcher:
     a. searchAuthors("Dario Amodei") → get authorId
     b. Verify affiliation includes org name
     c. getAuthorPapers(authorId) → all their papers
  3. Deduplicate across authors (co-authored papers)
  4. Bulk-create resources with pre-populated metadata

Constraints

• Rate limit: 100 requests per 5 minutes (free tier, no auth)
• Implemented delay: 350ms between requests in provider, 3.1s in enrichment
• Throughput: ~3-5 orgs per 5-minute window
• Best for: Frontier labs and safety orgs with prolific researchers (Anthropic, DeepMind, OpenAI, MIRI, ARC, Redwood)

Estimated Scale

Org	Est. Researchers	Est. Papers	API Requests
Anthropic	~20	~500	~40
DeepMind	~30	~1,000	~60
OpenAI	~20	~400	~40
MIRI	~10	~100	~20
ARC	~5	~50	~10
Redwood	~5	~30	~10
Total	~90	~2,000	~180

Cost: $0 | Time: ~10 minutes | Engineering: ~4 hours

---

2E. OpenAlex API — Institution-Level Paper Discovery ($0)

OpenAlex is the most powerful option for "find all papers from org X" — it supports native institution-level search.

API

GET https://api.openalex.org/works?filter=authorships.institutions.display_name:"Anthropic"&per_page=200&page=1

• Free, no auth required (polite pool: 10 req/sec with email in User-Agent)
• Native institution search — much better than S2's author-by-author approach
• Covers: All academic literature, preprints, books, conference papers
• Returns: title, DOI, abstract, authors with affiliations, citation counts, open access URLs, topics

Advantages over Semantic Scholar

• Can search by institution name directly (no need to know individual researchers)
• Higher rate limits (10 req/sec vs 20 req/min)
• Broader coverage (not just CS papers)
• Returns structured affiliation data

Estimated Scale

Could discover 5,000-15,000 papers across all tracked organizations — many of which won't be in the wiki yet.

Cost: $0 | Engineering: ~6-8 hours (new provider module needed)

---

2F. Org Research Page Scraping ($0 or Firecrawl credits)

Many orgs have /research or /publications pages listing their work.

Known Research Pages

Org	URL	Type
Anthropic	anthropic.com/research	Blog + papers
OpenAI	openai.com/research	Blog + papers
DeepMind	deepmind.google/research/publications	Papers
MIRI	intelligence.org/research	Papers
CAIS	safe.ai/research	Papers
GovAI	governance.ai/research	Papers
ARC	alignment.org/blog	Blog
Open Phil / Coefficient	openphilanthropy.org/research	Reports
80,000 Hours	80000hours.org/research	Articles
FLI	futureoflife.org/publications	Reports

What exists

• crux/lib/search/source-fetcher.ts — fetches URLs, handles Firecrawl, forum APIs, paywalls
• crux/lib/search/fetch-strategies.ts — domain-aware routing
• Firecrawl integration for JS-rendered sites

Implementation

1. Add publicationsUrl field to org entities (or hardcode top 20)
2. Fetch each org's research page
3. Extract links via HTML parsing
4. Filter for paper/blog/report URLs
5. Bulk-create resources

Cost: $0 if using basic fetch, or Firecrawl credits for JS-heavy sites
Engineering: ~6 hours

---

2G. Auto-Update RSS Expansion

The auto-update system already monitors RSS feeds for some orgs. Expanding this catches ongoing publications.

Currently Configured

• OpenAI Blog, DeepMind Blog (RSS)
• Alignment Forum, LessWrong, EA Forum (RSS)
• ML Safety Newsletter, CAIS Newsletter (RSS)
• Anthropic, Meta AI (web-search fallback — no RSS)
• arXiv cs.AI, cs.CL, cs.LG (RSS)

Gap

• Auto-update finds news items and routes to wiki pages — it does not create resource records
• Many org feeds are missing (GovAI, ARC, 80K Hours, FLI, etc.)

Opportunity

• Wire auto-update discovered items into the resource pipeline
• Add more org RSS feeds / web-search sources
• This handles ongoing discovery, not the initial bulk import

---

Existing Tooling Inventory

The codebase already has most of the infrastructure for this push:

Component	Status	Location
LW/EAF GraphQL client	✅ Built	crux/lib/forum-api.ts
Forum post enrichment	✅ Built	crux/resource-enrichment/enrich-forums.ts
Semantic Scholar provider	✅ Built	crux/lib/search/providers/semantic-scholar.ts
Crossref enrichment	✅ Built	crux/resource-enrichment/enrich-crossref.ts
Batch resource upsert (200/batch)	✅ Built	POST /api/resources/batch
Resource suggest + auto-enrich	✅ Built	POST /api/resources/suggest
LLM batch classification (Haiku)	✅ Built	crux/resource-enrichment/classify.ts
LLM deep enrichment (Sonnet)	✅ Built	crux/resource-enrichment/enrich.ts
Cross-referencing (entity linking)	✅ Built	crux/resource-enrichment/cross-reference.ts
Forum content fetching	✅ Built	crux/lib/search/fetch-strategies.ts
URL deduplication	✅ Built	crux/resource-lookup.ts
Single-author forum discovery	✅ Built	crux/scripts/fetch-forum-posts.ts
Source fetcher (Firecrawl, Wayback)	✅ Built	crux/lib/search/source-fetcher.ts
People-resource linking	✅ Built	crux/commands/people/link-resources.ts
Auto-update RSS pipeline	✅ Built	crux/auto-update/feed-fetcher.ts
Global forum post query	❌ Missing	Needs new function in forum-api.ts
discover-forums command	❌ Missing	New crux command
Alignment Forum in ALL_FORUMS	❌ Missing	One-line fix
OpenAlex provider	❌ Missing	New provider module
Org research page scraper	❌ Missing	New discovery module
Forum tag-based discovery	❌ Missing	New GraphQL query

Linked Issues

• Run free enrichment pipeline on all 5,346 existing resources #3576 — Phase 0: Run free enrichment pipeline on existing 5,346 resources
• Run LLM batch enrichment + cross-reference on existing resources #3577 — Phase 1: LLM batch enrichment + cross-referencing

Priority Order

1. Phase 0 (Run free enrichment pipeline on all 5,346 existing resources #3576) — $0, immediate, improves existing resources
2. Phase 1 (Run LLM batch enrichment + cross-reference on existing resources #3577) — ~$115, depends on Phase 0
3. Phase 2A (Forum global import) — $0-85, biggest single win (~19-37K posts)
4. Phase 2B (Forum author sweep) — $0, complements 2A with entity linking
5. Phase 2D (Semantic Scholar org discovery) — $0, ~2K academic papers
6. Phase 2E (OpenAlex) — $0, strongest academic source, more engineering
7. Phase 2F (Org page scraping) — $0, catches non-forum/non-academic content
8. Phase 2C (Forum tag discovery) — $0, niche but useful for topic coverage
9. Phase 2G (Auto-update expansion) — $0, ongoing discovery (not bulk)

Total Estimated Impact

Source	New Resources	API Cost	LLM Cost	Engineering
Phase 0 (existing enrichment)	0 (enriches 5,346)	$0	$0	~1 hour
Phase 1 (LLM enrichment)	0 (enriches 5,346)	$0	~$115	~1 hour
Forum global import	19,000-37,000	$0	$44-85	~6 hours
Forum author sweep	(overlap with above)	$0	$0	~3 hours
Semantic Scholar	~2,000	$0	~$5	~4 hours
OpenAlex	~5,000-15,000	$0	~$15-35	~8 hours
Org page scraping	~500	$0	~$1	~6 hours
TOTAL	~25,000-55,000	$0	~$180-240	~29 hours

[OAGr]

Linked Issues

• Run free enrichment pipeline on all 5,346 existing resources #3576 — Phase 0: Run free enrichment pipeline on existing 5,346 resources
• Run LLM batch enrichment + cross-reference on existing resources #3577 — Phase 1: LLM batch enrichment + cross-referencing

Both issues are part of this epic. Phase 2 sub-issues will be created as implementation begins.

[OAGr]

Phase 0-1 Complete (2026-04-02)

Phase 0: Free Enrichment

Step	Result
Semantic Scholar	86 papers enriched (citation counts, abstracts)
Crossref	126 papers enriched, 69 with author/date updates
Forum enrichment	186 posts enriched (karma, comments, tags)
Fetch-all	Already complete (previous run)
Cross-referencing	508 publisher entity links written
People linking	people-resources.yaml synced

Phase 1: LLM Batch Enrichment

Step	Result
Classification	Already complete (previous run)
Deep enrichment	3,764/3,801 resources enriched (~$20)

Impact

• Summary coverage: 42% → 98%
• Enrichment coverage: 99.3% of all 5,346 resources
• 508 resources linked to publisher organizations

Both #3576 and #3577 closed. Ready for Phase 2 (bulk discovery).