Skip to content

Discover Intelligence, Durable Search & Multi-Provider Reliability

Latest
Latest

Choose a tag to compare

View all tags
@ka8540 ka8540 released this 29 Sep 05:55
· 37 commits to master since this release
V1.5
ea5e5bb
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

This release significantly upgrades Sendloom Discover with a new durable, database-first search architecture, stronger semantic role matching, multi-provider public-profile discovery, improved background processing, and much more resilient provider fallback behavior.

Discover now reuses permanent public people before calling external providers, persists every valid unique candidate for future searches and Add More actions, and uses Firecrawl, Tavily, Bright Data, and Apify as a bounded fallback chain.

Highlights

  • Firecrawl-first Discover. Added Firecrawl as the first external public-profile provider, with high-volume LinkedIn search-result discovery and strict Sendloom-side validation.
  • Durable Postgres-first people reuse. Valid public professional data is persisted permanently and reused before any paid provider call.
  • Multi-provider fallback chain. Discover now uses Postgres → Firecrawl → Tavily → Bright Data → Apify.
  • Semantic role intelligence. Searches understand related role families such as Recruiter, Talent Acquisition, Software Engineering, HR, and other normalized role groups instead of relying only on exact titles.
  • Reliable Add More continuation. Provider progress is stored durably so expansion resumes where discovery stopped instead of restarting.
  • Background Discover processing. Long-running searches remain live and continue bounded provider work without prematurely appearing complete.
  • Improved Firecrawl credential reliability. Supports an authorized multi-key Firecrawl pool with safe failover on authentication, credit, and rate-limit errors.
  • Safer email-format regeneration. Existing people can correctly regenerate inferred addresses after company email-format changes.
  • Improved Bright Data and Tavily reliability. Added stronger timeout handling, transient-response protection, continuation, diagnostics, and fallback behavior.

Discover architecture

Discover now follows a durable database-first model.

Permanent public knowledge

Validated public professional data is stored in Postgres and reused across compatible searches.

This means:

  • Existing people are checked before external providers.
  • Provider results are persisted permanently after validation.
  • Redis is used only as an acceleration layer.
  • Redis expiration or loss does not delete durable people or provider progress.
  • Repeated searches can often be served without another paid provider request.

External provider chain

When durable data cannot satisfy a search, Discover uses:

  1. Firecrawl
  2. Tavily
  3. Bright Data
  4. Apify

Providers only discover candidates. Sendloom remains responsible for determining whether a person is valid.

Every candidate still passes:

  • LinkedIn profile validation
  • canonical identity normalization
  • current-employment validation
  • company matching
  • location validation
  • semantic role matching
  • cross-provider deduplication

A provider result alone is never enough to authorize a person.

Firecrawl-first discovery

Firecrawl is now the first external discovery provider.

  • Searches public LinkedIn result metadata without scraping profile pages.
  • Supports up to 50 raw search results per query by default.
  • Uses existing company, location, and semantic-role intelligence.
  • Persists all valid unique people, even when the current UI only needs a smaller batch.
  • Stops early when enough valid people have been discovered.
  • Ignores unrelated Firecrawl exchange-tool results.
  • Keeps Firecrawl API credentials entirely server-side.

Example:

A Firecrawl query may return 50 raw results.

After validation:

  • 18 may prove current employment.
  • 17 may pass location.
  • 17 may pass role validation.

All 17 are persisted even if the current search only displays 10.

The remaining people can be reused by Add More or later searches without another provider call.

Firecrawl credential failover

Firecrawl now supports an authorized server-side key pool.

If one configured key encounters:

  • authentication failure,
  • insufficient credits,
  • permission failure,
  • or rate limiting,

Sendloom can safely try the next configured credential.

The client:

  • stops on the first successful response,
  • never exposes credentials to the browser,
  • never stores credentials in Postgres or Redis,
  • never logs API keys or Authorization headers,
  • and falls back to Tavily when all Firecrawl credentials are unavailable.

This failover is separate from Discover query continuation and does not alter user quota usage.

Semantic role matching

Discover now has stronger semantic role intelligence.

Instead of requiring every candidate title to exactly match the user's text, role searches operate on controlled role families.

Examples include:

  • Software Engineer
  • Senior Software Engineer
  • Lead Software Engineer
  • Backend Software Engineer
  • Frontend Software Engineer

and recruiting-family roles such as:

  • Recruiter
  • Technical Recruiter
  • Talent Acquisition Specialist
  • Talent Acquisition Partner
  • Sourcer
  • Recruiting Leader

Candidates must still pass the configured category and policy rules, so unrelated titles are rejected.

Role intelligence is shared across database reuse and external providers.

Durable provider continuation

Provider continuation is now persisted in Postgres.

Discover tracks provider progress such as:

  • Firecrawl query index
  • Firecrawl queries fetched
  • Firecrawl exhaustion
  • Tavily query continuation
  • Bright Data page continuation
  • Apify page continuation

This prevents expensive provider work from restarting after:

  • application restarts,
  • Redis expiration,
  • instance replacement,
  • background processing boundaries,
  • or later Add More actions.

Per-action execution limits are no longer confused with provider exhaustion.

Add More improvements

Add More now prioritizes already-discovered durable people.

If a previous search discovered more valid people than it displayed:

  • Add More serves those unused people first.
  • No paid provider request is made while reusable people remain.
  • Later provider work resumes from the saved continuation point.
  • Providers do not restart from their first query or page.

This reduces duplicate work and provider cost.

Background Discover processing

Discover background status is now durable and visible while searches are still working.

Improvements include:

  • live processing state,
  • durable expansion status,
  • correct GraphQL timestamps,
  • bounded background continuation,
  • better retry behavior,
  • and protection against searches appearing READY while provider work is still incomplete.

This also fixes the class of issue where a provider could hit its per-action limit while still having valid continuation remaining.

Tavily reliability

Tavily received multiple discovery and reliability improvements:

  • LinkedIn-domain restriction fixes.
  • Search-query syntax fixes.
  • Bounded deterministic query continuation.
  • Safer diagnostics.
  • Provider exhaustion handling.
  • Correct fallback behavior to Bright Data.

Tavily now operates as the fallback after Firecrawl.

Bright Data reliability

Bright Data discovery received extensive hardening.

Improvements include:

  • Google SERP JSON parsing fixes.
  • Safe response-shape diagnostics.
  • transient-response handling,
  • bounded retries,
  • timeout resilience,
  • multiple-page continuation,
  • correct provider exhaustion,
  • and targeted Apify fallback.

Bright continues where Tavily cannot produce enough validated people.

Apify fallback

Apify remains the final targeted people-discovery fallback.

When upstream public-search providers cannot satisfy a request, Apify receives the resolved company, approved role titles, and requested locations.

Valid unique people are persisted to the same durable public-knowledge system.

Email-format regeneration

Company email-format correction and regeneration behavior was fixed.

When a company email format changes:

  • eligible inferred addresses can be regenerated,
  • protected suppression states remain protected,
  • invalid/unavailable people are handled safely,
  • and existing Discover searches can reuse the corrected company format.

Redis and caching

Redis is now explicitly an acceleration layer rather than the permanent source of Discover knowledge.

  • Public people live permanently in Postgres.
  • Provider provenance lives in Postgres.
  • Provider continuation lives in Postgres.
  • Redis stores sanitized reusable result payloads with TTL.
  • Redis loss does not reset provider progress.
  • Cross-user reusable public knowledge remains separated from tenant-private search allocations.

Reliability and diagnostics

Provider diagnostics were expanded with privacy-safe structured events.

Diagnostics now cover:

  • provider start and completion,
  • raw result counts,
  • accepted current-employment evidence,
  • company contradictions,
  • insufficient company evidence,
  • location rejection,
  • semantic role rejection,
  • duplicate rejection,
  • persisted unique people,
  • provider continuation,
  • exhaustion,
  • fallbacks,
  • and timeout state.

Sensitive provider payloads, API keys, emails, and profile URLs are not included in normal structured diagnostics.

Upgrade notes

  • Run:

    prisma migrate deploy

  • This release includes Discover schema changes for durable public people, provider provenance, provider continuation, background expansion state, and Firecrawl provider-chain state.

  • Configure Firecrawl when enabled:

    DISCOVER_FIRECRAWL_ENABLED=true

    FIRECRAWL_API_KEYS=...

    DISCOVER_FIRECRAWL_TIMEOUT_MS=30000

    DISCOVER_FIRECRAWL_MAX_RESULTS_PER_QUERY=50

    DISCOVER_FIRECRAWL_MAX_QUERIES=5

    DISCOVER_FIRECRAWL_MAX_QUERIES_PER_ACTION=2

    DISCOVER_FIRECRAWL_SCRAPE_RESULTS=false

  • Existing FIRECRAWL_API_KEY remains supported as a single-key fallback.

  • Keep existing Tavily, Bright Data, and Apify configuration if those fallbacks are enabled.

  • Node.js 22.x remains recommended for the production/test environment.

Verification

The Firecrawl-first implementation and credential failover were validated with the full automated test suite, provider-chain regression tests, TypeScript validation, Prisma validation, and production-style provider smoke testing.

Discover now behaves much more like a durable search system than a one-shot provider lookup: find once, validate once, persist once, and reuse wherever possible.

Assets 2
Loading