feat(data-warehouse): implement together_ai import source

[Gilbert09]

Problem

Together AI was scaffolded as a warehouse source but had no sync logic - the stub had empty fields and did nothing. Together AI is an AI-native cloud for open-source model inference and fine-tuning, and it exposes queryable historical entities (fine-tuning jobs, batch inference jobs, files, deployed endpoints, evaluations) that are useful to join against product data in the warehouse.

Changes

This fills in the stub end-to-end following the implementing-warehouse-sources skill's source.py / settings.py / together_ai.py split.

Endpoints synced (all GET /v1/<resource> with Bearer API-key auth):

Table	Path	Primary key	Notes
fine_tunes	/fine-tunes	id	wrapped {"data": [...]}
batches	/batches	id	bare array
files	/files	id	wrapped; created_at is a Unix int, so unpartitioned
endpoints	/endpoints	id	wrapped
evaluations	/evaluation	workflow_id	bare array; job id is workflow_id, not id
models	/models	id	bare array; global catalog, off by default, unpartitioned

Key decisions:

• SimpleSource, full refresh only. No Together AI list endpoint exposes pagination or a server-side "updated since" filter, so incremental would re-fetch the whole collection every sync anyway. Collections are small per account, so a single full fetch per table is cheap. This matches the API research and keeps the source simple (no resume state to persist).
• Response shape varies per endpoint. Some endpoints wrap rows in {"data": [...]}, others return a bare top-level array. settings.py carries a per-endpoint data_key and the transport unwraps accordingly. A shape mismatch raises ValueError to bypass the retry budget (permanent contract violation).
• Partitioning on stable created_at only where the field is an ISO-8601 datetime. files.created_at and models.created are integer Unix timestamps the datetime partitioner can't parse, so those tables are unpartitioned. Never partition on a churning field.
• All outbound HTTP goes through make_tracked_session() with the bearer token in redact_values.
• get_non_retryable_errors() fails fast on 401/403 (a bad key can't be fixed by retrying).
• lists_tables_without_credentials = True since get_schemas is a static catalog, so the public docs render the Supported tables section.

Kept behind unreleasedSource=True with releaseStatus=ALPHA per the task scope.

Note

API behavior (response shapes, absence of pagination/timestamp filters, evaluations keying on workflow_id) was verified against the current public Together AI API reference docs. It was not curl-verified against the live API - I don't have a Together AI API key in this environment. The parsing is conservative (explicit shape checks, unexpected shapes raise rather than silently drop rows), and the uncertainty is noted in code comments.

How did you test this code?

Automated tests only (I did not run against the live Together AI API). Two new test modules, run with pytest:

• tests/test_together_ai_source.py - source type, config (alpha + unreleased, category, docsUrl), fields, schema listing (full refresh, workflow_id key for evaluations, models off by default), validate_credentials mapping and per-endpoint path probing, source_for_pipeline arg plumbing, non-retryable error matching, canonical descriptions coverage.
• tests/test_together_ai.py - URL building, data_key unwrap vs bare-array parsing, retryable (429/5xx) vs non-retryable (4xx) status handling, unexpected-shape ValueError, credential validation, single-request full-collection fetch, and per-endpoint SourceResponse partition shape.

All 47 new tests pass. The existing test_source_categories and test_source_config_generator suites (1521 tests) also pass, which confirms the category is set and the hand-written generated_configs.py entry matches generator output exactly.

Note

Two things could not be completed in this environment and need follow-up:

• Config generator / schema build: pnpm run generate:source-configs and pnpm run schema:build need a booting Django app (DB connection), which isn't available here. I hand-edited TogetherAISourceConfig in generated_configs.py to api_key: str - the exact output the generator produces for a single required password field, and the config-generator test passing (regenerate-and-compare) confirms it matches. No enum/schema migration was needed (the enum value already exists on master).
• Icon: no frontend/public/services/together_ai.svg exists yet. Fetching a logo needs a Logo.dev key I don't have, so I did not commit a placeholder. iconPath still points at together_ai.png; the icon needs to be added before release. The source is hidden (unreleasedSource=True) so this doesn't affect users yet.
• Docs: the user-facing doc was written per documenting-warehouse-sources but lives at .../together_ai/docs/together-ai.mdx in this repo since no posthog.com checkout was available. It needs to move to posthog.com/contents/docs/cdp/sources/together-ai.mdx, after which audit_source_docs should be run.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

🤖 Agent context

Autonomy: Fully autonomous

Implemented by Claude (Opus 4.8) via PostHog Code. Skills invoked: /implementing-warehouse-sources (architecture contract, base-class choice, tracked transport, incremental rules) and /documenting-warehouse-sources (doc template + slug rule).

Reference sources studied: bluetally (closest analog - SimpleSource-shaped REST source with a static catalog and both test modules) and klaviyo. Together AI's API was researched from the current public reference docs to confirm each endpoint's response shape and the absence of pagination/timestamp filters. ResumableSource was considered and rejected: with no cursor or page token to persist, there is nothing to resume, so SimpleSource is correct. models was set off-by-default because it's a large global catalog identical for every account rather than account-specific data.

---

Created with PostHog Code

[github-actions]

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

• Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
• Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

[greptile-apps]

_{Reviews (1): Last reviewed commit: "feat(data-warehouse): implement together..." | Re-trigger Greptile}

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs

[estefaniarabadan]

Isolated new data-warehouse source scaffold — self-contained under its own source directory with only the expected shared-registry additions (SOURCES.md, generated_configs.py, icon), tests included, CI green. Reviewed as part of a batch. LGTM 👍

[Gilbert09]

This PR appears superseded by #69608 — needs a human call before I can proceed

While taking this PR to a clean, mergeable state I found that the together_ai warehouse source it implements is already fully implemented on master via sibling PR #69608 (same author, merged 2026-07-13). Every source file here now collides with master as an add/add conflict:

• together_ai/canonical_descriptions.py
• together_ai/settings.py
• together_ai/source.py
• together_ai/together_ai.py
• together_ai/tests/test_together_ai.py
• together_ai/tests/test_together_ai_source.py

The two implementations are functionally equivalent, near-complete rewrites of each other, covering the same endpoints (fine_tunes, batches, files, endpoints, evaluations, models). Master's merged version is slightly more refined — e.g. endpoints filters to params={"type": "dedicated"} to avoid duplicating the models catalog.

There's no conflict resolution that "preserves both sides" here:

• Take master's side → the PR diff empties out to nothing (feature already live).
• Take this PR's side → it reverts the already-merged, reviewed code from feat(data-warehouse): implement together_ai import source #69608.

The only content unique to this PR is a temporary in-repo doc, together_ai/docs/together-ai.mdx, which this PR's own description notes should live in the posthog.com repo rather than here.

I've stopped rather than guess, since each path is destructive and this is a product/ownership decision. My recommendation is to close this PR as superseded by #69608. Alternatively, if there's anything in this version worth keeping (e.g. the docs), let me know and I'll rebase so the PR's only diff is that net-new content on top of master.

_{🦉 via talyn.dev}

[Gilbert09]

Correction + concrete deltas vs the shipped #69608

Following up on my earlier "superseded by #69608" note: I said the two implementations were "functionally equivalent." On a closer file-by-file diff that's not accurate — they differ in ways that matter, and a couple look like they could be latent issues in the version that shipped to master via #69608. Flagging so the "just close it" call is made with full information (I have not curl-verified these against the live Together AI API — they need a human/API check):

Aspect	shipped on master (#69608)	this PR (#70011)
evaluations path	/evaluations	/evaluation (singular)
endpoints params	{"type": "dedicated"}	none
files partition key	created_at (an integer Unix timestamp)	unpartitioned
models partition / default sync	created (integer Unix), sync on	unpartitioned, sync off by default
HTTP retries	tenacity wrapping the session's default urllib3 retry	Retry(total=0) so tenacity is the single backoff layer

Two of these are worth verifying regardless of which PR wins, because if this PR is right they're bugs in the already-merged source:

1. Partitioning files/models on an integer Unix timestamp. If those fields are integers rather than ISO-8601 datetimes, the datetime partitioner can't parse them — this PR deliberately leaves both tables unpartitioned for that reason.
2. /evaluations vs /evaluation. The two PRs disagree on the endpoint path; only one can be correct.

The retry-layering change (Retry(total=0)) was the P2 fix reviewed and agreed on this PR.

Recommendation unchanged: master's #69608 is the canonical shipped implementation, so this PR should most likely be closed as superseded — but if the two differences above check out, please port them onto master separately. If you'd instead like me to (a) rebase this PR down to only those corrections on top of master as a small focused-fix diff, or (b) overwrite master with this PR's version, just say which and I'll proceed.

_{🦉 via talyn.dev}

[Gilbert09]

Heads-up on how I resolved the merge conflict here, because it turned out to be more than a mechanical fix.

The together_ai source has since landed on master via #69608, so this branch and master now contain two parallel implementations of the same source. Resolving the conflict by taking this branch's side would have:

• overwritten master's merged implementation (e.g. its more nuanced get_status_code-based credential validation) with this branch's older/lateral version, and
• reverted SOURCES.md / generated_configs.py to a stale state, dropping sources that were added to master after this branch forked (alguna, anthropic, …).

To avoid regressing master, I rebased onto it and took master's version for everything it already ships. I kept only what's genuinely net-new here:

• docs/together-ai.mdx — the user-facing source doc, which master doesn't have (I fixed one line so the "supported tables" note matches the shipped sync behavior, since master has no should_sync_default).
• The layered-retry fix — master's get_rows still uses the default Retry(total=3) under tenacity's 5 attempts, so a rate-limited endpoint can be hammered. I ported the retry=Retry(total=0) fix from bb009dd onto master's implementation. (The other greptile fix, the response-shape guard, is already on master via _extract_rows raising ValueError, so no port was needed.)

Net diff is now just those two files, cleanly on top of master. If you'd rather not land a reduced PR and consider this fully superseded by #69608, feel free to close it — the one useful code change (disabling the double retry layer) would then be worth a small standalone PR against master.

_{🦉 via talyn.dev}

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-14 06:07 UTC	Run
prod-us	✅ Deployed	2026-07-14 06:19 UTC	Run
prod-eu	✅ Deployed	2026-07-14 06:19 UTC	Run