Remove discovery-provider API code, keep indexing#14236
Merged
Conversation
Drop the legacy Flask REST API from the discovery-provider service. The indexing layer (Celery tasks, beat schedule, eth/sol/core indexers) is preserved. nginx already proxies user traffic to api.audius.co, so there's no consumer left for the local Flask server. - Delete src/api/v1/, src/api_helpers.py, src/wsgi.py, api-tests/ - Delete Flask blueprint files in src/queries/ (queries, search, notifications, health_check, block_confirmation, user_signals, prometheus_metrics_exporter, get_redirect_weights, search_queries) and the comments/ subpackage - Delete API-only data-fetching modules in src/queries/ that imported src/api/v1/helpers (get_trending, get_recommended_tracks, get_premium_tracks, search_es, get_top_playlists, get_top_playlists_es, get_trending_ids, get_health) plus the gated_content signature unit test - Refactor get_trending_tracks/get_trending_playlists/get_underground_trending to drop their src.api.v1.helpers dependency (extend_track was API-shaping; format_limit/format_offset are unused after removing the API entry points these queries exposed) - Rewrite src/app.py as celery-only: remove create_app, Flask blueprint registration, CORS, JSON encoder, exception handlers, and ProxyFix - Refactor src/utils/db_session.py to use module-level globals (set_session_managers) instead of Flask current_app, so indexing and tests don't need a Flask app context - Update integration_tests/conftest.py: replace the create_app fixture with a minimal stand-in that initializes the global session managers and exposes a no-op app_context(); drop the test_client fixture - Delete API-coupled integration tests (api/, test_flask_app.py, the two notification tests that imported src.api.v1.utils.extend_notification, test_get_track_comments/test_get_user_comments/test_get_milestones/ test_search_*_tags/test_get_top_playlists_es) - Update infra: drop dev-server.sh and prod-server.sh, prune the Flask-spawning branch from start.sh, drop EXPOSE 5000 and the curl-based HEALTHCHECK from the Dockerfiles, remove the legacy /health_check etc proxy block from nginx_container.conf, and drop [flask]/[cors]/ loglevel_flask from default_config.ini https://claude.ai/code/session_01F62WuJDRnKo6C5TNsviTNf
|
raymondjacobson
approved these changes
May 6, 2026
…r celery-only init Previously `create_new_challenges` was called only in the Flask init path (`create_app` mode) of `src/app.py`. Removing Flask removed the seeding step, so a fresh celery-only deploy starts with an empty `challenges` table and the test suite fails the same way (50+ challenge / notification / aggregate / trending tests broke). Move the seeding into `configure_celery` so production gets it on celery startup, and call it from the integration-test `app` fixture so tests get the rows they expect. Also extend the test `_TestApp` stand-in to wrap a real Flask app and expose `test_request_context()`. Several existing query tests (`test_get_playlists`, `test_mutual_follows`, `test_get_tips`, `test_get_usdc_transactions_history`) push a Flask request context so `paginate_query` can read `flask.request.args`; the previous no-op stand-in only provided `app_context()`. Brings the suite from 75 failed / 461 passed back to 24 failed / 512 passed — the remaining 24 failures are pre-existing flakes on `main` (plus one worktree-environment issue with es-indexer node_modules). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 12, 2026
dylanjeffers
added a commit
that referenced
this pull request
May 12, 2026
… removal (#14298) ## Summary Cleans up 54 Python files (~3.8k lines) in `packages/discovery-provider/` that were orphaned when PR #14236 removed the Flask REST API layer. Every deleted file has zero callers anywhere in the repo — verified by a basename grep sweep plus per-module `from .*.<module> import` / `import <module>` scans. ## Removed **`src/queries/` — query helpers with no callers (47):** `download_csv`, `get_associated_user_id`, `get_associated_user_wallet`, `get_audio_transactions_history`, `get_cid_source`, `get_cid_type_data`, `get_dashboard_wallet_users`, `get_db_seed_restore_status`, `get_developer_apps`, `get_entities_count_check`, `get_events`, `get_feed`, `get_followees_for_user`, `get_followers_for_user`, `get_latest_entities`, `get_oldest_unarchived_play`, `get_playlist_repost_intersection_users`, `get_previously_private_playlists`, `get_previously_unlisted_tracks`, `get_purchasers`, `get_random_tracks`, `get_remix_track_parents`, `get_remixers`, `get_reposters_for_playlist`, `get_reposters_for_track`, `get_sales_aggregate`, `get_savers_for_playlist`, `get_savers_for_track`, `get_saves`, `get_sol_plays`, `get_stems_of`, `get_subscribers`, `get_subsequent_tracks`, `get_support_for_user`, `get_top_followee_saves`, `get_top_followee_windowed`, `get_track_access_info`, `get_track_inspect_info`, `get_track_repost_intersection_users`, `get_track_signature`, `get_tracks_including_unlisted`, `get_unclaimed_id`, `get_user_history`, `get_user_tracks_remixed`, `get_user_with_wallet`, `search_track_tags`, `search_user_tags` **Prod + unit-test pairs that only test each other (4):** - `get_block_confirmation.py` + `get_block_confirmation_unit_test.py` - `get_latest_play.py` + `get_latest_play_unit_test.py` **Orphan test stub (1):** - `get_route_metrics_unit_test.py` — its target `get_route_metrics.py` was deleted with #14236; the file is a one-line docstring pointing to a sibling integration test that also doesn't exist. **Outside queries (2):** - `src/tracer.py` — OpenTelemetry tracer setup. Nothing called `configure_tracer()` after the Flask-app removal; the indexing worker uses Sentry directly. - `src/utils/track_event_constants.py` — a four-key dict (`track_event_types_lookup`) that nothing imports. ## How dead-ness was verified ```bash for f in packages/discovery-provider/src/queries/*.py; do base=$(basename "$f" .py) refs=$(rg -l "\b$base\b" packages/discovery-provider --type py \ | grep -v "^$f$" | wc -l) [ "$refs" = "0" ] && echo "DEAD: $f" done ``` Plus a second pass after deletion: `rg "from .*\.<module> import|import <module>"` for every deleted module name found zero remaining references in `packages/discovery-provider/`. All `__init__.py` files in `src/`, `src/queries/`, `src/utils/` are empty — no re-exports to clean up. ## What was intentionally NOT touched Indexing code (everything under `src/tasks/`, `src/eth_indexing/`, `src/solana/`, `src/challenges/`, the celery beat schedule) — none of it was flagged dead and none is removed. ## Out of scope (deferred to a follow-up audit) Several query files have callers, but only from tests that exercise indexer behavior — e.g. `get_notifications.py` is read by ~17 notification tests that run the indexer and assert what it wrote. Removing those would mean rewriting tests to hit the DB directly, which is a separate refactor. Examples in this bucket: `get_notifications`, `get_challenges`, `get_attestation`, `get_collection_library`, `get_app_name_metrics`, plus several `*_unit_test.py` files paired with prod modules that are themselves test-only. ## Test plan - [x] `python3 -m compileall -q packages/discovery-provider/src packages/discovery-provider/integration_tests` → exit 0 - [x] `rg` for orphan import references to any deleted module → no hits in `packages/discovery-provider/` - [ ] Boot the protocol stack (`npm run protocol`) and confirm celery beat + workers come up cleanly - [ ] Run the discovery-provider test suite — surviving tests should still pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Drop the legacy Flask REST API from the discovery-provider service. The indexing layer (Celery beat schedule, eth/sol/core indexers, notifications/aggregates/trending tasks) is preserved. nginx already proxies user traffic to
api.audius.co/api.staging.audius.co, so there's no remaining consumer for the local Flask server.What's deleted
src/api/v1/(all blueprints, marshmallow models, helpers, request parsers),src/api_helpers.py,src/wsgi.py,api-tests/(Postman collection)src/queries/:queries.py,search.py,search_queries.py,notifications.py,health_check.py,block_confirmation.py,user_signals.py,prometheus_metrics_exporter.py,get_redirect_weights.py, plus thecomments/subpackagesrc/api/v1/and imported deleted helpers:get_trending,get_recommended_tracks,get_premium_tracks,search_es,get_top_playlists,get_top_playlists_es,get_trending_ids,get_health, plus thegated_content/signature_unit_test.pyintegration_tests/api/,test_flask_app.py, the two notification tests that importedsrc.api.v1.utils.extend_notification, and the comment / search-tag / milestones /top_playlists_esquery testsWhat's refactored
src/queries/get_trending_tracks.py,get_trending_playlists.py,get_underground_trending.py: dropped thesrc.api.v1.helpersimports.extend_trackwas API-output shaping (encoded IDs, artwork URLs); the indexing path doesn't need it.format_limit/format_offsetwere only used by the API entry points (get_underground_trending(request, ...),get_full_trending_playlists) which were removed.src/app.py: rewritten as celery-only — no morecreate_app, no Flask blueprint registration, no CORS / JSON encoder / exception handlers /ProxyFix.create_celeryis the sole entry point.src/utils/db_session.py: replacedflask.current_app.db_session_managerlookup with module-level globals (set_session_managers). The celery worker init populates them; integration tests do the same. Indexing and tests no longer require a Flask app context.integration_tests/conftest.py: theappfixture now returns a minimal stand-in with a no-opapp_context(), so existing tests usingwith app.app_context(): db = get_db()keep working. Theclient(Flask test client) fixture is removed.Infra changes
scripts/dev-server.shandscripts/prod-server.shdeleted;scripts/start.shno longer launches a Flask/gunicorn server, only celery beat + workers.Dockerfile.dev/Dockerfile.prod: dropEXPOSE 5000; the dev healthcheck now greps for the celery worker process instead of curling/health_check.nginx_conf/nginx_container.conf: removed the legacy^/(health_check|audio_transactions_check|tips_check|purchases_check|usdc_transactions_check)$block that proxied toserver:5000. All other proxy rules (core, comms, solana-relay, anti-abuse, plugins, upstream API) are unchanged.default_config.ini: dropped the[flask]and[cors]sections and theloglevel_flaskkey.Test plan
python3 -m compileall packages/discovery-provider/src packages/discovery-provider/integration_tests— passes locallyaudius-compose test discovery-provideror equivalent) and confirm indexing-related tests still passnpm run protocol) and confirm:index_trending,calculate_trending_challenges, and notification tasks complete (these were the indexing-required consumers of the refactored trending queries)/core/...→ core service/comms/...→ comms service/solana/...,/attestation/...,/plugins/...→ respective upstreamsapi.audius.co/api.staging.audius.cohttps://claude.ai/code/session_01F62WuJDRnKo6C5TNsviTNf
Generated by Claude Code