Replies: 4 comments
|
This looks like a completion-status problem, not an indexing problem. In Onyx, the connector only flips out of The first things I’d inspect are:
The fact that it only moves to If you share the attempt row and the worker logs around the last batch, it should be possible to narrow it down quickly. |
0 replies
|
@starkmarkus many thanks for your feedback which helps and matches what I have found in these days. INFO: INFO 06/23/2026 11:30:29 PM beat.py:279 : celery.beat Scheduler: Sending due task celery-beat-heartbeat-public (celery_beat_heartbeat)
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:287 : celery.beat celery_beat_heartbeat sent. id->72244030-98ae-4722-a955-d22099b48e36
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:68 : onyx.background.celery.apps.beat Reload interval reached, initiating task update
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:159 : onyx.background.celery.apps.beat _try_updating_schedule starting
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:162 : onyx.background.celery.apps.beat Found 1 IDs
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for monitor-celery-queues-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task monitor-celery-queues-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900, 'queue': 'monitoring'}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for monitor-process-memory-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task monitor-process-memory-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900, 'queue': 'monitoring'}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for celery-beat-heartbeat-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task celery-beat-heartbeat-public: {'priority': <OnyxCeleryPriority.HIGHEST: 0>, 'expires': 900, 'queue': 'celery'}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-user-file-processing-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-user-file-processing-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-user-file-project-sync-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-user-file-project-sync-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-user-file-delete-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-user-file-delete-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-indexing-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-indexing-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-checkpoint-cleanup-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-checkpoint-cleanup-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-index-attempt-cleanup-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-index-attempt-cleanup-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-connector-deletion-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-connector-deletion-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-vespa-sync-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-vespa-sync-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-pruning-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-pruning-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-hierarchy-fetching-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-hierarchy-fetching-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for monitor-background-processes-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task monitor-background-processes-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900, 'queue': 'monitoring'}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for dispatch-due-scheduled-tasks-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task dispatch-due-scheduled-tasks-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 60, 'queue': 'celery'}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for cleanup-stuck-scheduled-runs-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task cleanup-stuck-scheduled-runs-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 3600, 'queue': 'celery'}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for cleanup-idle-sandboxes-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task cleanup-idle-sandboxes-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900, 'queue': 'sandbox'}
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-auto-llm-update-public
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-auto-llm-update-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900}
INFO: INFO 06/23/2026 11:30:29 PM beat.py:191 : onyx.background.celery.apps.beat _try_updating_schedule - Schedule unchanged: tasks=18 beat_multiplier=8.0
INFO: INFO 06/23/2026 11:30:29 PM beat.py:279 : celery.beat Scheduler: Sending due task cleanup-idle-sandboxes-public (cleanup_idle_sandboxes)
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:287 : celery.beat cleanup_idle_sandboxes sent. id->02f9c97a-00b8-4420-860f-aa636c4ca3e5
INFO: INFO 06/23/2026 11:30:29 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-user-file-processing-public (check_for_user_file_processing)
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:287 : celery.beat check_for_user_file_processing sent. id->ce0f961d-9624-4bb0-b964-29cdd37f6002
INFO: INFO 06/23/2026 11:30:29 PM beat.py:279 : celery.beat Scheduler: Sending due task dispatch-due-scheduled-tasks-public (scheduled_tasks_dispatch_due)
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:287 : celery.beat scheduled_tasks_dispatch_due sent. id->c7552ed5-8f31-4361-95b4-4643ae71c7f8
INFO: INFO 06/23/2026 11:30:29 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-user-file-project-sync-public (check_for_user_file_project_sync)
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:287 : celery.beat check_for_user_file_project_sync sent. id->8256f833-2dfd-4731-8eff-c6367db4541b
INFO: INFO 06/23/2026 11:30:29 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-indexing-public (check_for_indexing)
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:287 : celery.beat check_for_indexing sent. id->238b0a4a-a1fa-4c0d-a438-b4fc41e337db
INFO: INFO 06/23/2026 11:30:29 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-user-file-delete-public (check_for_user_file_delete)
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:287 : celery.beat check_for_user_file_delete sent. id->9cb216a5-3bea-4e65-b023-b6bedf32eb7b
INFO: INFO 06/23/2026 11:30:29 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-connector-deletion-public (check_for_connector_deletion_task)
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:287 : celery.beat check_for_connector_deletion_task sent. id->53c80b20-48c3-4a64-ba4d-ec467dabc3cb
INFO: INFO 06/23/2026 11:30:29 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-vespa-sync-public (check_for_vespa_sync_task)
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:287 : celery.beat check_for_vespa_sync_task sent. id->c7d8f0b5-c22a-4180-ac6c-aae8dde16885
INFO: INFO 06/23/2026 11:30:29 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-pruning-public (check_for_pruning)
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:287 : celery.beat check_for_pruning sent. id->6da3d893-e21b-4c56-8b2d-9901691d4a47
INFO: INFO 06/23/2026 11:30:29 PM beat.py:279 : celery.beat Scheduler: Sending due task monitor-celery-queues-public (monitor_celery_queues)
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:287 : celery.beat monitor_celery_queues sent. id->e7be3bd0-a772-4456-952d-7364dca8e686
DEBUG: DEBUG 06/23/2026 11:30:29 PM beat.py:646 : celery.beat beat: Waking up in 9.99 seconds.
INFO: INFO 06/23/2026 11:30:29 PM request.py:496 : celery.worker.request Discarding revoked task: celery_beat_heartbeat[72244030-98ae-4722-a955-d22099b48e36]
INFO: INFO 06/23/2026 11:30:29 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_user_file_processing[ce0f961d-9624-4bb0-b964-29cdd37f6002]
INFO: INFO 06/23/2026 11:30:29 PM request.py:496 : celery.worker.request Discarding revoked task: scheduled_tasks_dispatch_due[c7552ed5-8f31-4361-95b4-4643ae71c7f8]
INFO: INFO 06/23/2026 11:30:29 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_user_file_project_sync[8256f833-2dfd-4731-8eff-c6367db4541b]
INFO: INFO 06/23/2026 11:30:29 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_indexing[238b0a4a-a1fa-4c0d-a438-b4fc41e337db]
INFO: INFO 06/23/2026 11:30:29 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_user_file_delete[9cb216a5-3bea-4e65-b023-b6bedf32eb7b]
INFO: INFO 06/23/2026 11:30:29 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_connector_deletion_task[53c80b20-48c3-4a64-ba4d-ec467dabc3cb]
INFO: INFO 06/23/2026 11:30:29 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_vespa_sync_task[c7d8f0b5-c22a-4180-ac6c-aae8dde16885]
INFO: INFO 06/23/2026 11:30:29 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_pruning[6da3d893-e21b-4c56-8b2d-9901691d4a47]
INFO: INFO 06/23/2026 11:30:29 PM request.py:496 : celery.worker.request Discarding revoked task: cleanup_idle_sandboxes[02f9c97a-00b8-4420-860f-aa636c4ca3e5]
INFO: INFO 06/23/2026 11:30:29 PM request.py:496 : celery.worker.request Discarding revoked task: monitor_celery_queues[e7be3bd0-a772-4456-952d-7364dca8e686]
DEBUG: 06/23/2026 11:30:28 PM listener.py 510: Sending heartbeats for 0 active tenants
DEBUG: 06/23/2026 11:30:28 PM listener.py 205: Sent heartbeats for 0 active tenants
DEBUG: DEBUG 06/23/2026 11:30:30 PM primary.py:310 : Reacquiring primary worker lock.
WARNING: 06/23/2026 11:30:37 PM supervisord_watchdog.py 67: Key lookup failed: key=onyx:celery:beat:heartbeat lookup_failures=12 max_lookup_failures=5 elapsed=720.01 elapsed_threshold=900then the celery_beat was terminated and restarted. 2026-06-23 23:33:37,638 INFO waiting for celery_beat to stop
2026-06-23 23:33:38,005 INFO stopped: celery_beat (exit status 0)
2026-06-23 23:33:38,008 INFO spawned: 'celery_beat' with pid 1236
celery beat v5.5.1 (immunity) is starting.
__ - ... __ - _
LocalTime -> 2026-06-23 23:18:29
Configuration ->
. broker -> redis://onyx-cache:6379/15
. loader -> celery.loaders.app.AppLoader
. scheduler -> onyx.background.celery.apps.beat.DynamicTenantScheduler
. db -> celerybeat-schedule
. logfile -> [stderr]@%WARNING
. maxinterval -> 5.00 minutes (300s)
WARNING: 06/23/2026 11:33:37 PM supervisord_watchdog.py 78: Key lookup failure thresholds exceeded - restarting celery_beat: key=onyx:celery:beat:heartbeat lookup_failures=15 max_lookup_failures=5 elapsed=900.01 elapsed_threshold=900
DEBUG: 06/23/2026 11:33:39 PM app_base.py 76: Sentry DSN not provided, skipping Sentry initialization
INFO: 06/23/2026 11:33:39 PM app_base.py 529: Celery effective log level: DEBUG (source: LOG_LEVEL env var ('debug'))
INFO: INFO 06/23/2026 11:33:39 PM app_base.py:529 : onyx.utils.logger Celery effective log level: DEBUG (source: LOG_LEVEL env var ('debug'))
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:108 : celery.beat Setting default socket timeout to 30
INFO: INFO 06/23/2026 11:33:39 PM beat.py:633 : celery.beat beat: Starting...
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:565 : celery.beat Current schedule:
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:565 : celery.beat Current schedule:
INFO: INFO 06/23/2026 11:33:39 PM beat.py:45 : onyx.background.celery.apps.beat DynamicTenantScheduler initialized: reload_interval=0:01:00
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:634 : celery.beat beat: Ticking with max interval->5.00 minutes
INFO: INFO 06/23/2026 11:33:39 PM beat.py:251 : onyx.background.celery.apps.beat beat_init signal received.
INFO: 06/23/2026 11:33:39 PM sql_engine.py 262: Creating engine with kwargs: {'connect_args': {'keepalives': 1, 'keepalives_idle': 30, 'keepalives_interval': 10, 'keepalives_count': 5}, 'pool_size': 2, 'max_overflow': 0, 'pool_pre_ping': True, 'pool_recycle': 1200}
INFO: INFO 06/23/2026 11:33:39 PM sql_engine.py:262 : onyx.utils.logger Creating engine with kwargs: {'connect_args': {'keepalives': 1, 'keepalives_idle': 30, 'keepalives_interval': 10, 'keepalives_count': 5}, 'pool_size': 2, 'max_overflow': 0, 'pool_pre_ping': True, 'pool_recycle': 1200}
INFO: 06/23/2026 11:33:39 PM app_base.py 283: Redis: Readiness probe starting.
INFO: INFO 06/23/2026 11:33:39 PM app_base.py:283 : onyx.utils.logger Redis: Readiness probe starting.
INFO: 06/23/2026 11:33:39 PM app_base.py 309: Redis: Readiness probe succeeded. Continuing...
INFO: INFO 06/23/2026 11:33:39 PM app_base.py:309 : onyx.utils.logger Redis: Readiness probe succeeded. Continuing...
INFO: INFO 06/23/2026 11:33:39 PM beat.py:260 : onyx.background.celery.apps.beat Readiness signal touched at /tmp/onyx_k8s_beat_readiness.txt.
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:159 : onyx.background.celery.apps.beat _try_updating_schedule starting
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:162 : onyx.background.celery.apps.beat Found 1 IDs
DEBUG: 06/23/2026 11:33:39 PM variable_functionality.py 93: Fetching versioned implementation for onyx.background.celery.tasks.beat_schedule.get_tasks_to_schedule
DEBUG: DEBUG 06/23/2026 11:33:39 PM variable_functionality.py:93 : onyx.utils.logger Fetching versioned implementation for onyx.background.celery.tasks.beat_schedule.get_tasks_to_schedule
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for monitor-celery-queues-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task monitor-celery-queues-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900, 'queue': 'monitoring'}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for monitor-process-memory-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task monitor-process-memory-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900, 'queue': 'monitoring'}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for celery-beat-heartbeat-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task celery-beat-heartbeat-public: {'priority': <OnyxCeleryPriority.HIGHEST: 0>, 'expires': 900, 'queue': 'celery'}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-user-file-processing-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-user-file-processing-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-user-file-project-sync-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-user-file-project-sync-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-user-file-delete-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-user-file-delete-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-indexing-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-indexing-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-checkpoint-cleanup-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-checkpoint-cleanup-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-index-attempt-cleanup-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-index-attempt-cleanup-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-connector-deletion-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-connector-deletion-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-vespa-sync-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-vespa-sync-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-pruning-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-pruning-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-hierarchy-fetching-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-hierarchy-fetching-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for monitor-background-processes-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task monitor-background-processes-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900, 'queue': 'monitoring'}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for dispatch-due-scheduled-tasks-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task dispatch-due-scheduled-tasks-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 60, 'queue': 'celery'}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for cleanup-stuck-scheduled-runs-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task cleanup-stuck-scheduled-runs-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 3600, 'queue': 'celery'}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for cleanup-idle-sandboxes-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task cleanup-idle-sandboxes-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900, 'queue': 'sandbox'}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-auto-llm-update-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-auto-llm-update-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:199 : onyx.background.celery.apps.beat Schedule update required
INFO: INFO 06/23/2026 11:33:39 PM beat.py:226 : onyx.background.celery.apps.beat _try_updating_schedule - Schedule updated: prev_num_tasks=18 prev_beat_multiplier=8.0 tasks=18 beat_multiplier=8.0
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:68 : onyx.background.celery.apps.beat Reload interval reached, initiating task update
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:159 : onyx.background.celery.apps.beat _try_updating_schedule starting
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:162 : onyx.background.celery.apps.beat Found 1 IDs
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for monitor-celery-queues-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task monitor-celery-queues-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900, 'queue': 'monitoring'}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for monitor-process-memory-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task monitor-process-memory-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900, 'queue': 'monitoring'}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for celery-beat-heartbeat-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task celery-beat-heartbeat-public: {'priority': <OnyxCeleryPriority.HIGHEST: 0>, 'expires': 900, 'queue': 'celery'}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-user-file-processing-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-user-file-processing-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-user-file-project-sync-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-user-file-project-sync-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-user-file-delete-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-user-file-delete-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-indexing-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-indexing-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-checkpoint-cleanup-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-checkpoint-cleanup-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-index-attempt-cleanup-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-index-attempt-cleanup-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-connector-deletion-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-connector-deletion-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-vespa-sync-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-vespa-sync-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-pruning-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-pruning-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-hierarchy-fetching-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-hierarchy-fetching-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for monitor-background-processes-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task monitor-background-processes-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900, 'queue': 'monitoring'}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for dispatch-due-scheduled-tasks-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task dispatch-due-scheduled-tasks-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 60, 'queue': 'celery'}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for cleanup-stuck-scheduled-runs-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task cleanup-stuck-scheduled-runs-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 3600, 'queue': 'celery'}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for cleanup-idle-sandboxes-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task cleanup-idle-sandboxes-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900, 'queue': 'sandbox'}
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-auto-llm-update-public
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-auto-llm-update-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900}
INFO: INFO 06/23/2026 11:33:39 PM beat.py:191 : onyx.background.celery.apps.beat _try_updating_schedule - Schedule unchanged: tasks=18 beat_multiplier=8.0
DEBUG: DEBUG 06/23/2026 11:33:39 PM beat.py:646 : celery.beat beat: Waking up in 9.99 seconds.
DEBUG: 06/23/2026 11:33:43 PM listener.py 510: Sending heartbeats for 0 active tenants
DEBUG: 06/23/2026 11:33:43 PM listener.py 205: Sent heartbeats for 0 active tenants
DEBUG: DEBUG 06/23/2026 11:33:45 PM primary.py:310 : Reacquiring primary worker lock.
2026-06-23 23:33:48,491 INFO success: celery_beat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
celery_beat: stopped
celery_beat: startedthis happened periodically. |
0 replies
|
@starkmarkus and all DEBUG: DEBUG 06/29/2026 05:13:42 PM primary.py:310 : Reacquiring primary worker lock.
INFO: INFO 06/29/2026 05:13:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-user-file-processing-public (check_for_user_file_processing)
DEBUG: DEBUG 06/29/2026 05:13:47 PM beat.py:287 : celery.beat check_for_user_file_processing sent. id->ea0b950d-ecdf-492a-86af-a337948506dc
INFO: INFO 06/29/2026 05:13:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-user-file-project-sync-public (check_for_user_file_project_sync)
DEBUG: DEBUG 06/29/2026 05:13:47 PM beat.py:287 : celery.beat check_for_user_file_project_sync sent. id->048c7dd3-4fb9-448a-bd25-bc60ac532d37
INFO: INFO 06/29/2026 05:13:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-user-file-delete-public (check_for_user_file_delete)
DEBUG: DEBUG 06/29/2026 05:13:47 PM beat.py:287 : celery.beat check_for_user_file_delete sent. id->440bc651-f24c-4496-b118-1e29ef9c5c43
INFO: INFO 06/29/2026 05:13:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-connector-deletion-public (check_for_connector_deletion_task)
DEBUG: DEBUG 06/29/2026 05:13:47 PM beat.py:287 : celery.beat check_for_connector_deletion_task sent. id->cc5b549a-3bfe-436e-8ede-5bbbb7ce3723
INFO: INFO 06/29/2026 05:13:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-vespa-sync-public (check_for_vespa_sync_task)
DEBUG: DEBUG 06/29/2026 05:13:47 PM beat.py:287 : celery.beat check_for_vespa_sync_task sent. id->2d6a1ee1-8698-43a5-a727-8ec173dc7d1b
INFO: INFO 06/29/2026 05:13:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-pruning-public (check_for_pruning)
DEBUG: DEBUG 06/29/2026 05:13:47 PM beat.py:287 : celery.beat check_for_pruning sent. id->76e8870b-8789-4c29-9eda-266f46b75ee2
INFO: INFO 06/29/2026 05:13:47 PM beat.py:279 : celery.beat Scheduler: Sending due task monitor-celery-queues-public (monitor_celery_queues)
DEBUG: DEBUG 06/29/2026 05:13:47 PM beat.py:287 : celery.beat monitor_celery_queues sent. id->24465dd5-a9f7-48ba-84e1-0ddd60b9334e
DEBUG: DEBUG 06/29/2026 05:13:47 PM beat.py:646 : celery.beat beat: Waking up in 4.98 seconds.
INFO: INFO 06/29/2026 05:13:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_user_file_processing[ea0b950d-ecdf-492a-86af-a337948506dc]
INFO: INFO 06/29/2026 05:13:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_user_file_project_sync[048c7dd3-4fb9-448a-bd25-bc60ac532d37]
INFO: INFO 06/29/2026 05:13:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_user_file_delete[440bc651-f24c-4496-b118-1e29ef9c5c43]
INFO: INFO 06/29/2026 05:13:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_connector_deletion_task[cc5b549a-3bfe-436e-8ede-5bbbb7ce3723]
INFO: INFO 06/29/2026 05:13:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_vespa_sync_task[2d6a1ee1-8698-43a5-a727-8ec173dc7d1b]
INFO: INFO 06/29/2026 05:13:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_pruning[76e8870b-8789-4c29-9eda-266f46b75ee2]
INFO: INFO 06/29/2026 05:13:47 PM request.py:496 : celery.worker.request Discarding revoked task: monitor_celery_queues[24465dd5-a9f7-48ba-84e1-0ddd60b9334e]
INFO: INFO 06/29/2026 05:13:52 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-indexing-public (check_for_indexing)
DEBUG: DEBUG 06/29/2026 05:13:52 PM beat.py:287 : celery.beat check_for_indexing sent. id->f4faed17-104e-4a16-8cba-2de60c79c0c5
DEBUG: DEBUG 06/29/2026 05:13:52 PM beat.py:646 : celery.beat beat: Waking up in 4.99 seconds.
INFO: INFO 06/29/2026 05:13:52 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_indexing[f4faed17-104e-4a16-8cba-2de60c79c0c5]
DEBUG: 06/29/2026 05:13:56 PM listener.py 510: Sending heartbeats for 0 active tenants
DEBUG: 06/29/2026 05:13:56 PM listener.py 205: Sent heartbeats for 0 active tenants
INFO: INFO 06/29/2026 05:13:57 PM beat.py:279 : celery.beat Scheduler: Sending due task monitor-celery-queues-public (monitor_celery_queues)
DEBUG: DEBUG 06/29/2026 05:13:57 PM beat.py:287 : celery.beat monitor_celery_queues sent. id->a6c3418e-21df-4268-bee2-890970ed1d8d
DEBUG: DEBUG 06/29/2026 05:13:57 PM beat.py:646 : celery.beat beat: Waking up in 9.98 seconds.
DEBUG: DEBUG 06/29/2026 05:13:57 PM primary.py:310 : Reacquiring primary worker lock.
INFO: INFO 06/29/2026 05:13:57 PM request.py:496 : celery.worker.request Discarding revoked task: monitor_celery_queues[a6c3418e-21df-4268-bee2-890970ed1d8d]
DEBUG: DEBUG 06/29/2026 05:14:04 PM pidbox.py:98 : kombu.pidbox pidbox received method revoke(task_id='docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7', terminate=False, signal='SIGTERM') [reply_to:None ticket:None]
INFO: INFO 06/29/2026 05:14:04 PM control.py:233 : celery.worker.control Tasks flagged as revoked: docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7
DEBUG: DEBUG 06/29/2026 05:14:04 PM base.py:149 : celery.pool TaskPool: Apply <function fast_trace_task at 0x7fc7845b0d60> (args:('check_for_indexing', '0b6fe3e9-8aaf-4488-a1f9-36f639b58340', {'lang': 'py', 'task': 'check_for_indexing', 'id': '0b6fe3e9-8aaf-4488-a1f9-36f639b58340', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '0b6fe3e9-8aaf-4488-a1f9-36f639b58340', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': "{'tenant_id': 'public'}", 'origin': 'gen41@042454ac5931', 'ignore_result': False, 'replaced_task_nesting': 0, 'stamped_headers': None, 'stamps': {}, 'enqueued_at': 1782724444.9900773, 'properties': {'correlation_id': '0b6fe3e9-8aaf-4488-a1f9-36f639b58340', 'reply_to': 'afbf0be4-e35d-3fda-9cf0-998fb1a21ecd', 'delivery_mode': 2, 'delivery_info': {'exchange': '', 'routing_key': 'celery'}, 'priority': 1, 'body_encoding': 'base64', 'delivery_tag': '6c13a99a-54e0-4514-ba47-52156cdec240'}, 'reply_to': 'afbf0be4-e35d-3fda-9cf0-998fb1a21ecd', 'correlation_id': '0b6fe3e9-8aaf-4488-a1f9-36f639b58340', 'hostname': 'primary@bce5ebc2db34',... kwargs:{})
WARNING: WARNING 06/29/2026 05:14:04 PM tasks.py:862 : [check_for_indexing(0b6fe3e9-8aaf-4488-a1f9-36f639b58340)] check_for_indexing - Starting
DEBUG: DEBUG 06/29/2026 05:14:05 PM tasks.py:787 : [check_for_indexing(0b6fe3e9-8aaf-4488-a1f9-36f639b58340)] _kickoff_indexing_tasks - Not indexing cc_pair_id: 1 search_settings=3, secondary_index_building=False
INFO: 06/29/2026 05:14:05 PM tasks.py 183: Validating active indexing attempts
INFO: INFO 06/29/2026 05:14:05 PM tasks.py:183 : onyx.utils.logger Validating active indexing attempts
INFO: INFO 06/29/2026 05:14:05 PM tasks.py:220 : [check_for_indexing(0b6fe3e9-8aaf-4488-a1f9-36f639b58340)] Initialized heartbeat tracking for attempt 1: counter=458
INFO: INFO 06/29/2026 05:14:05 PM tasks.py:449 : [check_for_indexing(0b6fe3e9-8aaf-4488-a1f9-36f639b58340)] Indexing attempt progress: attempt=1 cc_pair=2 search_settings=3 completed_batches=179 total_batches=179 total_docs=2773 total_failures=0elapsed=65377
INFO: INFO 06/29/2026 05:14:05 PM tasks.py:462 : [check_for_indexing(0b6fe3e9-8aaf-4488-a1f9-36f639b58340)] Indexing attempt 1 has been cancelled
INFO: INFO 06/29/2026 05:14:05 PM tasks.py:1188: [check_for_indexing(0b6fe3e9-8aaf-4488-a1f9-36f639b58340)] check_for_indexing finished: elapsed=0.05s primary=[evaluated=1 created=0 skipped_active=0 skipped_not_found=0 skipped_not_indexable=1 failed=0]
DEBUG: DEBUG 06/29/2026 05:14:05 PM app_base.py:153 : [check_for_indexing(0b6fe3e9-8aaf-4488-a1f9-36f639b58340)] Task check_for_indexing (ID: 0b6fe3e9-8aaf-4488-a1f9-36f639b58340) completed with state: SUCCESS
DEBUG: DEBUG 06/29/2026 05:14:05 PM app_base.py:174 : [check_for_indexing(0b6fe3e9-8aaf-4488-a1f9-36f639b58340)] Task check_for_indexing (ID: 0b6fe3e9-8aaf-4488-a1f9-36f639b58340) completed with state: SUCCESS for tenant_id=public
DEBUG: DEBUG 06/29/2026 05:14:04 PM pidbox.py:98 : kombu.pidbox pidbox received method revoke(task_id='docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7', terminate=False, signal='SIGTERM') [reply_to:None ticket:None]
INFO: INFO 06/29/2026 05:14:04 PM control.py:233 : celery.worker.control Tasks flagged as revoked: docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7
DEBUG: DEBUG 06/29/2026 05:14:04 PM pidbox.py:98 : kombu.pidbox pidbox received method revoke(task_id='docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7', terminate=False, signal='SIGTERM') [reply_to:None ticket:None]
INFO: INFO 06/29/2026 05:14:04 PM control.py:233 : celery.worker.control Tasks flagged as revoked: docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7
DEBUG: DEBUG 06/29/2026 05:14:04 PM pidbox.py:98 : kombu.pidbox pidbox received method revoke(task_id='docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7', terminate=False, signal='SIGTERM') [reply_to:None ticket:None]
INFO: INFO 06/29/2026 05:14:04 PM control.py:233 : celery.worker.control Tasks flagged as revoked: docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7
DEBUG: DEBUG 06/29/2026 05:14:04 PM pidbox.py:98 : kombu.pidbox pidbox received method revoke(task_id='docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7', terminate=False, signal='SIGTERM') [reply_to:None ticket:None]
INFO: INFO 06/29/2026 05:14:04 PM control.py:233 : celery.worker.control Tasks flagged as revoked: docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7
DEBUG: DEBUG 06/29/2026 05:14:04 PM pidbox.py:98 : kombu.pidbox pidbox received method revoke(task_id='docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7', terminate=False, signal='SIGTERM') [reply_to:None ticket:None]
INFO: INFO 06/29/2026 05:14:04 PM control.py:233 : celery.worker.control Tasks flagged as revoked: docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7
DEBUG: DEBUG 06/29/2026 05:14:04 PM pidbox.py:98 : kombu.pidbox pidbox received method revoke(task_id='docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7', terminate=False, signal='SIGTERM') [reply_to:None ticket:None]
INFO: INFO 06/29/2026 05:14:04 PM control.py:233 : celery.worker.control Tasks flagged as revoked: docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7
DEBUG: DEBUG 06/29/2026 05:14:04 PM pidbox.py:98 : kombu.pidbox pidbox received method revoke(task_id='docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7', terminate=False, signal='SIGTERM') [reply_to:None ticket:None]
INFO: INFO 06/29/2026 05:14:04 PM control.py:233 : celery.worker.control Tasks flagged as revoked: docfetching_2_3_84ea1341-6380-416d-b7cf-faa2c23f81b7
WARNING: 06/29/2026 05:14:06 PM supervisord_watchdog.py 56: Key lookup failed: key=onyx:celery:beat:heartbeat lookup_failures=4 max_lookup_failures=5- resume the confluence connector DEBUG: DEBUG 06/29/2026 05:16:47 PM base.py:149 : celery.pool TaskPool: Apply <function fast_trace_task at 0x7fc7845b0d60> (args:('check_for_indexing', '99133ac9-7150-4a8e-940b-3f688c12f488', {'lang': 'py', 'task': 'check_for_indexing', 'id': '99133ac9-7150-4a8e-940b-3f688c12f488', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '99133ac9-7150-4a8e-940b-3f688c12f488', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': "{'tenant_id': 'public'}", 'origin': 'gen41@042454ac5931', 'ignore_result': False, 'replaced_task_nesting': 0, 'stamped_headers': None, 'stamps': {}, 'enqueued_at': 1782724607.3180094, 'properties': {'correlation_id': '99133ac9-7150-4a8e-940b-3f688c12f488', 'reply_to': 'afbf0be4-e35d-3fda-9cf0-998fb1a21ecd', 'delivery_mode': 2, 'delivery_info': {'exchange': '', 'routing_key': 'celery'}, 'priority': 1, 'body_encoding': 'base64', 'delivery_tag': '4663f44b-d936-4c3b-9ef6-fa9c71d3f7cd'}, 'reply_to': 'afbf0be4-e35d-3fda-9cf0-998fb1a21ecd', 'correlation_id': '99133ac9-7150-4a8e-940b-3f688c12f488', 'hostname': 'primary@bce5ebc2db34',... kwargs:{})
WARNING: WARNING 06/29/2026 05:16:47 PM tasks.py:862 : [check_for_indexing(99133ac9-7150-4a8e-940b-3f688c12f488)] check_for_indexing - Starting
DEBUG: DEBUG 06/29/2026 05:16:47 PM tasks.py:787 : [check_for_indexing(99133ac9-7150-4a8e-940b-3f688c12f488)] _kickoff_indexing_tasks - Not indexing cc_pair_id: 1 search_settings=3, secondary_index_building=False
DEBUG: DEBUG 06/29/2026 05:16:47 PM tasks.py:787 : [check_for_indexing(99133ac9-7150-4a8e-940b-3f688c12f488)] _kickoff_indexing_tasks - Not indexing cc_pair_id: 2 search_settings=3, secondary_index_building=False
INFO: 06/29/2026 05:16:47 PM tasks.py 183: Validating active indexing attempts
INFO: INFO 06/29/2026 05:16:47 PM tasks.py:183 : onyx.utils.logger Validating active indexing attempts
INFO: INFO 06/29/2026 05:16:47 PM tasks.py:1188: [check_for_indexing(99133ac9-7150-4a8e-940b-3f688c12f488)] check_for_indexing finished: elapsed=0.02s primary=[evaluated=2 created=0 skipped_active=0 skipped_not_found=0 skipped_not_indexable=2 failed=0]
DEBUG: DEBUG 06/29/2026 05:16:47 PM app_base.py:153 : [check_for_indexing(99133ac9-7150-4a8e-940b-3f688c12f488)] Task check_for_indexing (ID: 99133ac9-7150-4a8e-940b-3f688c12f488) completed with state: SUCCESS
DEBUG: DEBUG 06/29/2026 05:16:47 PM app_base.py:174 : [check_for_indexing(99133ac9-7150-4a8e-940b-3f688c12f488)] Task check_for_indexing (ID: 99133ac9-7150-4a8e-940b-3f688c12f488) completed with state: SUCCESS for tenant_id=public
INFO: INFO 06/29/2026 05:16:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-user-file-processing-public (check_for_user_file_processing)
DEBUG: DEBUG 06/29/2026 05:16:47 PM beat.py:287 : celery.beat check_for_user_file_processing sent. id->3022272a-b9bd-4d0a-a21e-16946d1104c1
INFO: INFO 06/29/2026 05:16:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-user-file-project-sync-public (check_for_user_file_project_sync)
DEBUG: DEBUG 06/29/2026 05:16:47 PM beat.py:287 : celery.beat check_for_user_file_project_sync sent. id->eb474480-89b4-410b-ae7a-135aa4376edd
INFO: INFO 06/29/2026 05:16:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-user-file-delete-public (check_for_user_file_delete)
DEBUG: DEBUG 06/29/2026 05:16:47 PM beat.py:287 : celery.beat check_for_user_file_delete sent. id->897e96c9-3765-47b4-94c7-edeac0eec71d
INFO: INFO 06/29/2026 05:16:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-connector-deletion-public (check_for_connector_deletion_task)
DEBUG: DEBUG 06/29/2026 05:16:47 PM beat.py:287 : celery.beat check_for_connector_deletion_task sent. id->a4ccdbc0-58e5-462b-84c4-1319eab83ace
INFO: INFO 06/29/2026 05:16:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-vespa-sync-public (check_for_vespa_sync_task)
DEBUG: DEBUG 06/29/2026 05:16:47 PM beat.py:287 : celery.beat check_for_vespa_sync_task sent. id->458c229c-e19e-4813-be1b-add484ca66a6
INFO: INFO 06/29/2026 05:16:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-pruning-public (check_for_pruning)
DEBUG: DEBUG 06/29/2026 05:16:47 PM beat.py:287 : celery.beat check_for_pruning sent. id->9ae8b383-51f9-42b5-9098-21c4d30eaef7
INFO: INFO 06/29/2026 05:16:47 PM beat.py:279 : celery.beat Scheduler: Sending due task monitor-celery-queues-public (monitor_celery_queues)
DEBUG: DEBUG 06/29/2026 05:16:47 PM beat.py:287 : celery.beat monitor_celery_queues sent. id->615715fb-dd93-4e87-9ed4-6f66d223d120
DEBUG: DEBUG 06/29/2026 05:16:47 PM beat.py:646 : celery.beat beat: Waking up in 4.98 seconds.
INFO: INFO 06/29/2026 05:16:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_user_file_processing[3022272a-b9bd-4d0a-a21e-16946d1104c1]
INFO: INFO 06/29/2026 05:16:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_user_file_project_sync[eb474480-89b4-410b-ae7a-135aa4376edd]
INFO: INFO 06/29/2026 05:16:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_user_file_delete[897e96c9-3765-47b4-94c7-edeac0eec71d]
INFO: INFO 06/29/2026 05:16:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_connector_deletion_task[a4ccdbc0-58e5-462b-84c4-1319eab83ace]
INFO: INFO 06/29/2026 05:16:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_vespa_sync_task[458c229c-e19e-4813-be1b-add484ca66a6]
INFO: INFO 06/29/2026 05:16:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_pruning[9ae8b383-51f9-42b5-9098-21c4d30eaef7]
INFO: INFO 06/29/2026 05:16:47 PM request.py:496 : celery.worker.request Discarding revoked task: monitor_celery_queues[615715fb-dd93-4e87-9ed4-6f66d223d120]
INFO: INFO 06/29/2026 05:16:52 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-indexing-public (check_for_indexing)
DEBUG: DEBUG 06/29/2026 05:16:52 PM beat.py:287 : celery.beat check_for_indexing sent. id->85f8f63b-45f0-4c06-b131-f421bf40c43a
DEBUG: DEBUG 06/29/2026 05:16:52 PM beat.py:646 : celery.beat beat: Waking up in 4.99 seconds.
INFO: INFO 06/29/2026 05:16:52 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_indexing[85f8f63b-45f0-4c06-b131-f421bf40c43a]
DEBUG: 06/29/2026 05:16:56 PM listener.py 510: Sending heartbeats for 0 active tenants
DEBUG: 06/29/2026 05:16:56 PM listener.py 205: Sent heartbeats for 0 active tenants- re-index the confluence connector DEBUG: DEBUG 06/29/2026 05:18:31 PM base.py:149 : celery.pool TaskPool: Apply <function fast_trace_task at 0x7fc7845b0d60> (args:('check_for_indexing', '201e4838-c63a-4296-89db-4036d442b08b', {'lang': 'py', 'task': 'check_for_indexing', 'id': '201e4838-c63a-4296-89db-4036d442b08b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '201e4838-c63a-4296-89db-4036d442b08b', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': "{'tenant_id': 'public'}", 'origin': 'gen41@042454ac5931', 'ignore_result': False, 'replaced_task_nesting': 0, 'stamped_headers': None, 'stamps': {}, 'enqueued_at': 1782724711.6730945, 'properties': {'correlation_id': '201e4838-c63a-4296-89db-4036d442b08b', 'reply_to': 'e3ebab82-4bb7-36ad-abb6-c3f4783a7921', 'delivery_mode': 2, 'delivery_info': {'exchange': '', 'routing_key': 'celery'}, 'priority': 1, 'body_encoding': 'base64', 'delivery_tag': 'a6d2b70d-f0ee-4fba-8f9a-3be071902a65'}, 'reply_to': 'e3ebab82-4bb7-36ad-abb6-c3f4783a7921', 'correlation_id': '201e4838-c63a-4296-89db-4036d442b08b', 'hostname': 'primary@bce5ebc2db34',... kwargs:{})
WARNING: WARNING 06/29/2026 05:18:31 PM tasks.py:862 : [check_for_indexing(201e4838-c63a-4296-89db-4036d442b08b)] check_for_indexing - Starting
DEBUG: DEBUG 06/29/2026 05:18:31 PM tasks.py:787 : [check_for_indexing(201e4838-c63a-4296-89db-4036d442b08b)] _kickoff_indexing_tasks - Not indexing cc_pair_id: 1 search_settings=3, secondary_index_building=False
DEBUG: DEBUG 06/29/2026 05:18:31 PM tasks.py:795 : [check_for_indexing(201e4838-c63a-4296-89db-4036d442b08b)] _kickoff_indexing_tasks - Will index cc_pair_id: 2 search_settings=3, secondary_index_building=False
INFO: INFO 06/29/2026 05:18:31 PM tasks.py:807 : [check_for_indexing(201e4838-c63a-4296-89db-4036d442b08b)] _kickoff_indexing_tasks - Connector indexing manual trigger detected: cc_pair=2 search_settings=3 indexing_mode=IndexingMode.UPDATE
INFO: 06/29/2026 05:18:31 PM indexing_coordination.py 85: Created Index Attempt: cc_pair=2 search_settings=3 attempt_id=2 celery_task_id=docfetching_2_3_9f2e2b7f-e8c9-4a29-a62d-212114405b8c
INFO: INFO 06/29/2026 05:18:31 PM indexing_coordination.py:85 : onyx.utils.logger Created Index Attempt: cc_pair=2 search_settings=3 attempt_id=2 celery_task_id=docfetching_2_3_9f2e2b7f-e8c9-4a29-a62d-212114405b8c
INFO: INFO 06/29/2026 05:18:31 PM task_creation_utils.py:100 : [check_for_indexing(201e4838-c63a-4296-89db-4036d442b08b)] Created docfetching task: cc_pair=2 search_settings=3 attempt_id=2 celery_task_id=docfetching_2_3_9f2e2b7f-e8c9-4a29-a62d-212114405b8c
INFO: INFO 06/29/2026 05:18:31 PM tasks.py:829 : [check_for_indexing(201e4838-c63a-4296-89db-4036d442b08b)] Connector indexing queued: index_attempt=2 cc_pair=2 search_settings=3
INFO: 06/29/2026 05:18:31 PM tasks.py 183: Validating active indexing attempts
INFO: INFO 06/29/2026 05:18:31 PM tasks.py:183 : onyx.utils.logger Validating active indexing attempts
INFO: INFO 06/29/2026 05:18:31 PM tasks.py:449 : [check_for_indexing(201e4838-c63a-4296-89db-4036d442b08b)] Indexing attempt progress: attempt=2 cc_pair=2 search_settings=3 completed_batches=0 total_batches=? total_docs=0 total_failures=0elapsed=0
INFO: 06/29/2026 05:18:31 PM tasks.py 523: Checking for indexing completion: attempt=2 tenant=public
INFO: INFO 06/29/2026 05:18:31 PM tasks.py:523 : onyx.utils.logger Checking for indexing completion: attempt=2 tenant=public
INFO: 06/29/2026 05:18:31 PM tasks.py 536: Indexing status: indexing_completed=False batches_processed=0/? total_docs=0 total_chunks=0 total_failures=0
INFO: INFO 06/29/2026 05:18:31 PM tasks.py:536 : onyx.utils.logger Indexing status: indexing_completed=False batches_processed=0/? total_docs=0 total_chunks=0 total_failures=0
INFO: INFO 06/29/2026 05:18:31 PM tasks.py:1188: [check_for_indexing(201e4838-c63a-4296-89db-4036d442b08b)] check_for_indexing finished: elapsed=0.04s primary=[evaluated=2 created=1 skipped_active=0 skipped_not_found=0 skipped_not_indexable=1 failed=0]
DEBUG: DEBUG 06/29/2026 05:18:31 PM app_base.py:153 : [check_for_indexing(201e4838-c63a-4296-89db-4036d442b08b)] Task check_for_indexing (ID: 201e4838-c63a-4296-89db-4036d442b08b) completed with state: SUCCESS
DEBUG: DEBUG 06/29/2026 05:18:31 PM app_base.py:174 : [check_for_indexing(201e4838-c63a-4296-89db-4036d442b08b)] Task check_for_indexing (ID: 201e4838-c63a-4296-89db-4036d442b08b) completed with state: SUCCESS for tenant_id=public
DEBUG: DEBUG 06/29/2026 05:18:31 PM base.py:149 : celery.pool TaskPool: Apply <function fast_trace_task at 0x7fe6aae702c0> (args:('connector_doc_fetching_task', 'docfetching_2_3_9f2e2b7f-e8c9-4a29-a62d-212114405b8c', {'lang': 'py', 'task': 'connector_doc_fetching_task', 'id': 'docfetching_2_3_9f2e2b7f-e8c9-4a29-a62d-212114405b8c', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '201e4838-c63a-4296-89db-4036d442b08b', 'parent_id': '201e4838-c63a-4296-89db-4036d442b08b', 'argsrepr': '()', 'kwargsrepr': "{'index_attempt_id': 2, 'cc_pair_id': 2, 'search_settings_id': 3, 'tenant_id': 'public'}", 'origin': 'gen13@bce5ebc2db34', 'ignore_result': False, 'replaced_task_nesting': 0, 'stamped_headers': None, 'stamps': {}, 'enqueued_at': 1782724711.7071092, 'properties': {'correlation_id': 'docfetching_2_3_9f2e2b7f-e8c9-4a29-a62d-212114405b8c', 'reply_to': 'e087d50c-c7b7-37d1-b609-979f31e747af', 'delivery_mode': 2, 'delivery_info': {'exchange': '', 'routing_key': 'connector_doc_fetching'}, 'priority': 1, 'body_encoding': 'base64', 'delivery_tag':... kwargs:{})
INFO: INFO 06/29/2026 05:18:32 PM tasks.py:401 : [connector_doc_fetching_task(docfetching_2_3_9f2e2b7f-e8c9-4a29-a62d-212114405b8c)] Indexing watchdog - starting: tenant_id=public attempt=2 cc_pair=2 search_settings=3 mp_start_method=spawn
INFO: INFO 06/29/2026 05:18:32 PM tasks.py:412 : [connector_doc_fetching_task(docfetching_2_3_9f2e2b7f-e8c9-4a29-a62d-212114405b8c)] submitting docfetching_task with tenant_id=public
INFO: INFO 06/29/2026 05:18:32 PM tasks.py:456 : [connector_doc_fetching_task(docfetching_2_3_9f2e2b7f-e8c9-4a29-a62d-212114405b8c)] Indexing watchdog - spawn succeeded: tenant_id=public attempt=2 cc_pair=2 search_settings=3 pid=231
DEBUG: 06/29/2026 05:18:33 PM app_base.py 76: Sentry DSN not provided, skipping Sentry initialization
INFO: 06/29/2026 05:18:34 PM job_client.py 60: Initializing spawned worker child process.
INFO: 06/29/2026 05:18:34 PM sql_engine.py 264: Creating engine with kwargs: {'connect_args': {'keepalives': 1, 'keepalives_idle': 30, 'keepalives_interval': 10, 'keepalives_count': 5}, 'pool_size': 4, 'max_overflow': 12, 'pool_pre_ping': True, 'pool_recycle': 60}
DEBUG: 06/29/2026 05:18:34 PM tasks.py 150: Sentry DSN not provided, skipping Sentry initialization
INFO: 06/29/2026 05:18:34 PM tasks.py 152: Indexing spawned task starting: attempt=2 tenant=public cc_pair=2 search_settings=3
INFO: 06/29/2026 05:18:34 PM tasks.py 97: docfetching_task - IndexAttempt verified: attempt_id=2 cc_pair=2 search_settings=3
INFO: 06/29/2026 05:18:34 PM tasks.py 210: Indexing spawned task running entrypoint: attempt=2 tenant=public cc_pair=2 search_settings=3
INFO: 06/29/2026 05:18:34 PM run_docfetching.py 336: [Index Attempt: 2] [CC Pair: 2] Docfetching starting: connector='ConfTest' config='{'space': 'test', 'is_cloud': False, 'wiki_base': 'https://10.0.0.1/', 'scoped_token': False}' credentials='1'
INFO: 06/29/2026 05:18:34 PM run_docfetching.py 400: [Index Attempt: 2] [CC Pair: 2] Document extraction starting: attempt=2 cc_pair=2 search_settings=3 tenant=public
DEBUG: 06/29/2026 05:18:35 PM variable_functionality.py 93: [Index Attempt: 2] [CC Pair: 2] Fetching versioned implementation for onyx.utils.encryption._decrypt_bytes
WARNING: 06/29/2026 05:18:35 PM encryption.py 29: [Index Attempt: 2] [CC Pair: 2] MIT version of Onyx does not support decryption of secrets.
INFO: 06/29/2026 05:18:35 PM onyx_confluence.py 485: [Index Attempt: 2] [CC Pair: 2] Connecting to Confluence with Personal Access Token as user: sw3
INFO: 06/29/2026 05:18:36 PM onyx_confluence.py 451: [Index Attempt: 2] [CC Pair: 2] Confluence probe succeeded.
INFO: 06/29/2026 05:18:36 PM onyx_confluence.py 485: [Index Attempt: 2] [CC Pair: 2] Connecting to Confluence with Personal Access Token as user: sw3
WARNING: 06/29/2026 05:18:36 PM encryption.py 29: [Index Attempt: 2] [CC Pair: 2] MIT version of Onyx does not support decryption of secrets.
INFO: 06/29/2026 05:18:36 PM onyx_confluence.py 485: [Index Attempt: 2] [CC Pair: 2] Connecting to Confluence with Personal Access Token as user: sw3
INFO: INFO 06/29/2026 05:18:37 PM memory_monitoring.py:70 : memory_monitoring PROCESS_MEMORY process_name=indexing_worker pid=231 rss_mb=236.53 vms_mb=1636.54 cpu=0.00 cc_pair_id=2 search_settings_id=3 index_attempt_id=2
INFO: INFO 06/29/2026 05:18:37 PM beat.py:279 : celery.beat Scheduler: Sending due task dispatch-due-scheduled-tasks-public (scheduled_tasks_dispatch_due)
DEBUG: DEBUG 06/29/2026 05:18:37 PM beat.py:287 : celery.beat scheduled_tasks_dispatch_due sent. id->c58dedda-1d72-48f2-80fc-10543116649c
INFO: INFO 06/29/2026 05:18:37 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-indexing-public (check_for_indexing)
DEBUG: DEBUG 06/29/2026 05:18:37 PM beat.py:287 : celery.beat check_for_indexing sent. id->f77ee425-a93d-4441-90b6-515e9b632979
INFO: INFO 06/29/2026 05:18:37 PM beat.py:279 : celery.beat Scheduler: Sending due task monitor-celery-queues-public (monitor_celery_queues)
DEBUG: DEBUG 06/29/2026 05:18:37 PM beat.py:287 : celery.beat monitor_celery_queues sent. id->a4dc1139-d23c-4d1a-81fb-e69253aff492
DEBUG: DEBUG 06/29/2026 05:18:37 PM beat.py:646 : celery.beat beat: Waking up in 9.98 seconds.
INFO: INFO 06/29/2026 05:18:37 PM request.py:496 : celery.worker.request Discarding revoked task: scheduled_tasks_dispatch_due[c58dedda-1d72-48f2-80fc-10543116649c]
INFO: INFO 06/29/2026 05:18:37 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_indexing[f77ee425-a93d-4441-90b6-515e9b632979]
INFO: INFO 06/29/2026 05:18:37 PM request.py:496 : celery.worker.request Discarding revoked task: monitor_celery_queues[a4dc1139-d23c-4d1a-81fb-e69253aff492]
INFO: 06/29/2026 05:18:37 PM onyx_confluence.py 451: [Index Attempt: 2] [CC Pair: 2] Confluence probe succeeded.
INFO: 06/29/2026 05:18:37 PM onyx_confluence.py 485: [Index Attempt: 2] [CC Pair: 2] Connecting to Confluence with Personal Access Token as user: sw3
INFO: 06/29/2026 05:18:37 PM connector.py 227: [Index Attempt: 2] [CC Pair: 2] Setting allow_images to True.
INFO: 06/29/2026 05:18:38 PM run_docfetching.py 559: [Index Attempt: 2] [CC Pair: 2] Getting latest valid checkpoint for index attempt 2
INFO: 06/29/2026 05:18:38 PM checkpointing_utils.py 161: [Index Attempt: 2] [CC Pair: 2] Using checkpoint from previous failed attempt with ID 1. Previous checkpoint: {"has_more":false,"next_page_url":"/rest/api/content/search?expand=body.storage.value%2Cversion%2Cspace%2Cmetadata.labels%2Chistory.lastUpdated%2Cancestors&body-format=atlas_doc_format&limit=16&start=240&cql=type%3Dpage%20and%20space%3D%27SD3%27%20and%20lastmodified%20%3C%3D%20%272026-06-24%2023%3A04%27%20order%20by%20lastmodified%20asc"}
INFO: 06/29/2026 05:18:38 PM run_docfetching.py 1044: [Index Attempt: 2] [CC Pair: 2] Starting from batch 179 due to re-issued batches: [], completed batches: 179
INFO: 06/29/2026 05:18:38 PM redis_hierarchy.py 581: [Index Attempt: 2] [CC Pair: 2] Ensured SOURCE node exists and cached for confluence: id=12
INFO: 06/29/2026 05:18:38 PM run_docfetching.py 824: [Index Attempt: 2] [CC Pair: 2] Document extraction completed: attempt=2 batches_queued=0 elapsed=3.81s
INFO: 06/29/2026 05:18:38 PM indexing_coordination.py 148: [Index Attempt: 2] [CC Pair: 2] Set total batches: attempt=2 total=179
INFO: 06/29/2026 05:18:38 PM run_docfetching.py 372: [Index Attempt: 2] [CC Pair: 2] Docfetching finished: connector='ConfTest' config='{'space': 'test', 'is_cloud': False, 'wiki_base': 'https://10.0.0.1/', 'scoped_token': False}' credentials='1'
INFO: 06/29/2026 05:18:38 PM tasks.py 256: Indexing spawned task finished: attempt=2 cc_pair=2 search_settings=3
DEBUG: 06/29/2026 05:18:41 PM listener.py 510: Sending heartbeats for 0 active tenants
DEBUG: 06/29/2026 05:18:41 PM listener.py 205: Sent heartbeats for 0 active tenants
INFO: INFO 06/29/2026 05:18:42 PM tasks.py:770 : [connector_doc_fetching_task(docfetching_2_3_9f2e2b7f-e8c9-4a29-a62d-212114405b8c)] Indexing watchdog - finished: tenant_id=public attempt=2 cc_pair=2 search_settings=3 source=confluence status=succeeded exit_code=0 elapsed=10.13s
DEBUG: DEBUG 06/29/2026 05:18:42 PM app_base.py:153 : [connector_doc_fetching_task(docfetching_2_3_9f2e2b7f-e8c9-4a29-a62d-212114405b8c)] Task connector_doc_fetching_task (ID: docfetching_2_3_9f2e2b7f-e8c9-4a29-a62d-212114405b8c) completed with state: SUCCESS
DEBUG: DEBUG 06/29/2026 05:18:42 PM app_base.py:174 : [connector_doc_fetching_task(docfetching_2_3_9f2e2b7f-e8c9-4a29-a62d-212114405b8c)] Task connector_doc_fetching_task (ID: docfetching_2_3_9f2e2b7f-e8c9-4a29-a62d-212114405b8c) completed with state: SUCCESS for tenant_id=public
DEBUG: DEBUG 06/29/2026 05:18:42 PM primary.py:310 : Reacquiring primary worker lock.
INFO: INFO 06/29/2026 05:18:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-user-file-processing-public (check_for_user_file_processing)
DEBUG: DEBUG 06/29/2026 05:18:47 PM beat.py:287 : celery.beat check_for_user_file_processing sent. id->c2052051-ebb6-4cfa-a10e-c435b0ad3c50
INFO: INFO 06/29/2026 05:18:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-user-file-project-sync-public (check_for_user_file_project_sync)
DEBUG: DEBUG 06/29/2026 05:18:47 PM beat.py:287 : celery.beat check_for_user_file_project_sync sent. id->7315d076-cb48-4dbe-9a77-55651b8b0f96
INFO: INFO 06/29/2026 05:18:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-user-file-delete-public (check_for_user_file_delete)
DEBUG: DEBUG 06/29/2026 05:18:47 PM beat.py:287 : celery.beat check_for_user_file_delete sent. id->209016cc-c373-4ecc-92e5-20811249ac3e
INFO: INFO 06/29/2026 05:18:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-connector-deletion-public (check_for_connector_deletion_task)
DEBUG: DEBUG 06/29/2026 05:18:47 PM beat.py:287 : celery.beat check_for_connector_deletion_task sent. id->cee74a7a-b46f-4bd6-8477-4cf7c881c437
INFO: INFO 06/29/2026 05:18:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-vespa-sync-public (check_for_vespa_sync_task)
DEBUG: DEBUG 06/29/2026 05:18:47 PM beat.py:287 : celery.beat check_for_vespa_sync_task sent. id->ca79b593-60cc-4d47-a2d8-8ee84f29379a
INFO: INFO 06/29/2026 05:18:47 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-pruning-public (check_for_pruning)
DEBUG: DEBUG 06/29/2026 05:18:47 PM beat.py:287 : celery.beat check_for_pruning sent. id->9462584e-5df9-48f1-9416-69e7701685d7
INFO: INFO 06/29/2026 05:18:47 PM beat.py:279 : celery.beat Scheduler: Sending due task monitor-celery-queues-public (monitor_celery_queues)
DEBUG: DEBUG 06/29/2026 05:18:47 PM beat.py:287 : celery.beat monitor_celery_queues sent. id->64b8acc4-38d6-4b37-9626-6e294a39aa3d
DEBUG: DEBUG 06/29/2026 05:18:47 PM beat.py:646 : celery.beat beat: Waking up in 4.98 seconds.
INFO: INFO 06/29/2026 05:18:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_user_file_processing[c2052051-ebb6-4cfa-a10e-c435b0ad3c50]
INFO: INFO 06/29/2026 05:18:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_user_file_project_sync[7315d076-cb48-4dbe-9a77-55651b8b0f96]
INFO: INFO 06/29/2026 05:18:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_user_file_delete[209016cc-c373-4ecc-92e5-20811249ac3e]
INFO: INFO 06/29/2026 05:18:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_connector_deletion_task[cee74a7a-b46f-4bd6-8477-4cf7c881c437]
INFO: INFO 06/29/2026 05:18:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_vespa_sync_task[ca79b593-60cc-4d47-a2d8-8ee84f29379a]
INFO: INFO 06/29/2026 05:18:47 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_pruning[9462584e-5df9-48f1-9416-69e7701685d7]
INFO: INFO 06/29/2026 05:18:47 PM request.py:496 : celery.worker.request Discarding revoked task: monitor_celery_queues[64b8acc4-38d6-4b37-9626-6e294a39aa3d]
INFO: INFO 06/29/2026 05:18:52 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-indexing-public (check_for_indexing)
DEBUG: DEBUG 06/29/2026 05:18:52 PM beat.py:287 : celery.beat check_for_indexing sent. id->69c4c8b7-5d4e-47f0-b2e3-ac758e788549
DEBUG: DEBUG 06/29/2026 05:18:52 PM beat.py:646 : celery.beat beat: Waking up in 4.99 seconds.
INFO: INFO 06/29/2026 05:18:52 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_indexing[69c4c8b7-5d4e-47f0-b2e3-ac758e788549]
DEBUG: 06/29/2026 05:18:56 PM listener.py 510: Sending heartbeats for 0 active tenants
DEBUG: 06/29/2026 05:18:56 PM listener.py 205: Sent heartbeats for 0 active tenants
INFO: INFO 06/29/2026 05:18:57 PM beat.py:279 : celery.beat Scheduler: Sending due task monitor-celery-queues-public (monitor_celery_queues)
DEBUG: DEBUG 06/29/2026 05:18:57 PM beat.py:287 : celery.beat monitor_celery_queues sent. id->263a030a-b094-46e9-8cb9-61688190a501
DEBUG: DEBUG 06/29/2026 05:18:57 PM beat.py:646 : celery.beat beat: Waking up in 9.98 seconds.
DEBUG: DEBUG 06/29/2026 05:18:57 PM primary.py:310 : Reacquiring primary worker lock.
INFO: INFO 06/29/2026 05:18:57 PM request.py:496 : celery.worker.request Discarding revoked task: monitor_celery_queues[263a030a-b094-46e9-8cb9-61688190a501]
WARNING: 06/29/2026 05:19:06 PM supervisord_watchdog.py 67: Key lookup failed: key=onyx:celery:beat:heartbeat lookup_failures=9 max_lookup_failures=5 elapsed=540.01 elapsed_threshold=900
INFO: INFO 06/29/2026 05:19:07 PM beat.py:279 : celery.beat Scheduler: Sending due task celery-beat-heartbeat-public (celery_beat_heartbeat)
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:287 : celery.beat celery_beat_heartbeat sent. id->2b5615af-7736-4305-8c73-75bf50cee91b
INFO: INFO 06/29/2026 05:19:07 PM beat.py:279 : celery.beat Scheduler: Sending due task cleanup-idle-sandboxes-public (cleanup_idle_sandboxes)
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:287 : celery.beat cleanup_idle_sandboxes sent. id->5e8bc6f5-d359-4253-9c14-cbebb315144b
INFO: INFO 06/29/2026 05:19:07 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-user-file-processing-public (check_for_user_file_processing)
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:287 : celery.beat check_for_user_file_processing sent. id->d8f9d4de-47e6-4b9d-8401-84b9e3567604
INFO: INFO 06/29/2026 05:19:07 PM beat.py:279 : celery.beat Scheduler: Sending due task dispatch-due-scheduled-tasks-public (scheduled_tasks_dispatch_due)
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:287 : celery.beat scheduled_tasks_dispatch_due sent. id->fb7e73d1-8d6e-4b1e-bd82-9a3ffb1c1400
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:68 : onyx.background.celery.apps.beat Reload interval reached, initiating task update
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:159 : onyx.background.celery.apps.beat _try_updating_schedule starting
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:162 : onyx.background.celery.apps.beat Found 1 IDs
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for monitor-celery-queues-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task monitor-celery-queues-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900, 'queue': 'monitoring'}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for monitor-process-memory-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task monitor-process-memory-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900, 'queue': 'monitoring'}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for celery-beat-heartbeat-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task celery-beat-heartbeat-public: {'priority': <OnyxCeleryPriority.HIGHEST: 0>, 'expires': 900, 'queue': 'celery'}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-user-file-processing-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-user-file-processing-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-user-file-project-sync-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-user-file-project-sync-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-user-file-delete-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-user-file-delete-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-indexing-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-indexing-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-checkpoint-cleanup-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-checkpoint-cleanup-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-index-attempt-cleanup-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-index-attempt-cleanup-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-connector-deletion-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-connector-deletion-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-vespa-sync-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-vespa-sync-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-pruning-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-pruning-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 900}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-hierarchy-fetching-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-hierarchy-fetching-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for monitor-background-processes-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task monitor-background-processes-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900, 'queue': 'monitoring'}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for dispatch-due-scheduled-tasks-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task dispatch-due-scheduled-tasks-public: {'priority': <OnyxCeleryPriority.MEDIUM: 2>, 'expires': 60, 'queue': 'celery'}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for cleanup-stuck-scheduled-runs-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task cleanup-stuck-scheduled-runs-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 3600, 'queue': 'celery'}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for cleanup-idle-sandboxes-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task cleanup-idle-sandboxes-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900, 'queue': 'sandbox'}
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:135 : onyx.background.celery.apps.beat Creating task configuration for check-for-auto-llm-update-public
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:144 : onyx.background.celery.apps.beat Adding options to task check-for-auto-llm-update-public: {'priority': <OnyxCeleryPriority.LOW: 3>, 'expires': 900}
INFO: INFO 06/29/2026 05:19:07 PM beat.py:191 : onyx.background.celery.apps.beat _try_updating_schedule - Schedule unchanged: tasks=18 beat_multiplier=8.0
INFO: INFO 06/29/2026 05:19:07 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-user-file-project-sync-public (check_for_user_file_project_sync)
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:287 : celery.beat check_for_user_file_project_sync sent. id->bba24639-2318-4e5b-8bfc-63bdb8e2e632
INFO: INFO 06/29/2026 05:19:07 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-indexing-public (check_for_indexing)
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:287 : celery.beat check_for_indexing sent. id->1108fd2c-2ff0-4550-9c18-74542764a572
INFO: INFO 06/29/2026 05:19:07 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-user-file-delete-public (check_for_user_file_delete)
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:287 : celery.beat check_for_user_file_delete sent. id->67d7a9f0-ca5e-4bbc-b087-e823313b26ad
INFO: INFO 06/29/2026 05:19:07 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-connector-deletion-public (check_for_connector_deletion_task)
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:287 : celery.beat check_for_connector_deletion_task sent. id->74970efa-39d8-4523-8443-06b943c1710f
INFO: INFO 06/29/2026 05:19:07 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-vespa-sync-public (check_for_vespa_sync_task)
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:287 : celery.beat check_for_vespa_sync_task sent. id->4de3efc9-0212-473f-b295-c98ca8edd30a
INFO: INFO 06/29/2026 05:19:07 PM beat.py:279 : celery.beat Scheduler: Sending due task check-for-pruning-public (check_for_pruning)
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:287 : celery.beat check_for_pruning sent. id->14b104c6-5eb7-4a77-a77f-4ee0c001bf29
INFO: INFO 06/29/2026 05:19:07 PM beat.py:279 : celery.beat Scheduler: Sending due task monitor-celery-queues-public (monitor_celery_queues)
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:287 : celery.beat monitor_celery_queues sent. id->8d491d17-b5aa-4220-9f17-6f1e167a5bd9
DEBUG: DEBUG 06/29/2026 05:19:07 PM beat.py:646 : celery.beat beat: Waking up in 9.99 seconds.
INFO: INFO 06/29/2026 05:19:07 PM request.py:496 : celery.worker.request Discarding revoked task: celery_beat_heartbeat[2b5615af-7736-4305-8c73-75bf50cee91b]
INFO: INFO 06/29/2026 05:19:07 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_user_file_processing[d8f9d4de-47e6-4b9d-8401-84b9e3567604]
INFO: INFO 06/29/2026 05:19:07 PM request.py:496 : celery.worker.request Discarding revoked task: scheduled_tasks_dispatch_due[fb7e73d1-8d6e-4b1e-bd82-9a3ffb1c1400]
INFO: INFO 06/29/2026 05:19:07 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_user_file_project_sync[bba24639-2318-4e5b-8bfc-63bdb8e2e632]
INFO: INFO 06/29/2026 05:19:07 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_indexing[1108fd2c-2ff0-4550-9c18-74542764a572]
INFO: INFO 06/29/2026 05:19:07 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_user_file_delete[67d7a9f0-ca5e-4bbc-b087-e823313b26ad]
INFO: INFO 06/29/2026 05:19:07 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_connector_deletion_task[74970efa-39d8-4523-8443-06b943c1710f]
INFO: INFO 06/29/2026 05:19:07 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_vespa_sync_task[4de3efc9-0212-473f-b295-c98ca8edd30a]
INFO: INFO 06/29/2026 05:19:07 PM request.py:496 : celery.worker.request Discarding revoked task: check_for_pruning[14b104c6-5eb7-4a77-a77f-4ee0c001bf29]
INFO: INFO 06/29/2026 05:19:07 PM request.py:496 : celery.worker.request Discarding revoked task: cleanup_idle_sandboxes[5e8bc6f5-d359-4253-9c14-cbebb315144b]
INFO: INFO 06/29/2026 05:19:07 PM request.py:496 : celery.worker.request Discarding revoked task: monitor_celery_queues[8d491d17-b5aa-4220-9f17-6f1e167a5bd9]
DEBUG: 06/29/2026 05:19:11 PM listener.py 510: Sending heartbeats for 0 active tenants
DEBUG: 06/29/2026 05:19:11 PM listener.py 205: Sent heartbeats for 0 active tenants
DEBUG: 06/29/2026 05:19:11 PM listener.py 341: Acquired lock for tenant public.
DEBUG: 06/29/2026 05:19:11 PM listener.py 395: No Slack bots for tenant public; lock released (if held).
DEBUG: 06/29/2026 05:19:11 PM listener.py 186: Current active tenants with Slack bots: 0
DEBUG: DEBUG 06/29/2026 05:19:12 PM primary.py:310 : Reacquiring primary worker lock. |
0 replies
|
I have identified the root cause of the issue today. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hello,
I have Onyx version v3.3.x and v4.1.x deployed using docker.
Everything is working fine as expected, including
Then an issue(Indexing stuck at Indexing for all Confluence connectors — Docfetching finished #11379 ) occurs where the status of the connector is remained as Indexing/In Progress even the loggings of background instance shows indexing is completed.
It moves to Finished only when I add another connector and kick-off the indexing on the new connector.
But the new connector is still remained as Indexing/In Progress after the indexing is completed.
Another issue(#11049 ) when changing the default embedding model is similar.
"Switch before re-indexing" is applied after changing the embedding model to embeddinggemma which is hosted on LiteLLM.
The logging of background instance or api_server instance shows no issue.
Can someone guide/advise me where the problem is or how to troubleshoot?
All reactions