link runpod views pipeline #221
Conversation
WalkthroughThis change introduces duplicate workflow operations in Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Backend (task_create_view)
participant Directus
participant TopicModelerAPI
User->>Backend (task_create_view): Request with project_analysis_run_id, user_query, context, language
Backend->>Directus: Get project_analysis_run by ID
Directus-->>Backend: Return project_id
Backend->>Directus: Get project by project_id (fetch conversations and segment IDs)
Directus-->>Backend: Return nested conversation segment IDs
Backend->>Backend: Flatten & deduplicate segment IDs, build prompt
Backend->>TopicModelerAPI: POST / (segment IDs, prompt, language, run_id)
TopicModelerAPI-->>Backend: Response (not handled in this flow)
Possibly related PRs
Suggested labels
Suggested reviewers
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Bug: Unchecked Query Results Lead to IndexError
IndexError occurs in task_create_view when accessing response[0]["project_id"] or segments[0]["conversations"] without checking if the client.get_items queries for project_analysis_run or project return any results.
echo/server/dembrane/tasks.py#L443-L444
echo/echo/server/dembrane/tasks.py
Lines 443 to 444 in 5237ccd
Bug: Duplicate Workflow Operations Cause Sync Conflicts
The echo/directus/sync/collections/operations.json file contains multiple duplicate workflow operations. Entries such as "Check Language", "Check Report Language", and "Dutch" are duplicated, having identical names and keys but distinct _syncId values and resolve/reject references. This issue causes workflow conflicts and unpredictable behavior, likely due to the sync process creating new operations instead of updating existing ones.
echo/directus/sync/collections/operations.json#L21-L603
Bug: Directus Response Handling Flaws
The task_create_view function performs unsafe access to segments[0] and nested dictionary keys (conversations, conversation_segments) without validation, leading to IndexError, KeyError, or TypeError if the Directus response is empty or has an unexpected structure. Furthermore, it uses RUNPOD_TOPIC_MODELER_URL and RUNPOD_TOPIC_MODELER_API_KEY without None checks, which can result in invalid URLs or malformed request headers.
echo/server/dembrane/tasks.py#L456-L478
echo/echo/server/dembrane/tasks.py
Lines 456 to 478 in 5237ccd
Bug: API URL Validation and Response Handling
The RUNPOD_TOPIC_MODELER_URL can be None, which causes the requests.post URL to become None/run and the request to fail. Additionally, the response from the requests.post call is not checked for success, leading to unhandled API errors.
echo/server/dembrane/tasks.py#L477-L480
echo/echo/server/dembrane/tasks.py
Lines 477 to 480 in 5237ccd
Was this report helpful? Give feedback by reacting with 👍 or 👎
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 13
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (6)
echo/directus/sync/collections/operations.json(13 hunks)echo/directus/sync/snapshot/fields/aspect/aspect_segment.json(1 hunks)echo/directus/sync/snapshot/fields/aspect_segment/aspect.json(1 hunks)echo/directus/sync/snapshot/relations/aspect_segment/aspect.json(2 hunks)echo/server/dembrane/config.py(1 hunks)echo/server/dembrane/tasks.py(2 hunks)
🧰 Additional context used
🧠 Learnings (4)
📓 Common learnings
Learnt from: spashii
PR: Dembrane/echo#142
File: echo/frontend/src/lib/query.ts:730-740
Timestamp: 2025-05-13T15:18:29.107Z
Learning: When working with Directus API in this codebase, foreign key relationships must be specified using nested objects with `id` properties (e.g., `conversation_id: { id: conversationId } as Conversation`) rather than direct ID values, even though this appears redundant.
echo/directus/sync/snapshot/relations/aspect_segment/aspect.json (1)
Learnt from: spashii
PR: Dembrane/echo#142
File: echo/frontend/src/lib/query.ts:730-740
Timestamp: 2025-05-13T15:18:29.107Z
Learning: When working with Directus API in this codebase, foreign key relationships must be specified using nested objects with `id` properties (e.g., `conversation_id: { id: conversationId } as Conversation`) rather than direct ID values, even though this appears redundant.
echo/server/dembrane/config.py (2)
Learnt from: ArindamRoy23
PR: Dembrane/echo#113
File: echo/server/dembrane/postgresdbmanager.py:23-40
Timestamp: 2025-04-18T08:45:19.122Z
Learning: Environment variables in the Echo project are defined and validated in `config.py` and should be imported from there rather than accessing `os.environ` directly in other modules. This ensures consistent validation, default values, and logging throughout the application.
Learnt from: ArindamRoy23
PR: Dembrane/echo#113
File: echo/server/dembrane/postgresdbmanager.py:23-40
Timestamp: 2025-04-18T08:45:19.122Z
Learning: Environment variables in the Echo project are defined and validated in `config.py` and should be imported from there rather than accessing `os.environ` directly in other modules. This ensures consistent validation, default values, and logging throughout the application.
echo/directus/sync/collections/operations.json (1)
Learnt from: ussaama
PR: Dembrane/echo#90
File: echo/directus/sync/collections/operations.json:132-145
Timestamp: 2025-03-31T10:29:01.012Z
Learning: In Directus flow operations of type "exec", template variables like {{$env.VARIABLE_NAME}} should be used without quotes in JavaScript code as they are processed by the Directus flow engine before JavaScript execution.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Cursor BugBot
- GitHub Check: ci-check-server
🔇 Additional comments (7)
echo/server/dembrane/config.py (1)
204-212: LGTM! Clean config pattern implementation.The RunPod topic modeler configuration follows the established patterns perfectly. The conditional assertion for the API key when URL is set is solid engineering - no auth token dangling without an endpoint to hit.
echo/directus/sync/snapshot/fields/aspect_segment/aspect.json (1)
19-19: LGTM! Schema refactoring in progress.Nullifying the m2o special designation as part of the broader relationship restructuring between aspect and aspect_segment collections. Clean coordination with the other schema changes.
echo/directus/sync/snapshot/relations/aspect_segment/aspect.json (1)
13-13: LGTM! Solid relationship engineering.Setting the one_field to establish bidirectional relations and CASCADE delete for proper data consistency. When you delete an aspect, the segments should go with it - clean data model design.
Also applies to: 23-23
echo/directus/sync/snapshot/fields/aspect/aspect_segment.json (1)
1-31: LGTM! Clean alias field setup.Perfect one-to-many alias field configuration. The list-o2m interface with segment.id template and enableLink will give users a smooth experience managing the relationship. Well-architected Directus field definition.
echo/server/dembrane/tasks.py (2)
17-23: LGTM! Clean imports for the new functionality.Adding the necessary config variables and directus_client_context import. Following the established import patterns in the codebase.
431-483: LGTM! Solid RunPod pipeline implementation.The task_create_view function is well-architected:
- Proper use of directus_client_context for DB operations
- Smart nested query to fetch conversation segments in one go
- Using set() for unique segment IDs is efficient
- Clean HTTP request construction with proper headers
- Error handling with logging maintains observability
The 600-second timeout is aggressive but reasonable for topic modeling workloads. The payload structure looks clean and the Bearer auth pattern is solid.
echo/directus/sync/collections/operations.json (1)
247-272:get_all_participantsitem-read duplicated – possible dangling resolves.Ensure downstream nodes reference the surviving operation after dedupe.
| { | ||
| "name": "Dutch", | ||
| "key": "dutch", | ||
| "type": "condition", | ||
| "position_x": 20, | ||
| "position_y": 20, | ||
| "options": { | ||
| "filter": { | ||
| "$trigger": { | ||
| "language": { | ||
| "_eq": "nl" | ||
| } | ||
| } | ||
| } | ||
| }, | ||
| "resolve": "ea78ec02-364d-4f18-80f8-ea5ac4c787ed", | ||
| "reject": "eb6f8253-647f-4fb1-9010-e93594ba065e", | ||
| "flow": "17703446-fef0-49e9-bdc4-385db1311137", | ||
| "_syncId": "4795cc24-3f3c-4be9-9844-24552da522fa" | ||
| }, |
There was a problem hiding this comment.
Ditto for "dutch" condition node.
Unique keys or bust – else the NL branch erases itself.
🤖 Prompt for AI Agents
In echo/directus/sync/collections/operations.json around lines 114 to 133, the
"dutch" condition node's "key" must be unique to prevent conflicts that cause
the NL branch to overwrite itself. Review the "key" value and ensure it is
unique within the file by changing it to a distinct identifier that does not
duplicate any other keys.
| "name": "Report Not Published", | ||
| "key": "report_not_published", | ||
| "type": "log", | ||
| "position_x": 19, | ||
| "position_y": 19, | ||
| "options": { | ||
| "message": "The report is not yet published" | ||
| }, | ||
| "resolve": null, | ||
| "reject": null, | ||
| "flow": "ec4e7ea5-72de-4365-b66f-d8f11b549495", | ||
| "_syncId": "e8274ad4-5844-42cd-8a6b-d40d08cf83d3" | ||
| }, |
There was a problem hiding this comment.
🧹 Nitpick (assertive)
report_not_published logger duplicated.
Minor, but still a key collision; cleanup recommended.
🤖 Prompt for AI Agents
In echo/directus/sync/collections/operations.json around lines 419 to 431, there
is a duplicate logger entry with the key "report_not_published" causing a key
collision. Locate and remove or rename one of the duplicate entries to ensure
each logger key is unique and avoid conflicts.
| { | ||
| "name": "PROCEED_ONLY_IF \"published\" in payload", | ||
| "key": "proceed_only_if_published_in_payload", | ||
| "type": "condition", | ||
| "position_x": 19, | ||
| "position_y": 1, | ||
| "options": { | ||
| "filter": { | ||
| "$trigger": { | ||
| "payload": { | ||
| "status": { | ||
| "_eq": "published" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| }, | ||
| "resolve": "d8554457-95b6-474f-ba67-dfd5f936d575", | ||
| "reject": "e8274ad4-5844-42cd-8a6b-d40d08cf83d3", | ||
| "flow": "ec4e7ea5-72de-4365-b66f-d8f11b549495", | ||
| "_syncId": "2784a141-2ae3-48d1-aedf-57c0be3bcc74" | ||
| }, |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Second "proceed_only_if_published_in_payload" – key clash again.
Resolve/reject IDs diverge; post-import you’ll only have one of them, breaking the other branch.
🤖 Prompt for AI Agents
In echo/directus/sync/collections/operations.json between lines 350 and 371,
there is a duplicate key "proceed_only_if_published_in_payload" causing a key
clash. This results in conflicting resolve and reject IDs that will break one of
the branches after import. To fix this, rename one of the duplicate keys to a
unique identifier and ensure its resolve and reject IDs correctly correspond to
the intended flow branches.
| { | ||
| "name": "Email Send Operation Failed", | ||
| "key": "email_send_operation_failed", | ||
| "type": "log", | ||
| "position_x": 58, | ||
| "position_y": 1, | ||
| "options": { | ||
| "message": "An email could not be sent due to some error: {{$last}}" | ||
| }, | ||
| "resolve": null, | ||
| "reject": null, | ||
| "flow": "17703446-fef0-49e9-bdc4-385db1311137", | ||
| "_syncId": "2b24450b-6a2e-4452-aba1-9814d17fef42" | ||
| }, |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Two "email_send_operation_failed" log nodes = one actually synced.
You’ll lose one set of resolve/reject plumbing. Rename or merge.
🤖 Prompt for AI Agents
In echo/directus/sync/collections/operations.json around lines 148 to 161, there
are two log nodes with the key "email_send_operation_failed," which causes a
conflict and loss of resolve/reject plumbing. To fix this, rename one of the
duplicate keys to a unique name or merge the two nodes into a single one,
ensuring that resolve and reject handlers are preserved and correctly linked.
| { | ||
| "name": "Check Report Language", | ||
| "key": "check_report_language", | ||
| "type": "item-read", | ||
| "position_x": 73, | ||
| "position_y": 1, | ||
| "options": { | ||
| "query": { | ||
| "filter": { | ||
| "project_id": { | ||
| "id": { | ||
| "_eq": "{{$trigger.payload.project_id.id}}" | ||
| } | ||
| } | ||
| }, | ||
| "fields": [ | ||
| "language" | ||
| ] | ||
| }, | ||
| "collection": "project_report" | ||
| }, | ||
| "resolve": "ca1ffbc5-cfce-4fb4-8f15-c128ea407d41", | ||
| "reject": null, | ||
| "flow": "ec4e7ea5-72de-4365-b66f-d8f11b549495", | ||
| "_syncId": "c5d2aff3-1a1a-4ea2-8e39-5189409b7dc1" | ||
| }, |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Same collision for "check_report_language" – import chaos ahead.
Identical key, different _syncId. One wins, one dies. Fix like above.
🤖 Prompt for AI Agents
In echo/directus/sync/collections/operations.json between lines 68 and 93, the
"check_report_language" key is duplicated with different _syncId values, causing
a collision. To fix this, ensure each operation key is unique by renaming one of
the "check_report_language" keys to a distinct name or merging the operations if
appropriate, and update the corresponding _syncId to maintain uniqueness.
| { | ||
| "name": "Check Language", | ||
| "key": "check_language", | ||
| "type": "condition", | ||
| "position_x": 20, | ||
| "position_y": 1, | ||
| "options": { | ||
| "filter": { | ||
| "$trigger": { | ||
| "language": { | ||
| "_eq": "en" | ||
| } | ||
| } | ||
| } | ||
| }, | ||
| "resolve": "9390ed2f-7dc6-4a6a-83da-2d87d478261d", | ||
| "reject": "4795cc24-3f3c-4be9-9844-24552da522fa", | ||
| "flow": "17703446-fef0-49e9-bdc4-385db1311137", | ||
| "_syncId": "af7a671d-78fa-4d07-adad-c3bfabc4617c" | ||
| }, |
There was a problem hiding this comment.
Duplicate key will clobber the original operation at import-time.
Directus Flow requires key be unique per flow. A second "check_language" with the same key but a different _syncId means only one of them survives the sync, nuking the other branch and potentially breaking resolve → 9390ed2f-….
Rename the key or delete the clone before shipping.
🤖 Prompt for AI Agents
In echo/directus/sync/collections/operations.json around lines 22 to 41, there
are duplicate "key" values named "check_language" which will cause one to
overwrite the other during import. To fix this, ensure each operation has a
unique "key" by renaming one of the duplicate keys or removing the duplicate
operation entirely before syncing to prevent breaking the flow connections.
| { | ||
| "name": "Project Data", | ||
| "key": "project_data", | ||
| "type": "item-read", | ||
| "position_x": 37, | ||
| "position_y": 1, | ||
| "options": { | ||
| "collection": "project", | ||
| "query": { | ||
| "filter": { | ||
| "id": { | ||
| "id": { | ||
| "_eq": "{{$trigger.payload.project_id}}" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| }, | ||
| "resolve": "47f1463c-3cb6-4420-a50e-92938fed3197", | ||
| "reject": null, | ||
| "flow": "ec4e7ea5-72de-4365-b66f-d8f11b549495", | ||
| "_syncId": "d8554457-95b6-474f-ba67-dfd5f936d575" | ||
| }, |
There was a problem hiding this comment.
🧹 Nitpick (assertive)
Duplicate "project_data" item-read node.
Same collision pattern – rename or remove.
🤖 Prompt for AI Agents
In echo/directus/sync/collections/operations.json between lines 395 and 417,
there is a duplicate item-read node with the key "project_data" causing a
collision. To fix this, either rename one of the duplicate "project_data" keys
to a unique identifier or remove the redundant node to avoid conflicts in the
flow.
| "name": "Send Email English", | ||
| "key": "send_email_english", | ||
| "type": "mail", | ||
| "position_x": 39, | ||
| "position_y": 1, | ||
| "options": { | ||
| "to": [ | ||
| "{{$trigger.email}}" | ||
| ], | ||
| "subject": "A Report Featuring Your Input is Ready", | ||
| "body": null, | ||
| "type": "template", | ||
| "template": "report-notification-en", | ||
| "data": { | ||
| "PARTICIPANT_BASE_URL": "{{$trigger.PARTICIPANT_BASE_URL}}", | ||
| "project_id": "{{$trigger.project_id}}", | ||
| "project_name": "{{$trigger.project_name}}", | ||
| "default_conversation_title": "{{$trigger.default_conversation_title}}", | ||
| "conversation_name": "{{$trigger.conversation_name}}", | ||
| "token": "{{$trigger.token}}" | ||
| }, | ||
| "replyTo": [ | ||
| "info@dembrane.com" | ||
| ] | ||
| }, | ||
| "resolve": null, | ||
| "reject": "2b24450b-6a2e-4452-aba1-9814d17fef42", | ||
| "flow": "17703446-fef0-49e9-bdc4-385db1311137", | ||
| "_syncId": "9390ed2f-7dc6-4a6a-83da-2d87d478261d" | ||
| }, |
There was a problem hiding this comment.
Two "send_email_english" mailers – same fatal collision.
Unique keys or nothing works.
🤖 Prompt for AI Agents
In echo/directus/sync/collections/operations.json between lines 509 and 538,
there are two mailer entries with the identical key "send_email_english,"
causing a fatal collision. To fix this, ensure each mailer has a unique "key"
value by renaming one of the duplicate keys to a distinct identifier. This will
prevent conflicts and allow the system to differentiate between the mailers.
| "name": "Send Email Dutch", | ||
| "key": "send_email_dutch", | ||
| "type": "mail", | ||
| "position_x": 39, | ||
| "position_y": 20, | ||
| "options": { | ||
| "to": [ | ||
| "{{$trigger.email}}" | ||
| ], | ||
| "type": "template", | ||
| "subject": "Er is een rapport klaar met uw inbreng", | ||
| "body": null, | ||
| "data": { | ||
| "PARTICIPANT_BASE_URL": "{{$trigger.PARTICIPANT_BASE_URL}}", | ||
| "project_id": "{{$trigger.project_id}}", | ||
| "project_name": "{{$trigger.project_name}}", | ||
| "default_conversation_title": "{{$trigger.default_conversation_title}}", | ||
| "conversation_name": "{{$trigger.conversation_name}}", | ||
| "token": "{{$trigger.token}}" | ||
| }, | ||
| "template": "report-notification-nl", | ||
| "replyTo": [ | ||
| "info@dembrane.com" | ||
| ] | ||
| }, | ||
| "resolve": null, | ||
| "reject": "615a54cd-a72e-41ad-9403-9577c80280d6", | ||
| "flow": "17703446-fef0-49e9-bdc4-385db1311137", | ||
| "_syncId": "ea78ec02-364d-4f18-80f8-ea5ac4c787ed" | ||
| }, |
There was a problem hiding this comment.
Two "send_email_dutch" mailers – one will disappear.
Avoid key overwrites, otherwise NL users may get ghosted.
🤖 Prompt for AI Agents
In echo/directus/sync/collections/operations.json around lines 447 to 476, there
are two mailer entries with the same key "send_email_dutch", which causes one to
overwrite the other and leads to issues for Dutch users. To fix this, ensure
each mailer has a unique "key" value by renaming one of the duplicate keys to a
distinct identifier to prevent overwriting and ensure both mailers function
correctly.
| "name": "log environment vars", | ||
| "key": "log_environment_vars", | ||
| "type": "log", | ||
| "position_x": 127, | ||
| "position_y": 1, | ||
| "options": { | ||
| "message": "{{data['$env']}} {{data.$env}} {{process.env}} {{$env}} {{$env.PARTICIPANT_BASE_URL}}" | ||
| }, | ||
| "resolve": null, | ||
| "reject": null, | ||
| "flow": "ec4e7ea5-72de-4365-b66f-d8f11b549495", | ||
| "_syncId": "7161e52e-b1cc-4c13-bedd-a9d64527026f" | ||
| }, |
There was a problem hiding this comment.
Logging full $env and process.env leaks secrets.
Dumping every env var to Directus logs is a GDPR/ops nightmare.
Log only the specific var you need, or strip this node in prod.
- "message": "{{data['$env']}} {{data.$env}} {{process.env}} {{$env}} {{$env.PARTICIPANT_BASE_URL}}"
+ "message": "PARTICIPANT_BASE_URL: {{$env.PARTICIPANT_BASE_URL}}"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "name": "log environment vars", | |
| "key": "log_environment_vars", | |
| "type": "log", | |
| "position_x": 127, | |
| "position_y": 1, | |
| "options": { | |
| "message": "{{data['$env']}} {{data.$env}} {{process.env}} {{$env}} {{$env.PARTICIPANT_BASE_URL}}" | |
| }, | |
| "resolve": null, | |
| "reject": null, | |
| "flow": "ec4e7ea5-72de-4365-b66f-d8f11b549495", | |
| "_syncId": "7161e52e-b1cc-4c13-bedd-a9d64527026f" | |
| }, | |
| "name": "log environment vars", | |
| "key": "log_environment_vars", | |
| "type": "log", | |
| "position_x": 127, | |
| "position_y": 1, | |
| "options": { | |
| "message": "PARTICIPANT_BASE_URL: {{$env.PARTICIPANT_BASE_URL}}" | |
| }, | |
| "resolve": null, | |
| "reject": null, | |
| "flow": "ec4e7ea5-72de-4365-b66f-d8f11b549495", | |
| "_syncId": "7161e52e-b1cc-4c13-bedd-a9d64527026f" | |
| }, |
🤖 Prompt for AI Agents
In echo/directus/sync/collections/operations.json around lines 301 to 313, the
current log action outputs the entire environment variables object, which risks
leaking sensitive information. Modify the "message" option to log only the
specific environment variable needed, such as "PARTICIPANT_BASE_URL", instead of
dumping all of process.env or $env. Alternatively, implement a condition to skip
or sanitize this log step in production environments to prevent sensitive data
exposure.
|
All ok |
Summary by CodeRabbit
New Features
Improvements
Bug Fixes