Feature Request: Tagging System for File Collections (Group & User Workspaces)

[paullizer]

Summary

Implement a comprehensive tagging system for organizing and filtering file collections in both group and user workspaces. Tags are stored as objects containing both the tag name and the associated group or user ID, ensuring clear scoping and efficient filtering. Tags are assignable to both documents and their chunks, and are always scoped to their creator (user or group).

---

Motivation

As the number of documents and chunks increases, users and groups need a scalable way to organize, search, and filter files. Tagging enables logical grouping and focused queries, reducing backend load and improving frontend usability. This is especially important as data sets grow and documents are shared across groups or managed by individual users.

---

Requirements & Code Integration

1. Data Model & Storage

• Tag Structure:

  • Tags are stored as objects in a tags array within each document and each chunk, e.g.:

    "tags": [
      { "tag_name": "ProjectX", "group_id": "abc123" }
    ]

    or

    "tags": [
      { "tag_name": "Urgent", "user_id": "user456" }
    ]

  • Tags are always scoped to their creator (user or group) and are not shared.

  • User documents and their chunks are only associated with users; group documents and their chunks are only associated with groups.

• Containers & Indexes:

  • User documents are stored in the DOCUMENTS Cosmos container and indexed in the user-index.
  • Group documents are stored in the GROUP_DOCUMENTS Cosmos container and indexed in the group-index.
  • Chunks for user documents are indexed in the user-index; chunks for group documents are indexed in the group-index.

• Tag Management:

  • Group tags are managed in the group's Cosmos record; user tags in the user's profile record.

2. API Endpoints

• User Documents & Chunks:
  • Extend endpoints in [route_backend_documents.py](https://github.com/microsoft/simplechat/issues/application/single_app/route_backend_documents.py:1) to support:
    • Tag assignment and filtering using the {tag_name, user_id} structure for both documents and chunks.
• Group Documents & Chunks:
  • Extend endpoints in [route_backend_group_documents.py](https://github.com/microsoft/simplechat/issues/application/single_app/route_backend_group_documents.py:1) to support:
    • Tag assignment and filtering using the {tag_name, group_id} structure for both documents and chunks.
• Tag Management:
  • Add endpoints for CRUD operations on group/user tags, e.g. /api/group_tags, /api/user_tags.

3. Frontend Integration

• Group Workspace UI:
  • Update [route_frontend_group_workspaces.py](https://github.com/microsoft/simplechat/issues/application/single_app/route_frontend_group_workspaces.py:1) and [group_workspaces.html](https://github.com/microsoft/simplechat/issues/application/single_app/templates/group_workspaces.html:1) to:
    • Display/manage group tags.
    • Assign tags to group documents and their chunks.
    • Filter group document and chunk lists by tags.
• User Workspace UI:
  • Update [route_frontend_workspace.py](https://github.com/microsoft/simplechat/issues/application/single_app/route_frontend_workspace.py:1) and [workspace.html](https://github.com/microsoft/simplechat/issues/application/single_app/templates/workspace.html:1) for user tag management, assignment, and filtering at both document and chunk level.
• Admin Settings UI:
  • Update [route_frontend_admin_settings.py](https://github.com/microsoft/simplechat/issues/application/single_app/route_frontend_admin_settings.py:1) and [admin_settings.html](https://github.com/microsoft/simplechat/issues/application/single_app/templates/admin_settings.html:1) to add a toggle for enabling/disabling the tagging feature globally.

4. Settings & Permissions

• Global Toggle:
  • Add a setting in [functions_settings.py](https://github.com/microsoft/simplechat/issues/application/single_app/functions_settings.py:1) and expose via admin UI to enable/disable tagging for all workspaces.
• Permissions:
  • Only group members can manage group tags.
  • Only users can manage their own tags.
  • Tag names must be unique within their scope (group/user).

5. Filtering & Querying

• Backend:
  • Update document and chunk query logic in [functions_documents.py](https://github.com/microsoft/simplechat/issues/application/single_app/functions_documents.py:1) and group document logic to filter by tags using both tag_name and the relevant group_id or user_id.
  • Update search logic in [functions_search.py](https://github.com/microsoft/simplechat/issues/application/single_app/functions_search.py:1) to support tag-based filtering, ensuring queries are scoped to the correct container and index.
• Frontend:
  • Add tag filter controls to both workspace UIs for documents and chunks.

6. Migration & Backward Compatibility

• Provide migration scripts for both user and group Cosmos DB containers to add the new tag object structure to documents and chunks.
• Ensure all existing document and chunk endpoints remain backward compatible.

7. Testing

• Add/extend tests for tag CRUD, assignment, filtering, and permissions for both user and group document and chunk flows.

---

Example Data Model (Cosmos DB)

User document:

{
  "document_id": "doc101",
  "user_id": "userA",
  "tags": [
    { "tag_name": "Supple", "user_id": "userA" }
  ]
}

User document chunk:

{
  "chunk_id": "chunk123",
  "document_id": "doc101",
  "tags": [
    { "tag_name": "Supple", "user_id": "userA" }
  ]
}

Group document:

{
  "document_id": "doc789",
  "group_id": "groupA",
  "tags": [
    { "tag_name": "Finance", "group_id": "groupA" }
  ]
}

Group document chunk:

{
  "chunk_id": "chunk456",
  "document_id": "doc789",
  "tags": [
    { "tag_name": "Finance", "group_id": "groupA" }
  ]
}

---

Mermaid Diagram

erDiagram
    USER {
      string user_id
    }
    GROUP {
      string group_id
    }
    DOCUMENT {
      string document_id
      string user_id
    }
    GROUP_DOCUMENT {
      string document_id
      string group_id
    }
    DOCUMENT_CHUNK {
      string chunk_id
      string document_id
    }
    GROUP_DOCUMENT_CHUNK {
      string chunk_id
      string document_id
    }
    USER_TAG {
      string tag_name
      string user_id
    }
    GROUP_TAG {
      string tag_name
      string group_id
    }

    USER ||--o{ DOCUMENT : "owns"
    GROUP ||--o{ GROUP_DOCUMENT : "owns"
    DOCUMENT ||--o{ DOCUMENT_CHUNK : "has"
    GROUP_DOCUMENT ||--o{ GROUP_DOCUMENT_CHUNK : "has"
    DOCUMENT ||--o{ USER_TAG : "has"
    GROUP_DOCUMENT ||--o{ GROUP_TAG : "has"
    DOCUMENT_CHUNK ||--o{ USER_TAG : "has"
    GROUP_DOCUMENT_CHUNK ||--o{ GROUP_TAG : "has"

Loading

---

Acceptance Criteria

• Admin can enable/disable tagging globally via [admin_settings.html](https://github.com/microsoft/simplechat/issues/application/single_app/templates/admin_settings.html:1).
• Group tags are managed and assigned from group workspace; user tags from user workspace.
• Tags are stored as objects with tag_name and group_id or user_id in Cosmos DB, with group and user documents and chunks in separate containers/indexes.
• Documents and chunks can be filtered by tags, with queries scoped to the active group/user and their respective containers and AI Search indexes.
• UI/UX is intuitive for tag management and assignment at both document and chunk level.
• Backend endpoints are secure and validated.
• All new/updated endpoints and UI components are covered by tests.

---

Key Files to Update

• [functions_documents.py](https://github.com/microsoft/simplechat/issues/application/single_app/functions_documents.py:1) (user documents and chunks)
• [route_backend_documents.py](https://github.com/microsoft/simplechat/issues/application/single_app/route_backend_documents.py:1) (user document endpoints)
• [route_backend_group_documents.py](https://github.com/microsoft/simplechat/issues/application/single_app/route_backend_group_documents.py:1) (group document endpoints)
• [functions_group.py](https://github.com/microsoft/simplechat/issues/application/single_app/functions_group.py:1) (group logic)
• [functions_settings.py](https://github.com/microsoft/simplechat/issues/application/single_app/functions_settings.py:1) (settings)
• [functions_search.py](https://github.com/microsoft/simplechat/issues/application/single_app/functions_search.py:1) (search logic)
• [route_backend_groups.py](https://github.com/microsoft/simplechat/issues/application/single_app/route_backend_groups.py:1)
• [route_backend_users.py](https://github.com/microsoft/simplechat/issues/application/single_app/route_backend_users.py:1)
• [route_frontend_group_workspaces.py](https://github.com/microsoft/simplechat/issues/application/single_app/route_frontend_group_workspaces.py:1)
• [route_frontend_workspace.py](https://github.com/microsoft/simplechat/issues/application/single_app/route_frontend_workspace.py:1)
• [route_frontend_admin_settings.py](https://github.com/microsoft/simplechat/issues/application/single_app/route_frontend_admin_settings.py:1)
• [group_workspaces.html](https://github.com/microsoft/simplechat/issues/application/single_app/templates/group_workspaces.html:1)
• [workspace.html](https://github.com/microsoft/simplechat/issues/application/single_app/templates/workspace.html:1)
• [admin_settings.html](https://github.com/microsoft/simplechat/issues/application/single_app/templates/admin_settings.html:1)