feat: add SDK support for multi label options_order (#4757)

# Description This PR add support to new `options_order` attribute for multi-label questions so it is possible to specify an order to use when rendering labels on the UI. In the backend `options_order` is the used attribute but in the SDK I have used `labels_order` so it matches with the rest of attributes like `visible_labels`. Closes #4756 **Type of change** (Please delete options that are not relevant. Remember to title the PR according to the type of change) - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Refactor (change restructuring the codebase without changing functionality) - [ ] Improvement (change adding some improvement to an existing functionality) - [ ] Documentation update **How Has This Been Tested** (Please describe the tests that you ran to verify your changes. And ideally, reference `tests`) - [x] Modify already existent tests to support new feature. **Checklist** - [ ] I added relevant documentation - [ ] follows the style guidelines of this project - [ ] I did a self-review of my code - [ ] I made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK) (see text above) - [ ] I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)
argilla-io · May 3, 2024 · 92c7815 · 92c7815
1 parent 2533c0d
commit 92c7815
Show file tree

Hide file tree

Showing 7 changed files with 65 additions and 18 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -18,36 +18,37 @@ These are the section headers that we use:
 
 ### Added
 
-- Added `suggestion multi score attribute`([#4730](https://github.com/argilla-io/argilla/pull/4730))
-- Added `order by suggestion first`([#4731](https://github.com/argilla-io/argilla/pull/4731))
-- Added `multi selection entity dropdown` for span annotation overlap ([#4735](https://github.com/argilla-io/argilla/pull/4735))
-- Added `pre selection highlight` for span annotation ([#4726](https://github.com/argilla-io/argilla/pull/4726))
-- Added `banner when persistent storage is not enabled`([#4744](https://github.com/argilla-io/argilla/pull/4744))
+- Added suggestion multi score attribute. ([#4730](https://github.com/argilla-io/argilla/pull/4730))
+- Added order by suggestion first. ([#4731](https://github.com/argilla-io/argilla/pull/4731))
+- Added multi selection entity dropdown for span annotation overlap. ([#4735](https://github.com/argilla-io/argilla/pull/4735))
+- Added pre selection highlight for span annotation. ([#4726](https://github.com/argilla-io/argilla/pull/4726))
+- Added banner when persistent storage is not enabled. ([#4744](https://github.com/argilla-io/argilla/pull/4744))
+- Added support on Python SDK for new multi-label questions `labels_order` attribute. ([#4757](https://github.com/argilla-io/argilla/pull/4757))
 
 ### Changed
 
-- Changed the way to show the `HF space` and `HF user` into sign in page for HF ([#4748](https://github.com/argilla-io/argilla/pull/4748))
+- Changed the way how Hugging Face space and user is showed in sign in. ([#4748](https://github.com/argilla-io/argilla/pull/4748))
 
 ### Fixed
 
-- Fixed Korean character reversed ([#4753](https://github.com/argilla-io/argilla/pull/4753))
+- Fixed Korean character reversed. ([#4753](https://github.com/argilla-io/argilla/pull/4753))
 
 ## [1.27.0](https://github.com/argilla-io/argilla/compare/v1.26.1...v1.27.0)
 
 ### Added
 
-- Added Allow overlap spans in the `FeedbackDataset` ([#4668](https://github.com/argilla-io/argilla/pull/4668))
+- Added Allow overlap spans in the `FeedbackDataset`. ([#4668](https://github.com/argilla-io/argilla/pull/4668))
 - Added `allow_overlapping` parameter for span questions. ([#4697](https://github.com/argilla-io/argilla/pull/4697))
-- Added overall progress bar on `Datasets` table ([#4696](https://github.com/argilla-io/argilla/pull/4696))
-- Added German language translation ([#4688](https://github.com/argilla-io/argilla/pull/4688))
+- Added overall progress bar on `Datasets` table. ([#4696](https://github.com/argilla-io/argilla/pull/4696))
+- Added German language translation. ([#4688](https://github.com/argilla-io/argilla/pull/4688))
 
 ### Changed
 
-- New UI design for suggestions ([#4682](https://github.com/argilla-io/argilla/pull/4682))
+- New UI design for suggestions. ([#4682](https://github.com/argilla-io/argilla/pull/4682))
 
 ### Fixed
 
-- Improve performance for more than 250 labels ([#4702](https://github.com/argilla-io/argilla/pull/4702))
+- Improve performance for more than 250 labels. ([#4702](https://github.com/argilla-io/argilla/pull/4702))
 
 ## [1.26.1](https://github.com/argilla-io/argilla/compare/v1.26.0...v1.26.1)
 

diff --git a/src/argilla/client/feedback/schemas/enums.py b/src/argilla/client/feedback/schemas/enums.py
@@ -59,3 +59,8 @@ class ResponseStatusFilter(str, Enum):
     submitted = "submitted"
     discarded = "discarded"
     pending = "pending"
+
+
+class LabelsOrder(str, Enum):
+    natural = "natural"
+    suggestion = "suggestion"
diff --git a/src/argilla/client/feedback/schemas/questions.py b/src/argilla/client/feedback/schemas/questions.py
@@ -16,7 +16,7 @@
 from abc import ABC, abstractmethod
 from typing import Any, Dict, List, Literal, Optional, Union
 
-from argilla.client.feedback.schemas.enums import QuestionTypes
+from argilla.client.feedback.schemas.enums import LabelsOrder, QuestionTypes
 from argilla.client.feedback.schemas.response_values import parse_value_response_for_question
 from argilla.client.feedback.schemas.responses import ResponseValue, ValueSchema
 from argilla.client.feedback.schemas.suggestions import SuggestionSchema
@@ -262,6 +262,12 @@ class MultiLabelQuestion(_LabelQuestion):
             are the labels that will be shown in the UI.
         visible_labels: The number of visible labels in the UI. Defaults to 20, and must
             be 3 or greater.
+        labels_order: An optional value that configures the order in which the labels are
+            presented in the UI. Possible values are 'natural' and 'suggestion', with 'natural'
+            displaying the labels in the natural order in which they were specified, and 'suggestion'
+            displaying the labels with priority for those associated with a suggestion.
+            The score of the suggestion will be taken into account for ordering if available.
+            Defaults to 'natural'.
 
     Examples:
         >>> from argilla.client.feedback.schemas.questions import MultiLabelQuestion
@@ -271,6 +277,15 @@ class MultiLabelQuestion(_LabelQuestion):
     type: Literal[QuestionTypes.multi_label_selection] = Field(
         QuestionTypes.multi_label_selection.value, allow_mutation=False, const=True
     )
+    labels_order: LabelsOrder = Field(LabelsOrder.natural, description="The order of the labels in the UI.")
+
+    @property
+    def server_settings(self) -> Dict[str, Any]:
+        settings = super().server_settings
+
+        settings["options_order"] = self.labels_order
+
+        return settings
 
 
 class RankingQuestion(QuestionSchema, LabelMappingMixin):

diff --git a/src/argilla/client/feedback/schemas/remote/questions.py b/src/argilla/client/feedback/schemas/remote/questions.py
@@ -15,6 +15,7 @@
 from typing import TYPE_CHECKING, Dict, List, Union
 
 from argilla.client.feedback.schemas import QuestionTypes
+from argilla.client.feedback.schemas.enums import LabelsOrder
 from argilla.client.feedback.schemas.questions import (
     LabelQuestion,
     MultiLabelQuestion,
@@ -114,6 +115,7 @@ def to_local(self) -> MultiLabelQuestion:
             required=self.required,
             labels=self.labels,
             visible_labels=self.visible_labels,
+            labels_order=self.labels_order,
         )
 
     @classmethod
@@ -126,6 +128,7 @@ def from_api(cls, payload: "FeedbackQuestionModel") -> "RemoteMultiLabelQuestion
             required=payload.required,
             labels=_parse_options_from_api(payload),
             visible_labels=payload.settings["visible_options"],
+            labels_order=payload.settings.get("options_order", LabelsOrder.natural),
         )
 
 

diff --git a/tests/unit/client/feedback/schemas/remote/test_questions.py b/tests/unit/client/feedback/schemas/remote/test_questions.py
@@ -17,7 +17,7 @@
 from uuid import uuid4
 
 import pytest
-from argilla.client.feedback.schemas.enums import QuestionTypes
+from argilla.client.feedback.schemas.enums import LabelsOrder, QuestionTypes
 from argilla.client.feedback.schemas.questions import (
     LabelQuestion,
     MultiLabelQuestion,
@@ -283,6 +283,7 @@ def test_remote_label_question_from_api(payload: FeedbackQuestionModel) -> None:
                     "type": "multi_label_selection",
                     "options": [{"text": "a", "value": "a"}, {"text": "b", "value": "b"}, {"text": "c", "value": "c"}],
                     "visible_options": None,
+                    "options_order": LabelsOrder.natural,
                 },
             },
         ),
@@ -294,6 +295,7 @@ def test_remote_label_question_from_api(payload: FeedbackQuestionModel) -> None:
                 "required": False,
                 "labels": {"a": "A", "b": "B", "c": "C"},
                 "visible_labels": 3,
+                "labels_order": LabelsOrder.suggestion,
             },
             {
                 "name": "a",
@@ -304,6 +306,7 @@ def test_remote_label_question_from_api(payload: FeedbackQuestionModel) -> None:
                     "type": "multi_label_selection",
                     "options": [{"text": "A", "value": "a"}, {"text": "B", "value": "b"}, {"text": "C", "value": "c"}],
                     "visible_options": 3,
+                    "options_order": LabelsOrder.suggestion,
                 },
             },
         ),
@@ -335,6 +338,7 @@ def test_remote_multi_label_question(schema_kwargs: Dict[str, Any], server_paylo
                 "type": "multi_label_selection",
                 "options": [{"text": "a", "value": "a"}, {"text": "b", "value": "b"}, {"text": "c", "value": "c"}],
                 "visible_options": None,
+                "options_order": LabelsOrder.natural,
             },
             inserted_at=datetime.now(),
             updated_at=datetime.now(),
@@ -349,6 +353,7 @@ def test_remote_multi_label_question(schema_kwargs: Dict[str, Any], server_paylo
                 "type": "multi_label_selection",
                 "options": [{"text": "A", "value": "a"}, {"text": "B", "value": "b"}, {"text": "C", "value": "c"}],
                 "visible_options": 3,
+                "options_order": LabelsOrder.suggestion,
             },
             inserted_at=datetime.now(),
             updated_at=datetime.now(),

diff --git a/tests/unit/client/feedback/schemas/test_questions.py b/tests/unit/client/feedback/schemas/test_questions.py
@@ -15,7 +15,7 @@
 from typing import Any, Dict
 
 import pytest
-from argilla.client.feedback.schemas.enums import QuestionTypes
+from argilla.client.feedback.schemas.enums import LabelsOrder, QuestionTypes
 from argilla.client.feedback.schemas.questions import (
     LabelQuestion,
     MultiLabelQuestion,
@@ -124,7 +124,7 @@ def test_rating_question_errors(schema_kwargs: Dict[str, Any], exception_cls: An
         ({"name": "a", "labels": {"a": "a", "b": "a"}}, ValidationError, "ensure this dict has unique values"),
     ],
 )
-def test__label_question_errors(schema_kwargs: Dict[str, Any], exception_cls: Any, exception_message: str) -> None:
+def test_label_question_errors(schema_kwargs: Dict[str, Any], exception_cls: Any, exception_message: str) -> None:
     with pytest.raises(exception_cls, match=exception_message):
         _LabelQuestion(**schema_kwargs, type="label_selection")
 
@@ -149,7 +149,7 @@ def test__label_question_errors(schema_kwargs: Dict[str, Any], exception_cls: An
         ),
     ],
 )
-def test__label_question_warnings(schema_kwargs: Dict[str, Any], warning_cls: Warning, warning_message: str) -> None:
+def test_label_question_warnings(schema_kwargs: Dict[str, Any], warning_cls: Warning, warning_message: str) -> None:
     with pytest.warns(warning_cls, match=warning_message):
         _LabelQuestion(**schema_kwargs, type="label_selection")
 
@@ -292,11 +292,12 @@ def test_label_question(schema_kwargs: Dict[str, Any], server_payload: Dict[str,
                     "type": "multi_label_selection",
                     "options": [{"value": "a", "text": "a"}, {"value": "b", "text": "b"}],
                     "visible_options": None,
+                    "options_order": LabelsOrder.natural,
                 },
             },
         ),
         (
-            {"name": "a", "labels": {"a": "A", "b": "B"}},
+            {"name": "a", "labels": {"a": "A", "b": "B"}, "labels_order": LabelsOrder.suggestion},
             {
                 "name": "a",
                 "title": "A",
@@ -306,6 +307,7 @@ def test_label_question(schema_kwargs: Dict[str, Any], server_payload: Dict[str,
                     "type": "multi_label_selection",
                     "options": [{"value": "a", "text": "A"}, {"value": "b", "text": "B"}],
                     "visible_options": None,
+                    "options_order": LabelsOrder.suggestion,
                 },
             },
         ),
@@ -320,6 +322,7 @@ def test_label_question(schema_kwargs: Dict[str, Any], server_payload: Dict[str,
                     "type": "multi_label_selection",
                     "options": [{"value": "a", "text": "a"}, {"value": "b", "text": "b"}],
                     "visible_options": None,
+                    "options_order": LabelsOrder.natural,
                 },
             },
         ),
@@ -334,6 +337,7 @@ def test_label_question(schema_kwargs: Dict[str, Any], server_payload: Dict[str,
                     "type": "multi_label_selection",
                     "options": [{"value": "a", "text": "a"}, {"value": "b", "text": "b"}],
                     "visible_options": None,
+                    "options_order": LabelsOrder.natural,
                 },
             },
         ),
@@ -348,6 +352,7 @@ def test_label_question(schema_kwargs: Dict[str, Any], server_payload: Dict[str,
                     "type": "multi_label_selection",
                     "options": [{"value": str(n), "text": str(n)} for n in list(range(20))],
                     "visible_options": None,
+                    "options_order": LabelsOrder.natural,
                 },
             },
         ),
@@ -362,6 +367,7 @@ def test_label_question(schema_kwargs: Dict[str, Any], server_payload: Dict[str,
                     "type": "multi_label_selection",
                     "options": [{"value": str(n), "text": str(n)} for n in list(range(21))],
                     "visible_options": 20,
+                    "options_order": LabelsOrder.natural,
                 },
             },
         ),
@@ -376,6 +382,7 @@ def test_label_question(schema_kwargs: Dict[str, Any], server_payload: Dict[str,
                     "type": "multi_label_selection",
                     "options": [{"value": str(n), "text": str(n)} for n in list(range(2))],
                     "visible_options": None,
+                    "options_order": LabelsOrder.natural,
                 },
             },
         ),
@@ -390,6 +397,7 @@ def test_label_question(schema_kwargs: Dict[str, Any], server_payload: Dict[str,
                     "type": "multi_label_selection",
                     "options": [{"value": str(n), "text": str(n)} for n in list(range(2))],
                     "visible_options": None,
+                    "options_order": LabelsOrder.natural,
                 },
             },
         ),

diff --git a/tests/unit/client/feedback/test_config.py b/tests/unit/client/feedback/test_config.py
@@ -18,6 +18,7 @@
 
 import pytest
 from argilla.client.feedback.config import DatasetConfig, DeprecatedDatasetConfig
+from argilla.client.feedback.schemas.enums import LabelsOrder
 from argilla.client.feedback.schemas.fields import FieldSchema
 from argilla.client.feedback.schemas.questions import QuestionSchema
 from yaml import SafeLoader, load
@@ -235,6 +236,7 @@ def test_dataset_config_json_deprecated(
                                 {"value": "label-3", "text": "label-3", "description": None},
                             ],
                             "visible_options": 3,
+                            "options_order": LabelsOrder.natural,
                         },
                         "visible_labels": 20,
                         "id": "a3a12c67-73d8-41b6-a697-f88be8f9386c",
@@ -327,6 +329,7 @@ def test_dataset_config_json_deprecated(
                                 {"value": "label-3", "text": "label-3", "description": None},
                             ],
                             "visible_options": 3,
+                            "options_order": LabelsOrder.natural,
                         },
                         "visible_labels": 20,
                         "id": "0fbcf59a-eef9-48d0-b50c-011b22a1b611",
@@ -406,6 +409,7 @@ def test_dataset_config_json_deprecated(
                                 {"value": "label-3", "text": "label-3"},
                             ],
                             "visible_options": 3,
+                            "options_order": LabelsOrder.natural,
                         },
                         "labels": ["label-1", "label-2", "label-3"],
                         "visible_labels": 3,
@@ -483,9 +487,11 @@ def test_dataset_config_json_deprecated(
                                 {"value": "label-3", "text": "label-3"},
                             ],
                             "visible_options": 3,
+                            "options_order": LabelsOrder.natural,
                         },
                         "labels": ["label-1", "label-2", "label-3"],
                         "visible_labels": 3,
+                        "labels_order": LabelsOrder.natural,
                     },
                     {
                         "name": "question-5",
@@ -652,9 +658,11 @@ def test_dataset_config_backwards_compatibility_argilla_cfg(
                   value: label-3
                 type: multi_label_selection
                 visible_options: 3
+                options_order: natural
               title: Question-4
               type: multi_label_selection
               visible_labels: 3
+              labels_order: natural
             - description: null
               id: !!python/object:uuid.UUID
                 int: 251163320782812347764238417960223431273
@@ -764,9 +772,11 @@ def test_dataset_config_backwards_compatibility_argilla_cfg(
                   value: label-3
                 type: multi_label_selection
                 visible_options: 3
+                options_order: natural
               title: Question-4
               type: multi_label_selection
               visible_labels: 3
+              labels_order: natural
             - description: null
               name: question-5
               required: false