Skip to content

Commit

Permalink
feat: argilla-server changes to support rating questions with zero va…
Browse files Browse the repository at this point in the history
…lues (#4858)

# Description

This PR includes changes to `argilla-server` to support the creation of
rating questions with `0` as value.

The maximum number of items for the rating question has been increased
from `10` to `11` so we allow to define a rating question using all
posible values like `0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10`.

Some small refactors have been included too in the PR.

Closes #4857 

**Type of change**

(Please delete options that are not relevant. Remember to title the PR
according to the type of change)

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Refactor (change restructuring the codebase without changing
functionality)
- [x] Improvement (change adding some improvement to an existing
functionality)
- [ ] Documentation update

**How Has This Been Tested**

(Please describe the tests that you ran to verify your changes. And
ideally, reference `tests`)

- [x] Changed some tests to support the improvement.

**Checklist**

- [ ] I added relevant documentation
- [ ] follows the style guidelines of this project
- [ ] I did a self-review of my code
- [ ] I made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK)
(see text above)
- [ ] I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)
  • Loading branch information
jfcalvo committed May 21, 2024
1 parent 3404dbe commit 71c5693
Show file tree
Hide file tree
Showing 7 changed files with 74 additions and 18 deletions.
1 change: 1 addition & 0 deletions argilla-server/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ These are the section headers that we use:

## [Unreleased]()

- Added support for rating questions to include `0` as a valid value. ([#4858](https://github.com/argilla-io/argilla/pull/4858))
- Added `POST /api/v1/token` endpoint to generate a new API token for a user. ([#138](https://github.com/argilla-io/argilla-server/pull/138))
- Added `GET /api/v1/me` endpoint to get the current user information. ([#140](https://github.com/argilla-io/argilla-server/pull/140))
- Added `GET /api/v1/users` endpoint to get a list of all users. ([#142](https://github.com/argilla-io/argilla-server/pull/142))
Expand Down
26 changes: 9 additions & 17 deletions argilla-server/src/argilla_server/schemas/v1/questions.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,9 @@
RANKING_OPTIONS_MAX_ITEMS = 50

RATING_OPTIONS_MIN_ITEMS = 2
RATING_OPTIONS_MAX_ITEMS = 10
RATING_LOWER_VALUE_ALLOWED = 1
RATING_UPPER_VALUE_ALLOWED = 10
RATING_OPTIONS_MAX_ITEMS = 11
RATING_VALUE_GREATER_THAN_OR_EQUAL = 0
RATING_VALUE_LESS_THAN_OR_EQUAL = 10

SPAN_OPTIONS_MIN_ITEMS = 1
SPAN_MIN_VISIBLE_OPTIONS = 3
Expand Down Expand Up @@ -124,30 +124,22 @@ class RatingQuestionSettingsOption(BaseModel):
value: int


class RatingQuestionSettingsOptionCreate(BaseModel):
value: int = Field(ge=RATING_VALUE_GREATER_THAN_OR_EQUAL, le=RATING_VALUE_LESS_THAN_OR_EQUAL)


class RatingQuestionSettings(BaseModel):
type: Literal[QuestionType.rating]
options: conlist(item_type=RatingQuestionSettingsOption)
options: List[RatingQuestionSettingsOption]


class RatingQuestionSettingsCreate(UniqueValuesCheckerMixin):
type: Literal[QuestionType.rating]
options: conlist(
item_type=RatingQuestionSettingsOption,
options: List[RatingQuestionSettingsOptionCreate] = Field(
min_items=RATING_OPTIONS_MIN_ITEMS,
max_items=RATING_OPTIONS_MAX_ITEMS,
)

@validator("options")
def check_option_value_range(cls, options: List[RatingQuestionSettingsOption]):
"""Validator to control all values are in allowed range 1 <= x <= 10"""
for option in options:
if not RATING_LOWER_VALUE_ALLOWED <= option.value <= RATING_UPPER_VALUE_ALLOWED:
raise ValueError(
f"Option value {option.value!r} out of range "
f"[{RATING_LOWER_VALUE_ALLOWED!r}, {RATING_UPPER_VALUE_ALLOWED!r}]"
)
return options


class RatingQuestionSettingsUpdate(UpdateSchema):
type: Literal[QuestionType.rating]
Expand Down
1 change: 1 addition & 0 deletions argilla-server/tests/factories.py
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,7 @@ class RatingQuestionFactory(QuestionFactory):
settings = {
"type": QuestionType.rating.value,
"options": [
{"value": 0},
{"value": 1},
{"value": 2},
{"value": 3},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -112,3 +112,63 @@ async def test_create_dataset_multi_label_selection_question_with_options_order_

assert response.status_code == 422
assert (await db.execute(select(func.count(Question.id)))).scalar_one() == 0

async def test_create_dataset_rating_question(
self, db: AsyncSession, async_client: AsyncClient, owner_auth_header: dict
):
dataset = await DatasetFactory.create()

response = await async_client.post(
self.url(dataset.id),
headers=owner_auth_header,
json={
"name": "name",
"title": "title",
"settings": {
"type": QuestionType.rating,
"options": [
{"value": 0},
{"value": 1},
{"value": 2},
{"value": 3},
{"value": 4},
{"value": 5},
{"value": 6},
{"value": 7},
{"value": 8},
{"value": 9},
{"value": 10},
],
},
},
)

question = (await db.execute(select(Question))).scalar_one()

assert response.status_code == 201
assert response.json() == {
"id": str(question.id),
"name": "name",
"title": "title",
"description": None,
"required": False,
"settings": {
"type": QuestionType.rating,
"options": [
{"value": 0},
{"value": 1},
{"value": 2},
{"value": 3},
{"value": 4},
{"value": 5},
{"value": 6},
{"value": 7},
{"value": 8},
{"value": 9},
{"value": 10},
],
},
"dataset_id": str(dataset.id),
"inserted_at": question.inserted_at.isoformat(),
"updated_at": question.updated_at.isoformat(),
}
1 change: 1 addition & 0 deletions argilla-server/tests/unit/api/v1/test_datasets.py
Original file line number Diff line number Diff line change
Expand Up @@ -326,6 +326,7 @@ async def test_list_dataset_questions(self, async_client: "AsyncClient", owner_a
"settings": {
"type": "rating",
"options": [
{"value": 0},
{"value": 1},
{"value": 2},
{"value": 3},
Expand Down
1 change: 1 addition & 0 deletions argilla-server/tests/unit/api/v1/test_questions.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@
{
"type": "rating",
"options": [
{"value": 0},
{"value": 1},
{"value": 2},
{"value": 3},
Expand Down
2 changes: 1 addition & 1 deletion argilla-server/tests/unit/api/v1/test_records.py
Original file line number Diff line number Diff line change
Expand Up @@ -790,7 +790,7 @@ async def test_create_record_response_with_extra_question_responses(
"rating_question_1": {"value": "wrong-rating-value"},
},
},
"'wrong-rating-value' is not a valid rating for rating question.\nValid ratings are: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]",
"'wrong-rating-value' is not a valid rating for rating question.\nValid ratings are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]",
),
(
create_label_selection_questions,
Expand Down

0 comments on commit 71c5693

Please sign in to comment.