Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(#951): new uncovered_by_rules records filter #991

Merged
merged 9 commits into from Jan 19, 2022
Merged

feat(#951): new uncovered_by_rules records filter #991

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
f3c663d
feat(api): new only_uncovered records filter
frascuchon Jan 17, 2022
f63cdf1
feat: add uncovered_by_rules filter param
frascuchon Jan 18, 2022
7f1c3f3
feat(text-class): update query request model
frascuchon Jan 18, 2022
1f027f0
refactor: configure must_not filters
frascuchon Jan 18, 2022
5ead4f2
test: add missing tests
frascuchon Jan 18, 2022
035651d
revert: list rules prior to generate query
frascuchon Jan 18, 2022
5805db6
Apply suggestions from code review
frascuchon Jan 18, 2022
44df158
test: add functional test
frascuchon Jan 19, 2022
6316d8c
Merge branch 'master' into feat/api-uncovered-records-filter
frascuchon Jan 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
16 changes: 15 additions & 1 deletion src/rubrix/server/tasks/text_classification/api/model.py
View file
Open in desktop
Expand Up @@ -489,7 +489,12 @@ class TextClassificationQuery(BaseModel):
status: List[TaskStatus] = Field(default_factory=list)
predicted: Optional[PredictionStatus] = Field(default=None, nullable=True)

def as_elasticsearch(self) -> Dict[str, Any]:
only_uncovered: bool = Field(
frascuchon marked this conversation as resolved.
Show resolved Hide resolved
default=False,
description="If enabled, filter records that are not affected by defined rules",
)

def as_elasticsearch(self, rules: List[LabelingRule]) -> Dict[str, Any]:
frascuchon marked this conversation as resolved.
Show resolved Hide resolved
"""Build an elasticsearch query part from search query"""

if self.ids:
Expand All @@ -506,6 +511,15 @@ def as_elasticsearch(self) -> Dict[str, Any]:
filters.status(self.status),
filters.predicted(self.predicted),
filters.score(self.score),
filters.boolean_filter(
must_not_query=filters.boolean_filter(
should_filters=[
filters.text_query(rule.query) for rule in rules
]
)
)
if self.only_uncovered and rules
else None,
]
if query_filter
]
Expand Down
6 changes: 4 additions & 2 deletions src/rubrix/server/tasks/text_classification/service/service.py
View file
Open in desktop
Expand Up @@ -135,10 +135,11 @@ def search(
The matched records with aggregation info for specified task_meta.py

"""
rules = self.__labeling__.list_rules(dataset)
results = self.__dao__.search_records(
dataset,
search=RecordSearch(
query=query.as_elasticsearch(),
query=query.as_elasticsearch(rules),
sort=sort_by2elasticsearch(
sort_by,
valid_fields=[
Expand Down Expand Up @@ -187,8 +188,9 @@ def read_dataset(
the provided query filters. Optional

"""
rules = self.__labeling__.list_rules(dataset)
for db_record in self.__dao__.scan_dataset(
dataset, search=RecordSearch(query=query.as_elasticsearch())
dataset, search=RecordSearch(query=query.as_elasticsearch(rules))
):
yield TextClassificationRecord.parse_obj(db_record)

Expand Down