TO BE PUSHED ONCE test on niko and complex-test are OK as well as DB migration Feat/sim search b2f backup before rebase #81

moilerat · 2024-05-17T17:36:21Z

No description provided.

…arity search to object_set/{project_id}/query

…om front to be continued....

…cal repo on lovnower@niko by VR

…nt change) https://app.clickup.com/t/86bwqp6jp

…cotaxa_back/pull/72/files/64579a46644272002e96bae7b255c960d41ba80f#diff-94bbcc1b33ad7aae0725c6a86f78b32f40d8686cfe46c8594267baefb504c73a

… @grololo advice

…ge in their config

… sim_search_b2f

…n pg

…dev (from my own's understanding)

# .github/workflows/auto_tests.yml

…not ea endpoint that wasn't used in prod

trying to set up the test before the rebase

…ch_merge_integration_0524

VR essai de merge de master dans ss

grololo06

On s'approche

grololo06 · 2024-06-16T07:18:16Z

py/main.py

+
+    sim_search_request = SimilaritySearchReq(
+        project_id = project_id,
+        target_id = int(filters.seed_object_id.lstrip("I")) if filters.seed_object_id else None,


Je ne comprends pas pourquoi target_id peut être None. C'est l'objet dont on va chercher les similaires, ça me semble fonctionnellement absurde qu'il soit absent.

grololo06 · 2024-06-16T07:20:02Z

QA/py/tests/test_similarity_search.py

+        "target_id": target_id,
+    }
+    filters = {}
+    req_and_filters = {"filters": filters, "request": req}


C'est la cause du problème de test planté dans l'action. Le filter est (je ne sais pourquoi) censé contenu la cible de la recherche par similarité, et le champ n'est pas rempli ici.

grololo06 · 2024-06-16T07:20:42Z

py/API_models/filters.py

@@ -73,6 +73,10 @@ class ProjectFiltersDict(TypedDict, total=False):
    filt_last_annot: Optional[str]
    """ Coma-separated list of annotator, i.e. person who validated the classification
        in last. """
+    seed_object_id: Optional[str]


Pourquoi c'est dans les filtres, qui sont censés limiter l'affichage? Et pourquoi une str? C'est censé être e.g. I123456?

grololo06 · 2024-06-16T07:25:28Z

QA/py/tests/test_similarity_search.py

+        "target_id": target_id,
+    }
+    filters = {"taxo": ",".join([str(taxo_id) for taxo_id in taxo_ids_to_filter])}
+    req_and_filters = {"filters": filters, "request": req}


Pareil que + haut.

grololo06 · 2024-06-16T07:26:12Z

documentation/query_update_cnnfeature_vector_migrate_0124.log

@@ -0,0 +1,12 @@
+ecotaxa4=# CREATE TABLE obj_cnn_features_vector AS SELECT objcnnid, ARRAY[cnn01, cnn02, cnn03, cnn04, cnn05, cnn06, cnn07, cnn08, cnn09, cnn10, cnn11, cnn12, cnn13, cnn14, cnn15, cnn16, cnn17, cnn18, cnn19, cnn20, cnn21, cnn22, cnn23, cnn24, cnn25, cnn26, cnn27, cnn28, cnn29, cnn30, cnn31, cnn32, cnn33, cnn34, cnn35, cnn36, cnn37, cnn38, cnn39, cnn40, cnn41, cnn42, cnn43, cnn44, cnn45, cnn46, cnn47, cnn48, cnn49, cnn50] as features FROM obj_cnn_features;


Ce n'est plus le code de migration actuel, autant virer.

grololo06 · 2024-06-16T07:38:51Z

py/main.py

+            elif len(rsp.details[len(rsp.details) - 1]) == 12:
+                rsp.details[len(rsp.details) - 1].append(sim_search_score_current)
+            else :
+                print("Unexpected short details, sim score not send to front : " + str(len(rsp.details[len(rsp.details) - 1])))


Utilise STP le log pour laisser des messages, on peut même le capturer dans les tests.

grololo06 · 2024-06-16T07:40:29Z

py/main.py

+                window_size=None,
+            )
+
+        object_ids = [with_p[0] for with_p in obj_with_parents]


Un set() serait mieux vu que c'est principalement pour chercher dedans.

grololo06 · 2024-06-16T07:41:08Z

py/main.py

+            if objid not in object_ids:
+                continue
+            index = object_ids.index(objid)
+            index_ss = sim_search_rsp.neighbor_ids.index(objid) # VR TODO est-ce qu'un zip serait mieux ?


Oui, un zip() serait plus efficace.

grololo06 · 2024-06-16T07:43:02Z

py/main.py

+        example="obj.longitude",
+    ),
+    # TODO: order_field should be a user-visible field name, not nXXX, in case of free field
+    window_start: Optional[int] = Query(


Ne me semble pas pertinent dans le contexte.

grololo06 · 2024-06-16T07:43:28Z

py/main.py

+        default=None,
+        example="obj.longitude,fre.feret",
+    ),
+    order_field: Optional[str] = Query(


Ne me semble pas pertinent dans le contexte.

grololo06 · 2024-06-17T04:52:43Z

py/API_operations/GPU_FeatureExtraction.py

+        self.set_job_result(errors=[], infos=done_infos)
+        return
+
+    def _ensure_deep_features_for(self, proj_id: ProjectIDT, model_name: str) -> str:


Mon IDE dit que c'est un copier/coller de la méthode de même nom dans GPU_Prediction.py, il faudrait l'enlever là-bas.

moilerat and others added 30 commits December 15, 2023 16:09

wip sim search integration

baedf7e

move feature extraction to import

8ce0f33

- JsonDict -> List[Dict]

add930f

samples are not correct, we need to have a similar endpoint for simil…

5fac3bd

…arity search to object_set/{project_id}/query

typo in endpoint that doesn't explain the fact it is not reachable fr…

43ff498

…om front to be continued....

add seed_object_id in filter to get them in api on back

5eaf4b1

I would rather put out of github the .idea

53ff797

fix typo in endpoint

6ded9bf

first working version of unsupervised search in ecotaxa by lovnower, wip

0649bea

cleaning

00180b8

replace obj_cnn_features table with obj_cnn_features_vector

732fa26

use postgres docker with pgvector installed for automated tests

5547438

add basic similarity search as a job + test

eeb5ecb

attempt to fix import bug in simsearch test

57bc453

add filters to similarity search

4899815

remove print and broken test

ab3ef53

launch feature extraction job before similarity search

4cf7731

add similarity scores + test similarity search with filters on taxo

ff780aa

fix filters bug

fa2f418

replace function used in similarity search endpoint

5a49108

remove limit to similarity search results set by the object_set query

5ebce10

remove prints

511a7fd

add blank new line in README to test right to commit and config of lo…

64579a4

…cal repo on lovnower@niko by VR

first version script upgrade db with alchemy

a9cbb3f

mention emma and lovnower for ss and cnnvector in license (only comme…

e0475c3

…nt change) https://app.clickup.com/t/86bwqp6jp

fix check that cnn_features are selected https://github.com/ecotaxa/e…

94bc185

…cotaxa_back/pull/72/files/64579a46644272002e96bae7b255c960d41ba80f#diff-94bbcc1b33ad7aae0725c6a86f78b32f40d8686cfe46c8594267baefb504c73a

trying to take into account @grololo remark #72

a1a9abc

update test value for missing descriptors in test ss #72

e673612

using pgvector/pgvector:pg16 docker in NRT github actions tests after…

a98114f

… @grololo advice

remove blank line to force github actions that weren't trigge by chan…

0b8d146

…ge in their config

Victor Reutenauer and others added 13 commits February 19, 2024 11:56

change version migration after following different advice

e34102b

Merge branch 'sim_search_b2f' of github.com:ecotaxa/ecotaxa_back into…

7068f40

… sim_search_b2f

remove useless migration create and drop linked to 'scories' with R i…

f48d3a2

…n pg

remove 'scories' due to different management of password in prod and …

3707c44

…dev (from my own's understanding)

script migrate data cnnfeature vector

f1c90fe

# Conflicts:

fb7b430

# .github/workflows/auto_tests.yml

fix: upgrade script for new pgvector table

b60f723

Merge branch 'master' into sim_search_b2f

190852f

remove useless endpoint ss

cf68094

protect some call in ss and change test to use existing endpoint and …

df5ad9b

…not ea endpoint that wasn't used in prod

Update test_similarity_search.py

73174f9

trying to set up the test before the rebase

Merge branch 'feat/sim_search_b2f_backup_before_rebase' into sim_sear…

d8e6fb9

…ch_merge_integration_0524

Merge pull request #80 from ecotaxa/sim_search_merge_integration_0524

73fbbf6

VR essai de merge de master dans ss

moilerat had a problem deploying to Env_for_Actions May 17, 2024 17:36 — with GitHub Actions Failure

grololo06 reviewed Jun 16, 2024

View reviewed changes

grololo06 reviewed Jun 17, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TO BE PUSHED ONCE test on niko and complex-test are OK as well as DB migration Feat/sim search b2f backup before rebase #81

TO BE PUSHED ONCE test on niko and complex-test are OK as well as DB migration Feat/sim search b2f backup before rebase #81

moilerat commented May 17, 2024

grololo06 left a comment

grololo06 Jun 16, 2024

grololo06 Jun 16, 2024

grololo06 Jun 16, 2024

grololo06 Jun 16, 2024

grololo06 Jun 16, 2024

grololo06 Jun 16, 2024

grololo06 Jun 16, 2024

grololo06 Jun 16, 2024

grololo06 Jun 16, 2024

grololo06 Jun 16, 2024

grololo06 Jun 17, 2024

		@@ -0,0 +1,12 @@
		ecotaxa4=# CREATE TABLE obj_cnn_features_vector AS SELECT objcnnid, ARRAY[cnn01, cnn02, cnn03, cnn04, cnn05, cnn06, cnn07, cnn08, cnn09, cnn10, cnn11, cnn12, cnn13, cnn14, cnn15, cnn16, cnn17, cnn18, cnn19, cnn20, cnn21, cnn22, cnn23, cnn24, cnn25, cnn26, cnn27, cnn28, cnn29, cnn30, cnn31, cnn32, cnn33, cnn34, cnn35, cnn36, cnn37, cnn38, cnn39, cnn40, cnn41, cnn42, cnn43, cnn44, cnn45, cnn46, cnn47, cnn48, cnn49, cnn50] as features FROM obj_cnn_features;

TO BE PUSHED ONCE test on niko and complex-test are OK as well as DB migration Feat/sim search b2f backup before rebase #81

Are you sure you want to change the base?

TO BE PUSHED ONCE test on niko and complex-test are OK as well as DB migration Feat/sim search b2f backup before rebase #81

Conversation

moilerat commented May 17, 2024

grololo06 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment