Search on es #3093

stereosteve · 2022-05-13T19:49:59Z

Description

full/search/full on ES
test against frontend
relevance tuning, share with ray/forrest
introduce new ENV variable to enable search: audius_elasticsearch_search_enabled
fallback to sql on error
autocomplete
other entity searches (tracks, etc.)
tests
script with assertions against production data

Completes PLAT-130

Tests

Using json-diff tool (npm i -g json-diff):

DISCPROV='http://x.x.x.x:5000'

time curl -s -H "X-User-ID: 2" "$DISCPROV/feed?limit=20" > es_feed.json
time curl -s "$DISCPROV/v1/full/search/full?limit=3&offset=0&query=isaac+dawn&user_id=aNzoj" > sql_user.json
time curl -s "$DISCPROV/v1/full/search/full?limit=3&offset=0&query=isaac+dawn" > sql_anon.json
time curl -s "$DISCPROV/v1/tracks/search?limit=15&query=isaac+dawn" > sql_anon.json
time curl -s "$DISCPROV/v1/playlists/search?limit=15&query=isaac+dawn" > sql_anon.json
time curl -s "$DISCPROV/v1/users/search?limit=15&query=isaac+dawn" > sql_anon.json

json-diff es_user.json sql_user.json | less

discovery-provider/src/utils/elasticdsl.py

discovery-provider/.test.env

discovery-provider/compose/docker-compose.backend.yml

jowlee · 2022-05-26T17:35:34Z

discovery-provider/es-indexer/src/indexers/BaseIndexer.ts

+    logger.info('refreshing index: ' + indexName)
+    await es.indices.refresh({ index: indexName })


does this take a while?
If already logging, might be nice to record the time it takes as well?

it's a few ms each index so probably ok without logging. it was added here since we extended the refresh_interval to 5s. an edge case would be when the indexer first catches up we want to refresh before making it available otherwise there could be no results for a few seconds. it's also necessary for test cases since we don't want to wait 5s.

discovery-provider/es-indexer/src/indexers/PlaylistIndexer.ts

discovery-provider/es-indexer/src/indexers/UserIndexer.ts

discovery-provider/es-indexer/src/types/docs.ts

jowlee · 2022-05-26T18:15:11Z

discovery-provider/src/utils/elasticdsl.py

+        + to_wei(user.get("associated_sol_wallets_balance", "0") or 0)
+        + to_wei(user.get("waudio", "0") or 0)


I believe we need to multiply the solana wormhole audio by 10^10

i think that's what to_wei does right? similar thing happens here https://github.com/AudiusProject/audius-protocol/blob/master/discovery-provider/src/queries/get_balances.py#L69

this looks right. @sddioulde can you confirm for us?

discovery-provider/src/utils/elasticdsl.py

raymondjacobson

This is so great. I don't want to be critical review path here, but it looks great to me.

raymondjacobson · 2022-05-26T23:33:29Z

discovery-provider/es-indexer/src/indexers/TrackIndexer.ts

@@ -85,19 +117,19 @@ export class TrackIndexer extends BaseIndexer<TrackDoc> {
    -- etl tracks
    select 
      tracks.*,
-      (tracks.download->>'is_downloadable')::boolean as is_downloadable,
+      (tracks.download->>'is_downloadable')::boolean as downloadable,


why this change? just curious

that's the naming we use in the response currently

"downloadable": false, "download": { "cid": null, "is_downloadable": false, "requires_follow": false },

discovery-provider/es-indexer/src/types/docs.ts

discovery-provider/scripts/search_quality.py

raymondjacobson · 2022-05-27T00:04:59Z

discovery-provider/src/queries/search_es.py

+
+    # users: finalize
+    for k in ["users", "followed_users"]:
+        users = drop_copycats(response[k])


just so i understand, we're well-protected here b/c of overfetch yeah?

No that's a good catch. I will add some over-fetching to user limit

raymondjacobson · 2022-05-27T00:05:49Z

discovery-provider/src/queries/search_es.py

+                "fields": [
+                    "suggest",
+                    "suggest._2gram",
+                    "suggest._3gram",


just curious does 2gram give us a lot in addition to having trigrams?

I was a bit confused about how search_as_you_type worked at first.

By default it uses the default tokenizer (words) and then builds ngrams with those word tokens. In the past I've always used ngrams to describe letter couplets and shingles to describe word couplets... but in current config I think these ngrams fields are working with word tokens

So for example with you took that photo:

2grams: you took took that that photo

3grams: you took that took that photo

This still helps with phrase matching because the default suggest will match if any of the words appear in any order... but additional matches on 2grams and 3grams will happen if words appear in same order in the query and document, so more relevant results will rank higher. At least that's my current understanding... the search_as_you_type docs are a bit sparse haha.

We were considering if search_as_you_type was giving us much over just using a standard analyzer, but it seemed to be working well enough, so we kept using it. Since these fields are short (a handful of terms) the overhead of building these 2gram and 3gram subfields is negligible.

But might revisit in future mapping changes.

yea that's how i understand it too and why i think track title artist vs artist track title ordering slightly matters.

raymondjacobson · 2022-05-27T00:06:29Z

discovery-provider/src/utils/elasticdsl.py

+        + to_wei(user.get("associated_sol_wallets_balance", "0") or 0)
+        + to_wei(user.get("waudio", "0") or 0)


this looks right. @sddioulde can you confirm for us?

stereosteve · 2022-06-15T15:49:23Z

discovery-provider/src/api/v1/users.py

-        users = response["users"]
-        users = list(map(extend_user, users))
-        return success_response(users)
+        return success_response(response)


should be like:

response = search(search_args) return success_response(response['users'])

Co-authored-by: Saliou Diallo <saliou@audius.co>

[e804aa1] [C-2248, C-2373] Use playlistUpdates, remove legacyNotifications (#3094) Dylan Jeffers [824933e] [C-2366] Improve web notification selection performance (#3103) Dylan Jeffers [4b8edef] [PLAT-696] Add trending-playlists/underground notifications (#3089) Dylan Jeffers [1f9cf3e] [C-2275] Fix android drawer offsets (#3095) Dylan Jeffers [fc14c82] [PAY-1063][PAY-1085][PAY-1086] Update UI for inaccessible gated tracks from favorites and history pages (#3100) Saliou Diallo [b0441f5] [C-2365] Update play buttons on web and mobile to show resume when track is current (#3101) Kyle Shanks [453910f] [C-2378] Add upload v2 feature flag (#3099) Sebastian Klingler [962a6df] [C-2337] Remove reachability mobile web (#3090) Raymond Jacobson [4ad5cd2] Fix visible collectibles for upload popup (#3093) Saliou Diallo [c143078] Fix feature flag bug (#3092) Saliou Diallo [44435b5] Fix upload prompt modal learn more url (#3091) Saliou Diallo [c9024ad] Use chat.messagesStatus instead of selector (#3087) Reed [38d43c4] [C-2369] Fix issue where notification poll can break app on signout (#3088) Dylan Jeffers [90122d9] [PAY-923] DMs: Add desktop entrypoints (#3083) Marcus Pasell [00f27e8] [PAY-907] Mobile chat reactions (#3020) Reed [4678b89] DMs: Fix broken typecheck on main (#3086) Marcus Pasell [756ade4] [PAY-1000][PAY-1084][PAY-1096][PAY-1097][PAY-1098] - More gated content fixes (#3085) Saliou Diallo [820aa9d] Fix upload and repost probers tests and lint (#3076) Sebastian Klingler [345607e] [C-2320] Fix profile socials alignment (#3079) Dylan Jeffers [569199c] Fix prod build timeout (#3084) Sebastian Klingler [12f6c22] Remove ports for local dev (#3082) Theo Ilie [1940618] Fix broken Main build due to typeerror (#3080) Marcus Pasell [eb8d47e] [PAY-1082] DMs: Dedupe sent messages (#3066) Marcus Pasell [50a11c3] Update SDK to 2.0.3-beta.0 (#3078) Marcus Pasell [c420fbb] Clean up NPM package lock (#3077) Marcus Pasell [35d1124] [C-2327] Add playlist updates slice (#3063) Dylan Jeffers [59862ad] [C-2344] Update the web playbar scrubber to respect the playback speed of podcasts (#3075) Kyle Shanks [ffeb0d3] [C-2349] Default download on wifi only to false (#3074) Andrew Mendelsohn [cafae41] [C-2325] Fix playlist table date-added column (#3073) Dylan Jeffers [384a510] [PAY-927] DMs: Empty messages state (#3068) Marcus Pasell [1132f83] Update @jup-ag/core to 2.0.0-beta.9 (#3072) Marcus Pasell [49c0ebf] [PAY-1072] Change "Download App" icon on Settings Page (#3067) Marcus Pasell [928dcaf] [PAY-1056] - More gated content updates and fixes (#3070) Saliou Diallo [1e1f769] [C-2345] Move PlaybackRate drawer to common drawers map (#3071) Kyle Shanks [f5d1251] Fix web-dist CI steps (#3069) Sebastian Klingler [5f89800] Fix heavy rotation playlist on client (#3056) sabrina-kiam [c0191e2] [C-2316] Add remote config for all oauth verification (#3052) Raymond Jacobson [40f5627] [PAY-1074][PAY-1075][PAY-1076][PAY-1080] - Update availability settings states + more QA fixes (#3059) Saliou Diallo [5be60ac] [C-2339] Update podcast control updates to also work for audiobooks (#3065) Kyle Shanks [163ebf5] [C-2297] Add fallback flag to podcast feature (#3064) Sebastian Klingler [f206391] [PAY-904] - Add gated content upload prompt (#3057) Saliou Diallo [1afc4e5] [C-1344] Move probers to monorepo and make tests pass (#3061) Sebastian Klingler [e198279] Remove random line (#3062) Saliou Diallo [24a001b] Add playback position logic for mobile (#3051) Kyle Shanks [d210124] [PAY-1070] Update TabSlider/SegmentedControl slider size on resize (#3044) Marcus Pasell

stereosteve added 2 commits May 13, 2022 16:48

es: track search

8d3e9f8

es: multisearch tracks, users, playlists

51a813c

pull-request-size bot added the size/L label May 13, 2022

stereosteve and others added 5 commits May 13, 2022 20:14

fix es search with no user_id present

d8bb8a3

track search updates almost complete set

e462a00

more track search fields

58b9cc5

search hydrates user from ES, does user + playlist search

7697950

search populates followee_reposts and favorites

03c278f

pull-request-size bot added size/XL and removed size/L labels May 17, 2022

stereosteve and others added 6 commits May 18, 2022 14:10

tracks: index comma separated tags correctly

aed280f

bug fix none balance

c7fa076

bug fix the balance indexing

f7d60a2

fix search query artist to user and add took

e6ca303

Merge branch 'master' into search-on-es

a970f67

add env var and fallback

7115b88

pull-request-size bot added size/L and removed size/XL labels May 19, 2022

stereosteve and others added 3 commits May 19, 2022 14:58

skip mat view refresh if ES search enabled

37c022b

default all search

500cb95

add saved entities and albums search, needs refactoring

d753f1a

pull-request-size bot added size/XL and removed size/L labels May 19, 2022

stereosteve added 3 commits May 20, 2022 13:13

Use ES for autocomplete

d381ce9

deep copy query dsl before changing

e95d7e2

search_as_you_type for autocomplete

6d39fb7

stereosteve force-pushed the search-on-es branch from aedf734 to 6d39fb7 Compare May 20, 2022 21:01

isaacsolo added 2 commits May 22, 2022 06:05

add integration tests against ES searches and fix some bugs

0d6d3af

fix test fixtures + add index refresh

2369a06

pull-request-size bot removed the size/XL label May 23, 2022

stereosteve commented May 25, 2022

View reviewed changes

discovery-provider/src/utils/elasticdsl.py Outdated Show resolved Hide resolved

isaacsolo and others added 4 commits May 25, 2022 19:32

update playlist query

581e5a9

cleanup comments

8e740bc

refactor es dsl building code

623884a

add fuzziness

a7d927d

stereosteve marked this pull request as ready for review May 26, 2022 15:08

stereosteve requested review from piazzatron and hareeshnagaraj May 26, 2022 16:26

Add note about current_user_followee_follow_count

6cd2b7e

isaacsolo requested a review from raymondjacobson May 26, 2022 17:27

stereosteve requested review from jowlee and removed request for hareeshnagaraj and piazzatron May 26, 2022 17:28

jowlee reviewed May 26, 2022

View reviewed changes

discovery-provider/.test.env Outdated Show resolved Hide resolved

jowlee reviewed May 26, 2022

View reviewed changes

discovery-provider/src/utils/elasticdsl.py Outdated Show resolved Hide resolved

isaacsolo added 5 commits May 26, 2022 18:42

edit test env

015487c

small naming fixes

2132760

lint

d3cfe6f

dedupe user type

b0a6642

Merge branch 'master' into search-on-es

b060030

raymondjacobson approved these changes May 27, 2022

View reviewed changes

stereosteve added 2 commits May 27, 2022 14:42

lower + asciifolding for all analyzed fields

1729393

Add some over-fetching on user search for drop_copycats

169ebf5

stereosteve merged commit 8f9fe4c into master May 27, 2022

stereosteve deleted the search-on-es branch May 27, 2022 16:23

stereosteve commented Jun 15, 2022

View reviewed changes

sliptype pushed a commit that referenced this pull request Sep 10, 2023

Fix visible collectibles for upload popup (#3093)

5dbe91c

Co-authored-by: Saliou Diallo <saliou@audius.co>

AudiusProject deleted a comment from linear bot Sep 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Search on es #3093

Search on es #3093

stereosteve commented May 13, 2022 •

edited

Loading

jowlee May 26, 2022

isaacsolo May 26, 2022

jowlee May 26, 2022

isaacsolo May 26, 2022

raymondjacobson May 27, 2022

raymondjacobson left a comment

raymondjacobson May 26, 2022

isaacsolo May 27, 2022

raymondjacobson May 27, 2022

stereosteve May 27, 2022

raymondjacobson May 27, 2022

stereosteve May 27, 2022

isaacsolo May 27, 2022

raymondjacobson May 27, 2022

stereosteve Jun 15, 2022

		logger.info('refreshing index: ' + indexName)
		await es.indices.refresh({ index: indexName })

		+ to_wei(user.get("associated_sol_wallets_balance", "0") or 0)
		+ to_wei(user.get("waudio", "0") or 0)

Search on es #3093

Search on es #3093

Conversation

stereosteve commented May 13, 2022 • edited Loading

Description

Tests

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raymondjacobson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stereosteve commented May 13, 2022 •

edited

Loading