feat: autocomplete entity/query/schema

[sshanzel]

Introduced the entity to handle generic autocompletes as discussed.

It was also mentioned that we should be moving other autocomplete queries to this new schema. To make it easier to review, I created a separate PR to clearly see the only changes made for migration: #3208

Jira ticket

MI-1064

[pulumi]

🍹 The Update (preview) for dailydotdev/api/prod (at 73b10e3) was successful.

Resource Changes

    Name                                                   Type                                  Operation
+-  vpc-native-k8s-secret                                  kubernetes:core/v1:Secret             create-replacement
~   vpc-native-check-analytics-report-cron                 kubernetes:batch/v1:CronJob           update
~   vpc-native-sync-subscription-with-cio-cron             kubernetes:batch/v1:CronJob           update
+   vpc-native-api-clickhouse-migration-5180160c           kubernetes:batch/v1:Job               create
~   vpc-native-update-trending-cron                        kubernetes:batch/v1:CronJob           update
-   api-sub-api.extract-cv-markdown                        gcp:pubsub/subscription:Subscription  delete
~   vpc-native-hourly-notification-cron                    kubernetes:batch/v1:CronJob           update
~   vpc-native-post-analytics-clickhouse-cron              kubernetes:batch/v1:CronJob           update
~   vpc-native-clean-zombie-user-companies-cron            kubernetes:batch/v1:CronJob           update
~   vpc-native-update-source-tag-view-cron                 kubernetes:batch/v1:CronJob           update
~   vpc-native-clean-stale-user-transactions-cron          kubernetes:batch/v1:CronJob           update
-   vpc-native-api-clickhouse-migration-92dcdefe           kubernetes:batch/v1:Job               delete
~   vpc-native-update-tags-str-cron                        kubernetes:batch/v1:CronJob           update
~   vpc-native-clean-gifted-plus-cron                      kubernetes:batch/v1:CronJob           update
~   vpc-native-update-tag-recommendations-cron             kubernetes:batch/v1:CronJob           update
~   vpc-native-generic-referral-reminder-cron              kubernetes:batch/v1:CronJob           update
~   vpc-native-calculate-top-readers-cron                  kubernetes:batch/v1:CronJob           update
~   vpc-native-update-current-streak-cron                  kubernetes:batch/v1:CronJob           update
~   vpc-native-ws-deployment                               kubernetes:apps/v1:Deployment         update
~   vpc-native-personalized-digest-deployment              kubernetes:apps/v1:Deployment         update
~   vpc-native-generate-search-invites-cron                kubernetes:batch/v1:CronJob           update
~   vpc-native-temporal-deployment                         kubernetes:apps/v1:Deployment         update
~   vpc-native-daily-digest-cron                           kubernetes:batch/v1:CronJob           update
~   vpc-native-update-highlighted-views-cron               kubernetes:batch/v1:CronJob           update
~   vpc-native-personalized-digest-cron                    kubernetes:batch/v1:CronJob           update
~   vpc-native-post-analytics-history-day-clickhouse-cron  kubernetes:batch/v1:CronJob           update
~   vpc-native-bg-deployment                               kubernetes:apps/v1:Deployment         update
~   vpc-native-private-deployment                          kubernetes:apps/v1:Deployment         update
+   vpc-native-api-db-migration-5180160c                   kubernetes:batch/v1:Job               create
~   vpc-native-validate-active-users-cron                  kubernetes:batch/v1:CronJob           update
~   vpc-native-clean-zombie-users-cron                     kubernetes:batch/v1:CronJob           update
~   vpc-native-update-source-public-threshold-cron         kubernetes:batch/v1:CronJob           update
~   vpc-native-update-views-cron                           kubernetes:batch/v1:CronJob           update
~   vpc-native-clean-zombie-images-cron                    kubernetes:batch/v1:CronJob           update
~   vpc-native-deployment                                  kubernetes:apps/v1:Deployment         update
-   vpc-native-api-db-migration-92dcdefe                   kubernetes:batch/v1:Job               delete

[sshanzel]

Just to allow importing a specific entity only.

[capJavert]

nice idea, you can also use parseArgs from node:util + zod to avoid using raw process argv, check assignCores script, not blocking

[sshanzel]

Updated. Nicely done on type-strict script!

[sshanzel]

Not too much for local import. This is about:

• Degree: ~100.
• Field of study: ~200.
• Role: ~1000.

[capJavert]

nice idea, you can also use parseArgs from node:util + zod to avoid using raw process argv, check assignCores script, not blocking

[capJavert]

Suggested change

	const { data, error } = autocompleteSchema.safeParse(payload);
	const { data } = autocompleteSchema.parse(payload);

you can just use parse, API will now convert the error thrown by it automatically to zod error, no need to throw ValidationError below

[rebelchris]

Just want to align on the autocomplete nuances.

[rebelchris]

From the RFC is wasn't really sure what datasets would be excluded from this.
It's only location for now?
https://dailydotdev.atlassian.net/wiki/spaces/HAN/pages/1816657933/RFC-146+-+Profile+4.0+Backend?focusedCommentId=1828716556

What makes us say "this is now too big" and then will it be a pain to remove?
I just want to avoid us having to eventually make seperate datasets later vs doing it now.

Autocomplete endpoint could still be uniform but take "Type" of dataset into account?
LMWYT

CC: @nensidosari

[sshanzel]

@rebelchris when it involves a separate entity, we won't store that here (location and company). If you only need a simple string from a single source (opposite example: location requires looking up three columns), we use this table.

Autocomplete endpoint could still be uniform but take "Type" of dataset into account?

Autocompletions that involve a separate entity normally require a more complicated query; hence, I think it is better to be separated (similar to keyword).

[sshanzel]

Even if the data grows very large for a specific type, as long as we only require a single column string for the lookup, we will store them here.

[rebelchris]

I won't block the PR, but so many unknowns, duplicating datasets seems way safer would be my point :)
(at no clear downside)

but again not blocking, just have a feeling sooner then later we'll get to this.

[omBratteng]

It's possible to create a partitioned table if we want.

-- Parent partitioned table
CREATE TABLE autocomplete (
  value text NOT NULL,
  type text NOT NULL,
  "createdAt" TIMESTAMP NOT NULL DEFAULT now(),
  enabled boolean NOT NULL DEFAULT true,
  CONSTRAINT "PK_autocomplete_value_type" PRIMARY KEY (value, type)
) PARTITION BY LIST (type);

-- Partitions for known types
CREATE TABLE autocomplete_type_user PARTITION OF autocomplete
  FOR VALUES IN ('user');

CREATE TABLE autocomplete_type_repo PARTITION OF autocomplete
  FOR VALUES IN ('repo');

-- Optional catch-all so inserts for new types do not fail
CREATE TABLE autocomplete_type_default PARTITION OF autocomplete
  DEFAULT;

-- Indexes (per partition where needed)
CREATE INDEX autocomplete_user_value_idx ON autocomplete_type_user (value);
CREATE INDEX autocomplete_repo_value_idx ON autocomplete_type_repo (value);
-- For the default partition, consider a composite for common lookups
CREATE INDEX autocomplete_default_type_value_idx
  ON autocomplete_type_default (type, value);

[sshanzel]

@nensidosari there are quite a few types that are pretty much static (if not all), and we don't expect the data to grow as large that it impacts performance. For example, the degrees list (even all three types we don't expect to grow as large as locations/company). There are techniques on how to handle large data in a single table; one example is what Ole-Martin said, which is to horizontally partition the table. It would be prematurely optimizing the table if we do that now (and just adds maintenance overhead for all the extra entities for each type and new ones we would introduce).

Current state of the size: #3207 (comment) - it would only be the role that would probably need more, but I doubt it would need more than 10k especially our focus is the tech field (although we will add common roles outside of the field for people who transitioned - other than that, they can simply type in the value).

[sshanzel]

@omBratteng iirc, typeorm works just fine on partitioned tables as if you are just running a query on a regular table.

[sshanzel]

@rebelchris totally agree, it can be a nightmare when we get to a point where the data reaches millions (then move the data left and right). Thankfully, for now, we don't have that type yet.

[rebelchris]

@sshanzel but what's the downside of doing the seperated tables now?
It feels like a safer/easy thing.

Im just not seeing why we wouldn't do it.

[sshanzel]

By prematurely optimizing, we are just adding a maintenance overhead while we don't clearly anticipate any upcoming type that will give us the trouble. Another option is to do horizontal partitioning over additional entities, but that can be for another discussion. I just think it is not worth creating duplicated two-column tables just for the type, on a straightforward use case. Imho we proceed with what we agreed upon in the RFC.

Let's say we want to add ranking by popularity or occurrences; we'll then have to implement it on three tables. Now, depending on the process of how we increment these values, we'll have to repeat it across all tables (whether through specific mutations or CDC, the required change compounds).

That's just one change of improvement.

[capJavert]

toLowerCase, lowercase is a validator to make sure input is lowercase

[capJavert]

@sshanzel can you add a test(s) with invalid value, like emoji, spaces and so on?

[sshanzel]

Me to Cursor:

[sshanzel]

Should now have more tests to cover all other cases. Reference commit: 6e28e2d

[rebelchris]

As discussed not blocking.
Still prefer myself the split tables so just putting it for reference.

[omBratteng]

YOLO