feat: embedder service #9417

jordanh · 2024-02-09T15:42:28Z

Description

Cleaned up for company, this is the embedder service that is being readied for merge.

While there are a few enhancements that I'd like to make in short order, this PR intends only to target our local development environments. When you review, keep your eye on that rather than asking, "is this ready for production?"

Enhancements to make after this PR merges:

Added OpenAI embedding and GPT API support (rather than local dev services)
Resume events that are stuck in a queued state
Made fully event-driven (rather than use a polling loop)

👉 Consider started your review by reading the README, then beginning your code review here

Demo

📺 https://www.loom.com/share/524994e87dc34c5db360d3045d576e62?sid=789e6b1e-9ef2-4d18-9f96-ee4476426753

Testing scenarios

Requirements

Checkout and run https://github.com/ParabolInc/node-llama-cpp-text-generation-interface
Update your .env from .env.example
- Set: POSTGRES_USE_PGVECTOR=true
- Set: AI_EMBEDDER_ENABLED
- Set AI_EMBEDDING_MODELS
- Set: AI_GENERATION_MODELS
Add the relatedDiscussions feature flag to any org you'd like retro discussions to be embedded (see here)

Tests

Can run embedder service via yarn dev
- Observes rows being created in EmbeddingsMetadata
- Observes rows entering/leaving EmbededingsJobQueue
- Observes rows being created into configured embedding vector table, e.g. Embeddings_ember_1

Final checklist

I checked the code review guidelines
I have added Metrics Representative as reviewer(s) if my PR invovles metrics/data/analytics related changes
I have performed a self-review of my code, the same way I'd do it for any other team member
I have tested all cases I listed in the testing scenarios and I haven't found any issues or regressions
Whenever I took a non-obvious choice I added a comment explaining why I did it this way
I added the label Skip Maintainer Review Indicating the PR only requires reviewer review and can be merged right after it's approved if the PR introduces only minor changes, does not contain any architectural changes or does not introduce any new patterns and I think one review is sufficient'
PR title is human readable and could be used in changelog

github-actions · 2024-02-09T15:42:52Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

github-actions · 2024-02-09T15:56:06Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

package.json

jordanh · 2024-02-09T16:06:47Z

packages/embedder/ai_models/TextGenerationInterface.ts

@@ -0,0 +1,80 @@
+import {


A thought for the future:

In the future, we could have deployments with local LLMs. It might be nice for us to have a generic GPT interface that abstracts away whether or not we are using a local LLM or OpenAI, etc. and have the executor services consume that interface rather than call OpenAI directly

packages/embedder/embedder.ts

jordanh · 2024-02-09T16:10:28Z

packages/embedder/embedder.ts

+          continue
+        }
+      }
+      // console.log(`embedText: ${embedText}`)


uncomment me if you want to see lots of fun user data

packages/embedder/embedder.ts

jordanh · 2024-02-09T16:23:56Z

packages/embedder/indexing/orgIdsWithFeatureFlag.ts

+import getRethink from 'parabol-server/database/rethinkDriver'
+import {RDatum} from 'parabol-server/database/stricterR'
+
+export const orgIdsWithFeatureFlag = async () => {


⚠️ in order to test embedding, you must manually add the relatedDiscussions feature flag to the orgId. In order to keep this already XL PR smaller, that feature flag management was omitted

packages/server/dataloader/customLoaderMakers.ts

packages/server/dataloader/customRedisQueries.ts

jordanh · 2024-02-09T16:27:23Z

packages/server/postgres/getKysely.ts

@@ -10,6 +10,13 @@ const getKysely = () => {
      dialect: new PostgresDialect({
        pool: pg
      })
+      // query logging, if you'd like it:


This was helpful in development

jordanh · 2024-02-09T16:28:42Z

scripts/webpack/toolbox.config.js

@@ -56,6 +56,26 @@ module.exports = {
    new webpack.DefinePlugin({
      __PRODUCTION__: true
    })
+    // new CircularDependencyPlugin({


This was helpful for debugging why I was having difficulty with a webpack build at one point. Decided to leave the full config in here if it helps anybody else in the future

github-actions · 2024-02-09T16:55:34Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

github-actions · 2024-02-13T01:43:15Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

github-actions · 2024-02-13T21:28:41Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

github-actions · 2024-02-13T21:32:38Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

github-actions · 2024-02-13T21:35:38Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

github-actions · 2024-02-13T22:05:49Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

github-actions · 2024-02-19T20:46:55Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

github-actions · 2024-02-20T00:31:09Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

packages/embedder/ai_models/AbstractModel.ts

packages/embedder/ai_models/ModelManager.ts

packages/embedder/embedder.ts

packages/embedder/indexing/embeddingsTablesOps.ts

packages/embedder/embedder.ts

mattkrick · 2024-02-22T02:27:53Z

packages/embedder/indexing/retrospectiveDiscussionTopic.ts

+
+export async function refreshEmbeddingsMeta(dataLoader: DataLoaderWorker) {
+  const r = await getRethink()
+  const {createdAt: newestMeetingDate} = (await r


Help me understand further. If we put a hook in the main app that adds a row to the metadata table, that would make newestMetadata == Date.now(), which would then be used as the startDateTime so no more history would get processed, right?

mattkrick · 2024-02-26T23:08:48Z

packages/embedder/ai_models/helpers/fetchWithRetry.ts

+      let waitTime = retryAfter ? parseInt(retryAfter, 10) * 1000 : Math.pow(2, attempt) * 1000
+
+      // cap waitTime to prevent exceeding the deadline
+      waitTime = Math.min(waitTime, deadline.getTime() - Date.now())


+1 add a Math.max(0, because the while condition could go invalid after the await fetch, which would result in a negative value, which would cause it to run immediately. not a huge issue, but it'd be 1 extra fetch

mattkrick · 2024-02-27T00:03:21Z

just 1 little thing & a merge conflict. merge when ready!

Signed-off-by: Matt Krick <matt.krick@gmail.com>

github-actions · 2024-02-27T22:42:06Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

jordanh · 2024-02-27T22:42:18Z

Fixed up the yarn.lock conflict now. Just waiting on tests...

* chore(env vars): Stripe vars moved to the Integrations section (#9427) * chore: fix misleading `isLead` field name on `Team` (#9413) * chore: fix misleading `isLead` field name on `Team` The field indicates whether the viewer is the lead, but when used in a query for a different user, the result could be read wrong. * Fix Team.isLead dependencies * feat: remove team template limit (#9424) * update error message and increase template limit * remove max team template limits * remove canClone prop from CloneTemplate * remove unused threshold * remove unused threshold * feat: Add Google calendar meeting series for recurrence (#9380) * feat: Add recurrence to GCal events * Fun with timezones * fix: Increase the number of projects fetched per request from Atlassian (#9435) We ran into timeouts in `getAllProjects`, presumably because we're doing too many roundtrips. As a quick fix, increse the number of projects fetched per request from 50 to 500. * chore(deps): bump ip from 1.1.8 to 1.1.9 (#9442) Bumps [ip](https://github.com/indutny/node-ip) from 1.1.8 to 1.1.9. - [Commits](indutny/node-ip@v1.1.8...v1.1.9) --- updated-dependencies: - dependency-name: ip dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(release): release v7.17.0 (#9428) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * feat(standalone-deployment): Standalone host deployment improved and documented (#9445) * Docker compose stack improved * Remove unused containers from docker-compse and add useful comment on .env.example about PGSSLMODE * Docker compose profiles added. Documentation extended on how to use the profiles to manage the stack. * README fixed as docker compose up and down commands were not working * Typo fixed and docker-compose command replaced by docker compose * feat: support env-defined saml issuer for PPMIs (#9455) * feat: support env-defined saml issuer for PPMIs Signed-off-by: Matt Krick <matt.krick@gmail.com> * feat: support single SAML for entire tenant Signed-off-by: Matt Krick <matt.krick@gmail.com> --------- Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore: Associate logs with traces (#9444) * chore: Associate logs with traces Add trace information to log output for server side log statements. This does not include logging from code exclusively used for debugging, deploying or development. * Actually add the logger * Fix DD_LOGS_INJECTION check * chore(release): release v7.18.0 (#9450) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * chore: no force-push to prod (#9401) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.18.1 (#9459) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * feat: embedder service (#9417) * feat: add embedder service --------- Signed-off-by: Matt Krick <matt.krick@gmail.com> Co-authored-by: Matt Krick <matt.krick@gmail.com> * merge production to avoid force push (#9461) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.0 (#9460) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: checkout prod before merging it (#9463) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.1 (#9464) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: mrege origin/production strategy (#9465) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.2 (#9466) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: force push 5 (#9467) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.3 (#9468) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Matt Krick <matt.krick@gmail.com> Co-authored-by: Rafa <101704572+rafaelromcar-parabol@users.noreply.github.com> Co-authored-by: Georg Bremer <github@dschoordsch.de> Co-authored-by: Nick O'Ferrall <nickoferrall@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> Co-authored-by: Matt Krick <matt.krick@gmail.com> Co-authored-by: Jordan Husney <jordan.husney@gmail.com> Co-authored-by: github-actions <github-actions@github.com>

* chore(env vars): Stripe vars moved to the Integrations section (#9427) * chore: fix misleading `isLead` field name on `Team` (#9413) * chore: fix misleading `isLead` field name on `Team` The field indicates whether the viewer is the lead, but when used in a query for a different user, the result could be read wrong. * Fix Team.isLead dependencies * feat: remove team template limit (#9424) * update error message and increase template limit * remove max team template limits * remove canClone prop from CloneTemplate * remove unused threshold * remove unused threshold * feat: Add Google calendar meeting series for recurrence (#9380) * feat: Add recurrence to GCal events * Fun with timezones * fix: Increase the number of projects fetched per request from Atlassian (#9435) We ran into timeouts in `getAllProjects`, presumably because we're doing too many roundtrips. As a quick fix, increse the number of projects fetched per request from 50 to 500. * chore(deps): bump ip from 1.1.8 to 1.1.9 (#9442) Bumps [ip](https://github.com/indutny/node-ip) from 1.1.8 to 1.1.9. - [Commits](indutny/node-ip@v1.1.8...v1.1.9) --- updated-dependencies: - dependency-name: ip dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(release): release v7.17.0 (#9428) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * feat(standalone-deployment): Standalone host deployment improved and documented (#9445) * Docker compose stack improved * Remove unused containers from docker-compse and add useful comment on .env.example about PGSSLMODE * Docker compose profiles added. Documentation extended on how to use the profiles to manage the stack. * README fixed as docker compose up and down commands were not working * Typo fixed and docker-compose command replaced by docker compose * feat: support env-defined saml issuer for PPMIs (#9455) * feat: support env-defined saml issuer for PPMIs Signed-off-by: Matt Krick <matt.krick@gmail.com> * feat: support single SAML for entire tenant Signed-off-by: Matt Krick <matt.krick@gmail.com> --------- Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore: Associate logs with traces (#9444) * chore: Associate logs with traces Add trace information to log output for server side log statements. This does not include logging from code exclusively used for debugging, deploying or development. * Actually add the logger * Fix DD_LOGS_INJECTION check * chore(release): release v7.18.0 (#9450) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * chore: no force-push to prod (#9401) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.18.1 (#9459) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * feat: embedder service (#9417) * feat: add embedder service --------- Signed-off-by: Matt Krick <matt.krick@gmail.com> Co-authored-by: Matt Krick <matt.krick@gmail.com> * merge production to avoid force push (#9461) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.0 (#9460) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: checkout prod before merging it (#9463) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.1 (#9464) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: mrege origin/production strategy (#9465) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.2 (#9466) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: force push 5 (#9467) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.3 (#9468) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: limit invites from spammers (#9416) * fix: limit invites from spammers * update where we check pending emails * check total plus pending invites * use invitees instead of pending * fix: Fetch Jira projects in parallel (#9456) Previously we tried to fetch more projects per page, but Jira only ever returns 50 max. Instead once we know how many projects there are after fetching the first page, we fetch all remaining pages in parallel. * fix: replace lone surrogates in draft-js content (#9415) * fix: replace lone surrogates in draft-js content Signed-off-by: Matt Krick <matt.krick@gmail.com> * fix typo Signed-off-by: Matt Krick <matt.krick@gmail.com> * fix: eslint errors Signed-off-by: Matt Krick <matt.krick@gmail.com> --------- Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(deps): bump es5-ext from 0.10.62 to 0.10.64 (#9457) Bumps [es5-ext](https://github.com/medikoo/es5-ext) from 0.10.62 to 0.10.64. - [Release notes](https://github.com/medikoo/es5-ext/releases) - [Changelog](https://github.com/medikoo/es5-ext/blob/main/CHANGELOG.md) - [Commits](medikoo/es5-ext@v0.10.62...v0.10.64) --- updated-dependencies: - dependency-name: es5-ext dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: packages/server/package.json to reduce vulnerabilities (#9434) The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JS-UNDICI-6252336 Co-authored-by: snyk-bot <snyk-bot@snyk.io> * fix: packages/server/package.json to reduce vulnerabilities (#9392) The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JS-NODEMAILER-6219989 Co-authored-by: snyk-bot <snyk-bot@snyk.io> * fix: packages/server/package.json to reduce vulnerabilities (#9298) The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JS-FOLLOWREDIRECTS-6141137 Co-authored-by: snyk-bot <snyk-bot@snyk.io> * chore(deps): bump follow-redirects from 1.14.8 to 1.15.4 (#9312) Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.8 to 1.15.4. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](follow-redirects/follow-redirects@v1.14.8...v1.15.4) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Matt Krick <matt.krick@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Matt Krick <matt.krick@gmail.com> * chore: add upload to GCS step in ironbank (#9471) * add upload to GCS step in ironbank * update workflow name * chore(release): release v7.19.4 (#9470) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: Fix seasonal templates for leap years (#9476) * fix: Fix seasonal templates for leap years It would produce invalid dates on February 29th. * Master was not clean * chore(release): release v7.19.5 (#9477) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Matt Krick <matt.krick@gmail.com> Co-authored-by: Rafa <101704572+rafaelromcar-parabol@users.noreply.github.com> Co-authored-by: Georg Bremer <github@dschoordsch.de> Co-authored-by: Nick O'Ferrall <nickoferrall@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> Co-authored-by: Matt Krick <matt.krick@gmail.com> Co-authored-by: Jordan Husney <jordan.husney@gmail.com> Co-authored-by: adaniels-parabol <71724289+adaniels-parabol@users.noreply.github.com> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Co-authored-by: Dale Bumblis <135627447+dbumblis-parabol@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com>

* chore(env vars): Stripe vars moved to the Integrations section (#9427) * chore: fix misleading `isLead` field name on `Team` (#9413) * chore: fix misleading `isLead` field name on `Team` The field indicates whether the viewer is the lead, but when used in a query for a different user, the result could be read wrong. * Fix Team.isLead dependencies * feat: remove team template limit (#9424) * update error message and increase template limit * remove max team template limits * remove canClone prop from CloneTemplate * remove unused threshold * remove unused threshold * feat: Add Google calendar meeting series for recurrence (#9380) * feat: Add recurrence to GCal events * Fun with timezones * fix: Increase the number of projects fetched per request from Atlassian (#9435) We ran into timeouts in `getAllProjects`, presumably because we're doing too many roundtrips. As a quick fix, increse the number of projects fetched per request from 50 to 500. * chore(deps): bump ip from 1.1.8 to 1.1.9 (#9442) Bumps [ip](https://github.com/indutny/node-ip) from 1.1.8 to 1.1.9. - [Commits](indutny/node-ip@v1.1.8...v1.1.9) --- updated-dependencies: - dependency-name: ip dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(release): release v7.17.0 (#9428) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * feat(standalone-deployment): Standalone host deployment improved and documented (#9445) * Docker compose stack improved * Remove unused containers from docker-compse and add useful comment on .env.example about PGSSLMODE * Docker compose profiles added. Documentation extended on how to use the profiles to manage the stack. * README fixed as docker compose up and down commands were not working * Typo fixed and docker-compose command replaced by docker compose * feat: support env-defined saml issuer for PPMIs (#9455) * feat: support env-defined saml issuer for PPMIs Signed-off-by: Matt Krick <matt.krick@gmail.com> * feat: support single SAML for entire tenant Signed-off-by: Matt Krick <matt.krick@gmail.com> --------- Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore: Associate logs with traces (#9444) * chore: Associate logs with traces Add trace information to log output for server side log statements. This does not include logging from code exclusively used for debugging, deploying or development. * Actually add the logger * Fix DD_LOGS_INJECTION check * chore(release): release v7.18.0 (#9450) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * chore: no force-push to prod (#9401) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.18.1 (#9459) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * feat: embedder service (#9417) * feat: add embedder service --------- Signed-off-by: Matt Krick <matt.krick@gmail.com> Co-authored-by: Matt Krick <matt.krick@gmail.com> * merge production to avoid force push (#9461) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.0 (#9460) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: checkout prod before merging it (#9463) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.1 (#9464) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: mrege origin/production strategy (#9465) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.2 (#9466) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: force push 5 (#9467) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.3 (#9468) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: limit invites from spammers (#9416) * fix: limit invites from spammers * update where we check pending emails * check total plus pending invites * use invitees instead of pending * fix: Fetch Jira projects in parallel (#9456) Previously we tried to fetch more projects per page, but Jira only ever returns 50 max. Instead once we know how many projects there are after fetching the first page, we fetch all remaining pages in parallel. * fix: replace lone surrogates in draft-js content (#9415) * fix: replace lone surrogates in draft-js content Signed-off-by: Matt Krick <matt.krick@gmail.com> * fix typo Signed-off-by: Matt Krick <matt.krick@gmail.com> * fix: eslint errors Signed-off-by: Matt Krick <matt.krick@gmail.com> --------- Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(deps): bump es5-ext from 0.10.62 to 0.10.64 (#9457) Bumps [es5-ext](https://github.com/medikoo/es5-ext) from 0.10.62 to 0.10.64. - [Release notes](https://github.com/medikoo/es5-ext/releases) - [Changelog](https://github.com/medikoo/es5-ext/blob/main/CHANGELOG.md) - [Commits](medikoo/es5-ext@v0.10.62...v0.10.64) --- updated-dependencies: - dependency-name: es5-ext dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: packages/server/package.json to reduce vulnerabilities (#9434) The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JS-UNDICI-6252336 Co-authored-by: snyk-bot <snyk-bot@snyk.io> * fix: packages/server/package.json to reduce vulnerabilities (#9392) The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JS-NODEMAILER-6219989 Co-authored-by: snyk-bot <snyk-bot@snyk.io> * fix: packages/server/package.json to reduce vulnerabilities (#9298) The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JS-FOLLOWREDIRECTS-6141137 Co-authored-by: snyk-bot <snyk-bot@snyk.io> * chore(deps): bump follow-redirects from 1.14.8 to 1.15.4 (#9312) Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.8 to 1.15.4. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](follow-redirects/follow-redirects@v1.14.8...v1.15.4) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Matt Krick <matt.krick@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Matt Krick <matt.krick@gmail.com> * chore: add upload to GCS step in ironbank (#9471) * add upload to GCS step in ironbank * update workflow name * chore(release): release v7.19.4 (#9470) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: Fix seasonal templates for leap years (#9476) * fix: Fix seasonal templates for leap years It would produce invalid dates on February 29th. * Master was not clean * chore(release): release v7.19.5 (#9477) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: After parameter for meetingCount was ignored (#9479) * chore(docker-build): simplify the docker build process and reduce docker image size (#9447) * Dockerfile basic created. Improvements added to reduce build time and size (down from 795MB to 445MB, depending on systemtap). Readme reduced, removing the old process used to build the image. * basic-env file using a RethinkDB database name that is clearly dedicated to the building proces. * Readme improved to run all three components * Unused dockerfiles removed. Docker entrypoint renamed. Docker Readme adapted * Legacy build kept in both dockerfile and env file. Readme adapted to use the new basic image. Build GH workflow adapted to use the new basic.dockerfile. * chore(release): release v7.19.6 (#9480) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix(docker-build): home folder is /home/node now (#9482) * chore(release): release v7.19.7 (#9483) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Matt Krick <matt.krick@gmail.com> Co-authored-by: Rafa <101704572+rafaelromcar-parabol@users.noreply.github.com> Co-authored-by: Georg Bremer <github@dschoordsch.de> Co-authored-by: Nick O'Ferrall <nickoferrall@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> Co-authored-by: Matt Krick <matt.krick@gmail.com> Co-authored-by: Jordan Husney <jordan.husney@gmail.com> Co-authored-by: adaniels-parabol <71724289+adaniels-parabol@users.noreply.github.com> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Co-authored-by: Dale Bumblis <135627447+dbumblis-parabol@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com>

* chore(env vars): Stripe vars moved to the Integrations section (#9427) * chore: fix misleading `isLead` field name on `Team` (#9413) * chore: fix misleading `isLead` field name on `Team` The field indicates whether the viewer is the lead, but when used in a query for a different user, the result could be read wrong. * Fix Team.isLead dependencies * feat: remove team template limit (#9424) * update error message and increase template limit * remove max team template limits * remove canClone prop from CloneTemplate * remove unused threshold * remove unused threshold * feat: Add Google calendar meeting series for recurrence (#9380) * feat: Add recurrence to GCal events * Fun with timezones * fix: Increase the number of projects fetched per request from Atlassian (#9435) We ran into timeouts in `getAllProjects`, presumably because we're doing too many roundtrips. As a quick fix, increse the number of projects fetched per request from 50 to 500. * chore(deps): bump ip from 1.1.8 to 1.1.9 (#9442) Bumps [ip](https://github.com/indutny/node-ip) from 1.1.8 to 1.1.9. - [Commits](indutny/node-ip@v1.1.8...v1.1.9) --- updated-dependencies: - dependency-name: ip dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(release): release v7.17.0 (#9428) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * feat(standalone-deployment): Standalone host deployment improved and documented (#9445) * Docker compose stack improved * Remove unused containers from docker-compse and add useful comment on .env.example about PGSSLMODE * Docker compose profiles added. Documentation extended on how to use the profiles to manage the stack. * README fixed as docker compose up and down commands were not working * Typo fixed and docker-compose command replaced by docker compose * feat: support env-defined saml issuer for PPMIs (#9455) * feat: support env-defined saml issuer for PPMIs Signed-off-by: Matt Krick <matt.krick@gmail.com> * feat: support single SAML for entire tenant Signed-off-by: Matt Krick <matt.krick@gmail.com> --------- Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore: Associate logs with traces (#9444) * chore: Associate logs with traces Add trace information to log output for server side log statements. This does not include logging from code exclusively used for debugging, deploying or development. * Actually add the logger * Fix DD_LOGS_INJECTION check * chore(release): release v7.18.0 (#9450) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * chore: no force-push to prod (#9401) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.18.1 (#9459) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * feat: embedder service (#9417) * feat: add embedder service --------- Signed-off-by: Matt Krick <matt.krick@gmail.com> Co-authored-by: Matt Krick <matt.krick@gmail.com> * merge production to avoid force push (#9461) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.0 (#9460) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: checkout prod before merging it (#9463) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.1 (#9464) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: mrege origin/production strategy (#9465) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.2 (#9466) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: force push 5 (#9467) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.19.3 (#9468) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: limit invites from spammers (#9416) * fix: limit invites from spammers * update where we check pending emails * check total plus pending invites * use invitees instead of pending * fix: Fetch Jira projects in parallel (#9456) Previously we tried to fetch more projects per page, but Jira only ever returns 50 max. Instead once we know how many projects there are after fetching the first page, we fetch all remaining pages in parallel. * fix: replace lone surrogates in draft-js content (#9415) * fix: replace lone surrogates in draft-js content Signed-off-by: Matt Krick <matt.krick@gmail.com> * fix typo Signed-off-by: Matt Krick <matt.krick@gmail.com> * fix: eslint errors Signed-off-by: Matt Krick <matt.krick@gmail.com> --------- Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(deps): bump es5-ext from 0.10.62 to 0.10.64 (#9457) Bumps [es5-ext](https://github.com/medikoo/es5-ext) from 0.10.62 to 0.10.64. - [Release notes](https://github.com/medikoo/es5-ext/releases) - [Changelog](https://github.com/medikoo/es5-ext/blob/main/CHANGELOG.md) - [Commits](medikoo/es5-ext@v0.10.62...v0.10.64) --- updated-dependencies: - dependency-name: es5-ext dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: packages/server/package.json to reduce vulnerabilities (#9434) The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JS-UNDICI-6252336 Co-authored-by: snyk-bot <snyk-bot@snyk.io> * fix: packages/server/package.json to reduce vulnerabilities (#9392) The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JS-NODEMAILER-6219989 Co-authored-by: snyk-bot <snyk-bot@snyk.io> * fix: packages/server/package.json to reduce vulnerabilities (#9298) The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JS-FOLLOWREDIRECTS-6141137 Co-authored-by: snyk-bot <snyk-bot@snyk.io> * chore(deps): bump follow-redirects from 1.14.8 to 1.15.4 (#9312) Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.8 to 1.15.4. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](follow-redirects/follow-redirects@v1.14.8...v1.15.4) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Matt Krick <matt.krick@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Matt Krick <matt.krick@gmail.com> * chore: add upload to GCS step in ironbank (#9471) * add upload to GCS step in ironbank * update workflow name * chore(release): release v7.19.4 (#9470) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: Fix seasonal templates for leap years (#9476) * fix: Fix seasonal templates for leap years It would produce invalid dates on February 29th. * Master was not clean * chore(release): release v7.19.5 (#9477) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix: After parameter for meetingCount was ignored (#9479) * chore(docker-build): simplify the docker build process and reduce docker image size (#9447) * Dockerfile basic created. Improvements added to reduce build time and size (down from 795MB to 445MB, depending on systemtap). Readme reduced, removing the old process used to build the image. * basic-env file using a RethinkDB database name that is clearly dedicated to the building proces. * Readme improved to run all three components * Unused dockerfiles removed. Docker entrypoint renamed. Docker Readme adapted * Legacy build kept in both dockerfile and env file. Readme adapted to use the new basic image. Build GH workflow adapted to use the new basic.dockerfile. * chore(release): release v7.19.6 (#9480) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * fix(docker-build): home folder is /home/node now (#9482) * chore(release): release v7.19.7 (#9483) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> * feat: OpenAIGeneration model for embedder (#9474) * fix: support single-tenant saml record (#9486) Signed-off-by: Matt Krick <matt.krick@gmail.com> * chore(release): release v7.20.0 (#9485) Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Matt Krick <matt.krick@gmail.com> Co-authored-by: Rafa <101704572+rafaelromcar-parabol@users.noreply.github.com> Co-authored-by: Georg Bremer <github@dschoordsch.de> Co-authored-by: Nick O'Ferrall <nickoferrall@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: parabol-release-bot[bot] <150284312+parabol-release-bot[bot]@users.noreply.github.com> Co-authored-by: Matt Krick <matt.krick@gmail.com> Co-authored-by: Jordan Husney <jordan.husney@gmail.com> Co-authored-by: adaniels-parabol <71724289+adaniels-parabol@users.noreply.github.com> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Co-authored-by: Dale Bumblis <135627447+dbumblis-parabol@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com>

jordanh requested a review from mattkrick February 9, 2024 15:42

github-actions bot requested a review from tghanken February 9, 2024 15:42

github-actions bot added the size/xl label Feb 9, 2024

jordanh changed the title ~~Feat embedder service~~ feat: embedder service Feb 9, 2024

jordanh changed the title ~~feat: embedder service~~ feat: embedder service (base: feat-embedder-migration) Feb 9, 2024

jordanh commented Feb 9, 2024

View reviewed changes

package.json Show resolved Hide resolved

jordanh commented Feb 9, 2024

View reviewed changes

packages/embedder/embedder.ts Show resolved Hide resolved

jordanh commented Feb 9, 2024

View reviewed changes

packages/embedder/embedder.ts Show resolved Hide resolved

jordanh commented Feb 9, 2024

View reviewed changes

packages/embedder/embedder.ts Show resolved Hide resolved

jordanh commented Feb 9, 2024

View reviewed changes

packages/server/dataloader/customLoaderMakers.ts Show resolved Hide resolved

jordanh commented Feb 9, 2024

View reviewed changes

packages/server/dataloader/customRedisQueries.ts Show resolved Hide resolved

jordanh commented Feb 9, 2024

View reviewed changes

jordanh force-pushed the feat-embedder-service branch from e074663 to 86da386 Compare February 9, 2024 16:55

jordanh removed the request for review from tghanken February 9, 2024 16:56

jordanh force-pushed the feat-embedder-service branch from 86da386 to 192a4d0 Compare February 13, 2024 01:42

jordanh force-pushed the feat-embedder-service branch from 01ae7f2 to 4070815 Compare February 13, 2024 21:35

jordanh force-pushed the feat-embedder-service branch from 4070815 to eb4dd07 Compare February 13, 2024 22:05

jordanh force-pushed the feat-embedder-service branch from eb4dd07 to 5ac2b54 Compare February 13, 2024 22:23

jordanh force-pushed the feat-embedder-service branch from 61b456f to fcd0f16 Compare February 19, 2024 20:46

jordanh force-pushed the feat-embedder-service branch from fcd0f16 to 7ba1bf5 Compare February 20, 2024 00:30

jordanh force-pushed the feat-embedder-service branch 3 times, most recently from 3168a01 to 51a19f3 Compare February 21, 2024 20:31

mattkrick requested changes Feb 22, 2024

View reviewed changes

jordanh force-pushed the feat-embedder-service branch from d9dbe82 to 68d1304 Compare February 22, 2024 22:43

mattkrick approved these changes Feb 27, 2024

View reviewed changes

jordanh and others added 9 commits February 27, 2024 14:41

feat: add embedder service

d8f61f0

chore: add logs to prod.js

658ebe3

Signed-off-by: Matt Krick <matt.krick@gmail.com>

fix: rename AbstractModel.ts, fix case-sentitive import

f9cb23e

fix: pm2 prod config

147672c

fix: type and class interface changes, assert AbstractModelCase commit 1

ee5a46f

fix: final rename AbstractModel.ts

0782034

fix: code review updates

654d62b

fix: code review comments take 3

02057ca

fix: retrospectiveDiscussionTopic add to meta

c41943a

jordanh force-pushed the feat-embedder-service branch from dc34fd9 to c41943a Compare February 27, 2024 22:41

jordanh merged commit 55faa17 into master Feb 27, 2024
5 checks passed

jordanh deleted the feat-embedder-service branch February 27, 2024 22:54

parabol-release-bot bot mentioned this pull request Feb 27, 2024

chore(release): release v7.19.0 #9460

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: embedder service #9417

feat: embedder service #9417

jordanh commented Feb 9, 2024 •

edited

github-actions bot commented Feb 9, 2024

github-actions bot commented Feb 9, 2024

jordanh Feb 9, 2024

jordanh Feb 9, 2024

jordanh Feb 9, 2024

jordanh Feb 9, 2024

jordanh Feb 9, 2024

github-actions bot commented Feb 9, 2024

github-actions bot commented Feb 13, 2024

github-actions bot commented Feb 13, 2024

github-actions bot commented Feb 13, 2024

github-actions bot commented Feb 13, 2024

github-actions bot commented Feb 13, 2024

github-actions bot commented Feb 19, 2024

github-actions bot commented Feb 20, 2024

mattkrick Feb 22, 2024

mattkrick Feb 26, 2024

mattkrick commented Feb 27, 2024

github-actions bot commented Feb 27, 2024

jordanh commented Feb 27, 2024

feat: embedder service #9417

feat: embedder service #9417

Conversation

jordanh commented Feb 9, 2024 • edited

Description

Demo

Testing scenarios

Requirements

Tests

Final checklist

github-actions bot commented Feb 9, 2024

github-actions bot commented Feb 9, 2024

jordanh Feb 9, 2024

Choose a reason for hiding this comment

jordanh Feb 9, 2024

Choose a reason for hiding this comment

jordanh Feb 9, 2024

Choose a reason for hiding this comment

jordanh Feb 9, 2024

Choose a reason for hiding this comment

jordanh Feb 9, 2024

Choose a reason for hiding this comment

github-actions bot commented Feb 9, 2024

github-actions bot commented Feb 13, 2024

github-actions bot commented Feb 13, 2024

github-actions bot commented Feb 13, 2024

github-actions bot commented Feb 13, 2024

github-actions bot commented Feb 13, 2024

github-actions bot commented Feb 19, 2024

github-actions bot commented Feb 20, 2024

mattkrick Feb 22, 2024

Choose a reason for hiding this comment

mattkrick Feb 26, 2024

Choose a reason for hiding this comment

mattkrick commented Feb 27, 2024

github-actions bot commented Feb 27, 2024

jordanh commented Feb 27, 2024

jordanh commented Feb 9, 2024 •

edited