Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: related discussions refactor #9557

Merged
merged 63 commits into from Apr 10, 2024
Merged

feat: related discussions refactor #9557

merged 63 commits into from Apr 10, 2024

Conversation

mattkrick
Copy link
Member

@mattkrick mattkrick commented Mar 21, 2024 •
edited

Description

Adds support for embedding workflows.
The first workflow we had was an embed workflow, but since that workflow was just 1 step long, I didn't really know how to build it.

The RelatedDiscussions workflow is a few steps long, which forced me to do a refactor.
The goal here is to be able to add & edit workflows quickly with as many reusable steps as possible.

Demo

https://www.loom.com/share/d60e3cefb76948589a0ee2e03984e626?sid=bcf00f4f-645d-487e-9984-54b18a483a7f

jordanh and others added 30 commits February 27, 2024 15:04
@jordanh
feat: add embedder service
b3fa7ba
@jordanh
fix: type and class interface changes, assert AbstractModelCase commit 1
854f0ef
@jordanh
fix: code review updates
372e7c9
@jordanh
feat: added related discussions feature
97aae3b
@mattkrick
feat: support ad-hoc metadata inserts
d86eb2d 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
feat: triggers from metadata to queue
8628e3e 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
feat: move job queue to pg
285f69b 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
begin retry
0b024dd
@mattkrick
feat: support retries after queue goes dry
ae6126f 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
chore: separate abstract models into their own files
9fb9c4f 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
feat: support recursive text splitting for chunks
20f5d96 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
validate discussions to only include ended meetings
eff8eb9 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
support historical info
7ec3d6a 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
Merge branch 'master' into feat/embedder1
49c3eae
@mattkrick
feat: support listening to jobs from app
46dafac 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
feat: support calls from app
47e83d9
@mattkrick
merge master
7cee59e
@mattkrick
fix migration conflict
5c96c41
@mattkrick
fix: import history
49b349b 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
fix: dataloader mem leak
bf9a379 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
feat: handle stalled jobs
f7ccdc6
@mattkrick
self-review
b573083 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
remove redlock for non-historical jobs
ba906e0 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
feat: clean comments
10953b5 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
remove pubsub
7953939 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
fix lint errors
c2b6753 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
fix lint
8852ac2 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
cast to any for CI
2ba07c1 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
build embeddings table in CI for the typings
7bcd083 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@mattkrick
codecheck after server starts
368b34c 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 9, 2024

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 9, 2024

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

@mattkrick
Merge branch 'master' into feat/embedder-related-discussions1
0f88e71 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 9, 2024

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

@mattkrick
fix: prettier
743a93d 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 9, 2024

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

mattkrick
mattkrick commented Apr 9, 2024
View reviewed changes
packages/server/postgres/migrations/1712074131388_priority.ts
ALTER COLUMN "priority" TYPE INTEGER;
ALTER TABLE "EmbeddingsJobQueue"
ADD COLUMN "model" VARCHAR(255),
ADD COLUMN "embeddingsMetadataId" INTEGER,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i added this & model back in because the foreign key is nice to have for debugging. writing SQL against JSON is just a drag

@mattkrick
support instances that do not use embedder
0ce2075 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 9, 2024

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

@mattkrick mattkrick marked this pull request as ready for review April 9, 2024 20:41
@mattkrick
fix similar retro topics query
69e67b5 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@github-actions GitHub Actions
Copy link
Contributor

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

@mattkrick
fix: publishing from embedder
b465ee7 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@github-actions GitHub Actions
Copy link
Contributor

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

@mattkrick
only sleep for 10 seconds if no jobs are found
c57ff5c 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@github-actions GitHub Actions
Copy link
Contributor

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

@mattkrick
fix: simplify orchestrator API
06d68d3 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@github-actions GitHub Actions
Copy link
Contributor

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

@mattkrick
fix: use only 1 embedder for related discussions
07abeaf 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@github-actions GitHub Actions
Copy link
Contributor

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

@mattkrick
fix runtime error
c0cab43 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@github-actions GitHub Actions
Copy link
Contributor

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

@mattkrick
fix lint
3f4fdf3 
Signed-off-by: Matt Krick <matt.krick@gmail.com>
@github-actions GitHub Actions
Copy link
Contributor

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR will be delayed and might be rejected due to its size.

@mattkrick mattkrick merged commit 15a54fb into master Apr 10, 2024
6 checks passed
@mattkrick mattkrick deleted the feat/embedder-related-discussions1 branch April 10, 2024 23:26
@parabol-release-bot parabol-release-bot bot mentioned this pull request Apr 10, 2024
Merged
@github-actions github-actions bot mentioned this pull request Apr 10, 2024
Closed
24 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
size/xl
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@mattkrick @jordanh