Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

✨ feat: support server db mode with Postgres / Drizzle ORM / tRPC #2556

Merged
merged 2 commits into from
Jun 16, 2024
Merged

✨ feat: support server db mode with Postgres / Drizzle ORM / tRPC #2556

merged 2 commits into from
Jun 16, 2024
+14,663 −227

Conversation

arvinxx
Copy link
Contributor

@arvinxx arvinxx commented May 18, 2024 •
edited

💻 变更类型 | Change Type

  • ✨ feat
  • 🐛 fix
  • ♻️ refactor
  • 💄 style
  • 🔨 chore
  • ⚡️ perf
  • 📝 docs

🔀 变更说明 | Description of Change

整体进展

LobeChat 服务端 DB 版本功能实现，目前完成度为 100% 。

Note

由于之前并没有写过 SQL 和 Postgres ，所以可能整体服务端实现上仍然存在瑕疵，也可能存在 critical 的问题， 欢迎大家点评指正。

技术栈

  • DB：Postgres ，默认采用 Neon；
  • ORM: Dirzzle ORM
  • Auth: Clerk
  • server router: tRPC

由于基于 Drizzle ORM， 非常期待看到未来拓展到其他 Postgres DB、甚至 SQLite、MySQL 。 以及有待 @cy948 帮忙实现 next-auth 相关的 db 鉴权方案。

待解决工程问题

工程链路上还有两个问题待解决：

服务端相关的配置

# Postgres DB Url
DATABASE_URL=

# 指定 service mode 为 server 
NEXT_PUBLIC_SERVICE_MODE=server

# Clerk 相关配置
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=
CLERK_SECRET_KEY=
CLERK_WEBHOOK_SECRET=

# S3 相关配置（上传图片必须）
NEXT_PUBLIC_S3_DOMAIN=

S3_ACCESS_KEY_ID=
S3_SECRET_ACCESS_KEY=
S3_ENDPOINT=
S3_BUCKET=

📝 补充信息 | Additional Information

@vercel Vercel
Copy link

vercel bot commented May 18, 2024 •
edited

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
lobe-chat-community ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jun 16, 2024 10:59am

@lobehubbot
Copy link
Member

👍 @arvinxx

Thank you for raising your pull request and contributing to our Community
Please make sure you have followed our contributing guidelines. We will review it as soon as possible.
If you encounter any problems, please feel free to connect with us.
非常感谢您提出拉取请求并为我们的社区做出贡献，请确保您已经遵循了我们的贡献指南，我们会尽快审查它。
如果您遇到任何问题，请随时与我们联系。

@arvinxx arvinxx changed the title ✨ feat: support server db with postgres ✨ feat: support server db with postgres / Drizzle / trpc May 18, 2024
@arvinxx arvinxx changed the title ✨ feat: support server db with postgres / Drizzle / trpc ✨ feat: support server db with postgres / Drizzle ORM / trpc May 18, 2024
@arvinxx arvinxx changed the title ✨ feat: support server db with postgres / Drizzle ORM / trpc ✨ feat: support server db mode with postgres / Drizzle ORM / trpc May 18, 2024
@arvinxx arvinxx changed the title ✨ feat: support server db mode with postgres / Drizzle ORM / trpc ✨ feat: support server db mode with postgres / Drizzle ORM / tRPC May 18, 2024
@arvinxx arvinxx changed the title ✨ feat: support server db mode with postgres / Drizzle ORM / tRPC ✨ feat: support server db mode with Postgres / Drizzle ORM / tRPC May 19, 2024
@cy948 cy948 mentioned this pull request May 21, 2024
Open
@cfwl1111
Copy link

目前是可以配置服务端数据库了吗

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Is it possible to configure the server database now?

@arvinxx
Copy link
Contributor Author

arvinxx commented May 24, 2024

目前是可以配置服务端数据库了吗

@cfwl1111 等这个 PR 合并后，就可以了

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Is it possible to configure the server database now?

@cfwl1111 After this PR is merged, it will be fine

@coderabbitai coderabbitai
Copy link

coderabbitai bot commented May 24, 2024

Important

Review Skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@lobehub lobehub deleted a comment from lobehubbot May 29, 2024
@cfwl1111
Copy link

目前是可以配置服务端数据库了吗

@cfwl1111 等这个 PR 合并后，就可以了

服务端 Postgres 准备好了，期待它的到来

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Is it possible to configure the server database now?

@cfwl1111 After this PR is merged, it will be fine

Server-side Postgres is ready and looking forward to its arrival

@lee88688
Copy link
Contributor

图片的保存可以考虑支持直接存储在PostgreSQL数据库中。自部署还需要单独起一个S3兼容的后端服务管理起来更复杂，特别是大部分自部署用户的使用强度不是那么高，使用数据库完全没问题。

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

The saving of pictures can be considered to be directly stored in the PostgreSQL database. Self-deployment also requires a separate S3-compatible back-end service, which is more complicated to manage, especially since most self-deployment users are not so intensive in usage, and there is no problem using the database.

@arvinxx
Copy link
Contributor Author

arvinxx commented May 31, 2024 •
edited

图片的保存可以考虑支持直接存储在PostgreSQL数据库中。自部署还需要单独起一个S3兼容的后端服务管理起来更复杂，特别是大部分自部署用户的使用强度不是那么高，使用数据库完全没问题。

不建议，后面还要支持各种文件上传的。如果不想承担这个复杂度，要么仍然可以用客户端数据库db的模式，不需要s3。 或者用未来我们host的cloud 版

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

The saving of images can be considered to be directly stored in the PostgreSQL database. Self-deployment also requires a separate S3-compatible back-end service, which is more complicated to manage, especially since most self-deployment users are not so intensive in usage, and there is no problem using the database.

Not recommended, it will support various file uploads later. If you don’t want to bear this complexity, you can either use the locally deployed version without s3, or use the cloud version of our host in the future.

@arvinxx arvinxx marked this pull request as ready for review June 4, 2024 12:32
@semantic-release-bot
🔖 chore(release): v0.162.25 [skip ci]
7789340 
### [Version&nbsp;0.162.25](v0.162.24...v0.162.25)
<sup>Released on **2024-06-16**</sup>

#### 🐛 Bug Fixes

- **misc**: Fix issues for client fetch.

<br/>

<details>
<summary><kbd>Improvements and Fixes</kbd></summary>

#### What's fixed

* **misc**: Fix issues for client fetch, closes [#2753](#2753) ([6f5be5d](6f5be5d))

</details>

<div align="right">

[![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top)

</div>
@arvinxx arvinxx changed the base branch from main to release/1.0 June 16, 2024 10:44
@arvinxx arvinxx merged commit d8bb6f4 into release/1.0 Jun 16, 2024
8 checks passed
@arvinxx arvinxx deleted the feat/server-db branch June 16, 2024 11:01
@lobehubbot
Copy link
Member

❤️ Great PR @arvinxx ❤️

The growth of project is inseparable from user feedback and contribution, thanks for your contribution! If you are interesting with the lobehub developer community, please join our discord and then dm @arvinxx or @canisminor1990. They will invite you to our private developer channel. We are talking about the lobe-chat development or sharing ai newsletter around the world.
项目的成长离不开用户反馈和贡献，感谢您的贡献! 如果您对 LobeHub 开发者社区感兴趣，请加入我们的 discord，然后私信 @arvinxx 或 @canisminor1990。他们会邀请您加入我们的私密开发者频道。我们将会讨论关于 Lobe Chat 的开发，分享和讨论全球范围内的 AI 消息。

arvinxx added a commit that referenced this pull request Jun 17, 2024
@arvinxx
✨ feat: support server db mode with Postgres / Drizzle ORM / tRPC (#2556
b26afbf 
)
github-actions bot pushed a commit that referenced this pull request Jun 17, 2024
@semantic-release-bot
🔖 chore(release): v0.163.0 [skip ci]
96e1789 
## [Version&nbsp;0.163.0](v0.162.25...v0.163.0)
<sup>Released on **2024-06-17**</sup>

#### ✨ Features

- **misc**: Support server db mode with Postgres / Drizzle ORM / tRPC.

<br/>

<details>
<summary><kbd>Improvements and Fixes</kbd></summary>

#### What's improved

* **misc**: Support server db mode with Postgres / Drizzle ORM / tRPC, closes [#2556](#2556) ([b26afbf](b26afbf))

</details>

<div align="right">

[![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top)

</div>
@xingpingcn
Copy link

这个新特性是不是在文档中没有更新？

客户端数据库db的模式

这个又应该如何配置？

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Is this new feature not updated in the documentation?

Schema of client database db

How should this be configured?

github-actions bot pushed a commit to bentwnghk/lobe-chat that referenced this pull request Jun 17, 2024
@semantic-release-bot
🔖 chore(release): v1.43.0 [skip ci]
0e03e19 
## [Version&nbsp;1.43.0](v1.42.19...v1.43.0)
<sup>Released on **2024-06-17**</sup>

#### ✨ Features

- **misc**: Release LobeChat 1.0, support server db mode with Postgres / Drizzle ORM / tRPC.

#### 💄 Styles

- **misc**: Remove r2 cdn url.

#### 🔧 Chores

- **misc**: Bump version.

<br/>

<details>
<summary><kbd>Improvements and Fixes</kbd></summary>

#### What's improved

* **misc**: Release LobeChat 1.0, closes [lobehub#2897](https://github.com/bentwnghk/lobe-chat/issues/2897) ([97bb377](97bb377))
* **misc**: Support server db mode with Postgres / Drizzle ORM / tRPC, closes [lobehub#2556](https://github.com/bentwnghk/lobe-chat/issues/2556) ([b26afbf](b26afbf))

#### Styles

* **misc**: Remove r2 cdn url, closes [lobehub#2878](https://github.com/bentwnghk/lobe-chat/issues/2878) ([19fd4d5](19fd4d5))

#### Chores

* **misc**: Bump version, closes [lobehub#2896](https://github.com/bentwnghk/lobe-chat/issues/2896) ([6e2d03b](6e2d03b))

</details>

#### 💥 BREAKING CHANGES

* **misc**: Update LICENSE to Apache 2.0
* **misc**: update to 1.0.0

<div align="right">

[![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top)

</div>
@ZeroDeng01
Copy link

CLERK_WEBHOOK_SECRET这个配置后是否需要去clerk配置Endpoint？我clerk的相关密钥已经做了配置，用户也可以登录，但是我查看数据库并没有新建clerk的用户信息，现在我的项目完全无法使用。
image
我查询日志，就是因为在数据库获取不到用户信息。应为users表为空：
image
image
@arvinxx

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

After configuring CLERK_WEBHOOK_SECRET, do I need to go to clerk to configure the Endpoint? The relevant keys of my clerk have been configured and users can log in, but when I checked the database, there was no new user information for the clerk. Now my project is completely unusable.
image
I checked the log because I couldn't get the user information in the database. The users table should be empty:
image
image
@arvinxx

@arvinxx
Copy link
Contributor Author

arvinxx commented Jun 17, 2024 •
edited

CLERK_WEBHOOK_SECRET这个配置后是否需要去clerk配置Endpoint？我clerk的相关密钥已经做了配置，用户也可以登录，但是我查看数据库并没有新建clerk的用户信息，现在我的项目完全无法使用。

需要的，类似这样：

image

@ZeroDeng01 @xingpingcn 目前文档还没完善，有问题可以在 Discord 里咨询，我们这几天会把部署指南补充完善

@cmj2002
Copy link

cmj2002 commented Jun 19, 2024

以及有待 @cy948 帮忙实现 next-auth 相关的 db 鉴权方案。

该部分工作是否有相关联的 issue/discussion/pull request ?

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

And waiting for @cy948 to help implement next-auth related db authentication scheme.

Is there an associated issue/discussion/pull request for this part of the work?

@arvinxx
Copy link
Contributor Author

arvinxx commented Jun 19, 2024

该部分工作是否有相关联的 issue/discussion/pull request ?

@cy948 感觉可以先写一篇 RFC 来讨论下？

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Is there an associated issue/discussion/pull request for this part of the work?

@cy948 I feel like I can write an RFC to discuss it first?

@cy948
Copy link
Contributor

cy948 commented Jun 19, 2024

@arvinxx 我试完第一个方案之后写个rfc描述下现在的情况吧

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

@arvinxx After I try the first solution, I will write an RFC to describe the current situation.

@gaarry
Copy link
Contributor

gaarry commented Jun 20, 2024

用上数据库了，朋友也注册我的网站，体验很棒～

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

I have used the database and my friends have also registered on my website. The experience is great~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
10 participants
@arvinxx @lobehubbot @cfwl1111 @lee88688 @xingpingcn @ZeroDeng01 @cmj2002 @cy948 @gaarry @semantic-release-bot