Sync with langgenius (#3)

* feat: increase read timeout of OpenAI Compatible API, Ollama, Nvidia LLM (langgenius#3538)

* feat: agent log (langgenius#3537)

Co-authored-by: jyong <718720800@qq.com>

* fix: typo of PublishConfig (langgenius#3540)

* fix: workflow delete edge (langgenius#3541)

* feat: filter empty content messages in llm node (langgenius#3547)

* fix: json-reader-json-output (langgenius#3552)

* fix: tool node show output text variable type error (langgenius#3556)

* feat: economical index support retrieval testing (langgenius#3563)

* optimize question classifier prompt and support keyword hit test (langgenius#3565)

* fix event/stream ping (langgenius#3553)

* enhance: preload general packages (langgenius#3567)

* added claude 3 opus (langgenius#3545)

* feat: code (langgenius#3557)

* feat: add workflow api in Node.js sdk (langgenius#3584)

* Fix: use debounce for switch (langgenius#3585)

* fix: json in raw text sometimes changed back to key value in HTTP node (langgenius#3586)

* test: add scripts for running tests on api module both locally and CI jobs (langgenius#3497)

* add-open-mixtral-8x22b (langgenius#3591)

* docs: Update README.md (langgenius#3577)

* enhance:speedup xinference embedding & rerank  (langgenius#3587)

* fix(openai_api_compatible): fixing the error when converting chunk to json (langgenius#3570)

* feat: stable diffusion 3 (langgenius#3599)

* Feat/enterprise sso (langgenius#3602)

* Add mixtral 8x22b (langgenius#3606)

* fix: copy invite link has duplicated origin (langgenius#3608)

* seucirty: http smuggling (langgenius#3609)

* chore: apply ruff rules on tests and app.py (langgenius#3605)

* feat: Vision switch functionality is provided on OpenRouter (langgenius#3564)

* get dict key indexing_technique in DocumentAddByFileApi (langgenius#3615)

Co-authored-by: songqijun <songqijun@qipeng.com>

* fix: chat rename (langgenius#3627)

* feat: moonshot fc (langgenius#3629)

* add-llama3-for-nvidia-api-catalog (langgenius#3631)

* content fix to continue (langgenius#3633)

Co-authored-by: xiaohan <fuck@qq.com>

* Fix error in [Update yaml and py file in Tavily Tool] (langgenius#3465)

Co-authored-by: Yeuoly <admin@srmxy.cn>

* feat: add file log (langgenius#3612)

Co-authored-by: liuzhenghua-jk <liuzhenghua-jk@360shuke.com>

* fix: validate languages (langgenius#3638)

* Fix problem with scroll inside chat window (langgenius#3578)

* fix: in alembic's offline mode (db migrate with --sql option), skip data operations (langgenius#3533)

* fix: workflow_run_id not log_id in workflow api doc (langgenius#3658)

* Optimize README_CN (langgenius#3660)

* fix: delete tool parameters cache when sync draft workflow for run workflow use new parameter change in draft workflow  (langgenius#3637)

* python 3.12 support (langgenius#3652)

* version to 0.6.4 (langgenius#3670)

---------

Co-authored-by: takatost <takatost@users.noreply.github.com>
Co-authored-by: KVOJJJin <jzongcode@gmail.com>
Co-authored-by: jyong <718720800@qq.com>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: zxhlyh <jasonapring2015@outlook.com>
Co-authored-by: Yeuoly <45712896+Yeuoly@users.noreply.github.com>
Co-authored-by: Joel <iamjoel007@gmail.com>
Co-authored-by: Jyong <76649700+JohnJyong@users.noreply.github.com>
Co-authored-by: liuzhenghua <1090179900@qq.com>
Co-authored-by: Siddharth Jain <137015071+tellsiddh@users.noreply.github.com>
Co-authored-by: Joshua <138381132+joshua20231026@users.noreply.github.com>
Co-authored-by: Matheus Mondaini <matheus.mondaini@outlook.com>
Co-authored-by: 呆萌闷油瓶 <253605712@qq.com>
Co-authored-by: aniaan <hi@aniaan.dev>
Co-authored-by: Garfield Dai <dai.hai@foxmail.com>
Co-authored-by: jeessy2 <6205259+jeessy2@users.noreply.github.com>
Co-authored-by: sqj8899 <sqj8899@126.com>
Co-authored-by: songqijun <songqijun@qipeng.com>
Co-authored-by: fuckqqcom <9391575+fuckqqcom@users.noreply.github.com>
Co-authored-by: xiaohan <fuck@qq.com>
Co-authored-by: Richards Tu <142148415+richards199999@users.noreply.github.com>
Co-authored-by: Yeuoly <admin@srmxy.cn>
Co-authored-by: liuzhenghua-jk <liuzhenghua-jk@360shuke.com>
Co-authored-by: YidaHu <huyidada@gmail.com>
Co-authored-by: rmmedia <125268006+rmmedia-pl@users.noreply.github.com>
Co-authored-by: saga.rey <saga.rey@outlook.com>
Co-authored-by: xin.gao <34891602+gaoxin-pen@users.noreply.github.com>

.github/workflows/api-tests.yml

@@ -8,6 +8,9 @@ on:
 jobs:
   test:
     runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
 
     env:
       OPENAI_API_KEY: sk-IamNotARealKeyJustForMockTestKawaiiiiiiiiii
@@ -37,10 +40,10 @@ jobs:
         with:
           packages: ffmpeg
 
-      - name: Set up Python
+      - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v5
         with:
-          python-version: '3.10'
+          python-version: ${{ matrix.python-version }}
           cache: 'pip'
           cache-dependency-path: |
             ./api/requirements.txt
@@ -50,10 +53,10 @@ jobs:
         run: pip install -r ./api/requirements.txt -r ./api/requirements-dev.txt
 
       - name: Run ModelRuntime
-        run: pytest api/tests/integration_tests/model_runtime/anthropic api/tests/integration_tests/model_runtime/azure_openai api/tests/integration_tests/model_runtime/openai api/tests/integration_tests/model_runtime/chatglm api/tests/integration_tests/model_runtime/google api/tests/integration_tests/model_runtime/xinference api/tests/integration_tests/model_runtime/huggingface_hub/test_llm.py
+        run: dev/pytest/pytest_model_runtime.sh
 
       - name: Run Tool
-        run: pytest api/tests/integration_tests/tools/test_all_provider.py
+        run: dev/pytest/pytest_tools.sh
 
       - name: Run Workflow
-        run: pytest api/tests/integration_tests/workflow
+        run: dev/pytest/pytest_workflow.sh

README.md

@@ -33,8 +33,8 @@
   <a href="./README_CN.md"><img alt="Commits last month" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
   <a href="./README_JA.md"><img alt="Commits last month" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
   <a href="./README_ES.md"><img alt="Commits last month" src="https://img.shields.io/badge/Español-d9d9d9"></a>
-  <a href="./README_KL.md"><img alt="Commits last month" src="https://img.shields.io/badge/Français-d9d9d9"></a>
-  <a href="./README_FR.md"><img alt="Commits last month" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
+  <a href="./README_FR.md"><img alt="Commits last month" src="https://img.shields.io/badge/Français-d9d9d9"></a>
+  <a href="./README_KL.md"><img alt="Commits last month" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
 </p>
 
 #

README_CN.md

@@ -44,19 +44,19 @@
   <a href="https://trendshift.io/repositories/2152" target="_blank"><img src="https://trendshift.io/api/badge/repositories/2152" alt="langgenius%2Fdify | 趋势转变" style="width: 250px; height: 55px;" width="250" height="55"/></a>
 </div>
 
-Dify 是一个开源的LLM应用开发平台。其直观的界面结合了AI工作流程、RAG管道、代理功能、模型管理、可观察性功能等，让您可以快速从原型到生产。以下是其核心功能列表：
+Dify 是一个开源的 LLM 应用开发平台。其直观的界面结合了 AI 工作流、RAG 管道、Agent、模型管理、可观测性功能等，让您可以快速从原型到生产。以下是其核心功能列表：
 </br> </br>
 
 **1. 工作流**: 
-  在视觉画布上构建和测试功能强大的AI工作流程，利用以下所有功能以及更多功能。
+  在画布上构建和测试功能强大的 AI 工作流程，利用以下所有功能以及更多功能。
 
 
   https://github.com/langgenius/dify/assets/13230914/356df23e-1604-483d-80a6-9517ece318aa
 
 
 
 **2. 全面的模型支持**: 
-  与数百种专有/开源LLMs以及数十种推理提供商和自托管解决方案无缝集成，涵盖GPT、Mistral、Llama2以及任何与OpenAI API兼容的模型。完整的支持模型提供商列表可在[此处](https://docs.dify.ai/getting-started/readme/model-providers)找到。
+  与数百种专有/开源 LLMs 以及数十种推理提供商和自托管解决方案无缝集成，涵盖 GPT、Mistral、Llama3 以及任何与 OpenAI API 兼容的模型。完整的支持模型提供商列表可在[此处](https://docs.dify.ai/getting-started/readme/model-providers)找到。
 
 ![providers-v5](https://github.com/langgenius/dify/assets/13230914/5a17bdbe-097a-4100-8363-40255b70f6e3)
 
@@ -65,16 +65,16 @@ Dify 是一个开源的LLM应用开发平台。其直观的界面结合了AI工
   用于制作提示、比较模型性能以及向基于聊天的应用程序添加其他功能（如文本转语音）的直观界面。
 
 **4. RAG Pipeline**: 
-  广泛的RAG功能，涵盖从文档摄入到检索的所有内容，支持从PDF、PPT和其他常见文档格式中提取文本的开箱即用的支持。
+  广泛的 RAG 功能，涵盖从文档摄入到检索的所有内容，支持从 PDF、PPT 和其他常见文档格式中提取文本的开箱即用的支持。
 
 **5. Agent 智能体**: 
-  您可以基于LLM函数调用或ReAct定义代理，并为代理添加预构建或自定义工具。Dify为AI代理提供了50多种内置工具，如谷歌搜索、DELL·E、稳定扩散和WolframAlpha等。
+  您可以基于 LLM 函数调用或 ReAct 定义 Agent，并为 Agent 添加预构建或自定义工具。Dify 为 AI Agent 提供了50多种内置工具，如谷歌搜索、DELL·E、Stable Diffusion 和 WolframAlpha 等。
 
 **6. LLMOps**: 
-  随时间监视和分析应用程序日志和性能。您可以根据生产数据和注释持续改进提示、数据集和模型。
+  随时间监视和分析应用程序日志和性能。您可以根据生产数据和标注持续改进提示、数据集和模型。
 
 **7. 后端即服务**: 
-  所有Dify的功能都带有相应的API，因此您可以轻松地将Dify集成到自己的业务逻辑中。
+  所有 Dify 的功能都带有相应的 API，因此您可以轻松地将 Dify 集成到自己的业务逻辑中。
 
 
 ## 功能比较
@@ -84,21 +84,21 @@ Dify 是一个开源的LLM应用开发平台。其直观的界面结合了AI工
     <th align="center">Dify.AI</th>
     <th align="center">LangChain</th>
     <th align="center">Flowise</th>
-    <th align="center">OpenAI助理API</th>
+    <th align="center">OpenAI Assistant API</th>
   </tr>
   <tr>
     <td align="center">编程方法</td>
     <td align="center">API + 应用程序导向</td>
-    <td align="center">Python代码</td>
+    <td align="center">Python 代码</td>
     <td align="center">应用程序导向</td>
-    <td align="center">API导向</td>
+    <td align="center">API 导向</td>
   </tr>
   <tr>
-    <td align="center">支持的LLMs</td>
+    <td align="center">支持的 LLMs</td>
     <td align="center">丰富多样</td>
     <td align="center">丰富多样</td>
     <td align="center">丰富多样</td>
-    <td align="center">仅限OpenAI</td>
+    <td align="center">仅限 OpenAI</td>
   </tr>
   <tr>
     <td align="center">RAG引擎</td>
@@ -108,21 +108,21 @@ Dify 是一个开源的LLM应用开发平台。其直观的界面结合了AI工
     <td align="center">✅</td>
   </tr>
   <tr>
-    <td align="center">代理</td>
+    <td align="center">Agent</td>
     <td align="center">✅</td>
     <td align="center">✅</td>
     <td align="center">✅</td>
     <td align="center">✅</td>
   </tr>
   <tr>
-    <td align="center">工作流程</td>
+    <td align="center">工作流</td>
     <td align="center">✅</td>
     <td align="center">❌</td>
     <td align="center">✅</td>
     <td align="center">❌</td>
   </tr>
   <tr>
-    <td align="center">可观察性</td>
+    <td align="center">可观测性</td>
     <td align="center">✅</td>
     <td align="center">✅</td>
     <td align="center">❌</td>
@@ -202,7 +202,7 @@ docker compose up -d
 ## Contributing
 
 对于那些想要贡献代码的人，请参阅我们的[贡献指南](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md)。
-同时，请考虑通过社交媒体、活动和会议来支持Dify的分享。
+同时，请考虑通过社交媒体、活动和会议来支持 Dify 的分享。
 
 > 我们正在寻找贡献者来帮助将Dify翻译成除了中文和英文之外的其他语言。如果您有兴趣帮助，请参阅我们的[i18n README](https://github.com/langgenius/dify/blob/main/web/i18n/README.md)获取更多信息，并在我们的[Discord社区服务器](https://discord.gg/8Tpq4AcN9c)的`global-users`频道中留言。
 

api/README.md

@@ -55,3 +55,16 @@
 9. If you need to debug local async processing, please start the worker service by running 
 `celery -A app.celery worker -P gevent -c 1 --loglevel INFO -Q dataset,generation,mail`.
 The started celery app handles the async tasks, e.g. dataset importing and documents indexing.
+
+
+## Testing
+
+1. Install dependencies for both the backend and the test environment
+   ```bash
+   pip install -r requirements.txt -r requirements-dev.txt
+   ``` 
+
+2. Run the tests locally with mocked system environment variables in `tool.pytest_env` section in `pyproject.toml`
+   ```bash
+   dev/pytest/pytest_all_tests.sh
+   ```

api/app.py

@@ -1,4 +1,6 @@
 import os
+import sys
+from logging.handlers import RotatingFileHandler
 
 if not os.environ.get("DEBUG") or os.environ.get("DEBUG").lower() != 'true':
     from gevent import monkey
@@ -17,10 +19,13 @@
 
 from flask import Flask, Response, request
 from flask_cors import CORS
-
 from werkzeug.exceptions import Unauthorized
+
 from commands import register_commands
 from config import CloudEditionConfig, Config
+
+# DO NOT REMOVE BELOW
+from events import event_handlers
 from extensions import (
     ext_celery,
     ext_code_based_extension,
@@ -37,11 +42,8 @@
 from extensions.ext_database import db
 from extensions.ext_login import login_manager
 from libs.passport import PassportService
-from services.account_service import AccountService
-
-# DO NOT REMOVE BELOW
-from events import event_handlers
 from models import account, dataset, model, source, task, tool, tools, web
+from services.account_service import AccountService
 
 # DO NOT REMOVE ABOVE
 
@@ -86,7 +88,25 @@ def create_app(test_config=None) -> Flask:
 
     app.secret_key = app.config['SECRET_KEY']
 
-    logging.basicConfig(level=app.config.get('LOG_LEVEL', 'INFO'))
+    log_handlers = None
+    log_file = app.config.get('LOG_FILE')
+    if log_file:
+        log_dir = os.path.dirname(log_file)
+        os.makedirs(log_dir, exist_ok=True)
+        log_handlers = [
+            RotatingFileHandler(
+                filename=log_file,
+                maxBytes=1024 * 1024 * 1024,
+                backupCount=5
+            ),
+            logging.StreamHandler(sys.stdout)
+        ]
+    logging.basicConfig(
+        level=app.config.get('LOG_LEVEL'),
+        format=app.config.get('LOG_FORMAT'),
+        datefmt=app.config.get('LOG_DATEFORMAT'),
+        handlers=log_handlers
+    )
 
     initialize_extensions(app)
     register_blueprints(app)
@@ -115,7 +135,7 @@ def initialize_extensions(app):
 @login_manager.request_loader
 def load_user_from_request(request_from_flask_login):
     """Load user based on the request."""
-    if request.blueprint == 'console':
+    if request.blueprint in ['console', 'inner_api']:
         # Check if the user_id contains a dot, indicating the old format
         auth_header = request.headers.get('Authorization', '')
         if not auth_header:
@@ -151,6 +171,7 @@ def unauthorized_handler():
 def register_blueprints(app):
     from controllers.console import bp as console_app_bp
     from controllers.files import bp as files_bp
+    from controllers.inner_api import bp as inner_api_bp
     from controllers.service_api import bp as service_api_bp
     from controllers.web import bp as web_bp
 
@@ -188,6 +209,8 @@ def register_blueprints(app):
          )
     app.register_blueprint(files_bp)
 
+    app.register_blueprint(inner_api_bp)
+
 
 # create app
 app = create_app()

api/config.py

@@ -38,6 +38,9 @@
     'QDRANT_CLIENT_TIMEOUT': 20,
     'CELERY_BACKEND': 'database',
     'LOG_LEVEL': 'INFO',
+    'LOG_FILE': '',
+    'LOG_FORMAT': '%(asctime)s.%(msecs)03d %(levelname)s [%(threadName)s] [%(filename)s:%(lineno)d] - %(message)s',
+    'LOG_DATEFORMAT': '%Y-%m-%d %H:%M:%S',
     'HOSTED_OPENAI_QUOTA_LIMIT': 200,
     'HOSTED_OPENAI_TRIAL_ENABLED': 'False',
     'HOSTED_OPENAI_TRIAL_MODELS': 'gpt-3.5-turbo,gpt-3.5-turbo-1106,gpt-3.5-turbo-instruct,gpt-3.5-turbo-16k,gpt-3.5-turbo-16k-0613,gpt-3.5-turbo-0613,gpt-3.5-turbo-0125,text-davinci-003',
@@ -69,6 +72,8 @@
     'TOOL_ICON_CACHE_MAX_AGE': 3600,
     'MILVUS_DATABASE': 'default',
     'KEYWORD_DATA_SOURCE_TYPE': 'database',
+    'INNER_API': 'False',
+    'ENTERPRISE_ENABLED': 'False',
 }
 
 
@@ -99,12 +104,15 @@ def __init__(self):
         # ------------------------
         # General Configurations.
         # ------------------------
-        self.CURRENT_VERSION = "0.6.3"
+        self.CURRENT_VERSION = "0.6.4"
         self.COMMIT_SHA = get_env('COMMIT_SHA')
         self.EDITION = "SELF_HOSTED"
         self.DEPLOY_ENV = get_env('DEPLOY_ENV')
         self.TESTING = False
         self.LOG_LEVEL = get_env('LOG_LEVEL')
+        self.LOG_FILE = get_env('LOG_FILE')
+        self.LOG_FORMAT = get_env('LOG_FORMAT')
+        self.LOG_DATEFORMAT = get_env('LOG_DATEFORMAT')
 
         # The backend URL prefix of the console API.
         # used to concatenate the login authorization callback or notion integration callback.
@@ -133,6 +141,11 @@ def __init__(self):
         # Alternatively you can set it with `SECRET_KEY` environment variable.
         self.SECRET_KEY = get_env('SECRET_KEY')
 
+        # Enable or disable the inner API.
+        self.INNER_API = get_bool_env('INNER_API')
+        # The inner API key is used to authenticate the inner API.
+        self.INNER_API_KEY = get_env('INNER_API_KEY')
+
         # cors settings
         self.CONSOLE_CORS_ALLOW_ORIGINS = get_cors_allow_origins(
             'CONSOLE_CORS_ALLOW_ORIGINS', self.CONSOLE_WEB_URL)
@@ -327,6 +340,8 @@ def __init__(self):
         self.TOOL_ICON_CACHE_MAX_AGE = get_env('TOOL_ICON_CACHE_MAX_AGE')
 
         self.KEYWORD_DATA_SOURCE_TYPE = get_env('KEYWORD_DATA_SOURCE_TYPE')
+        self.ENTERPRISE_ENABLED = get_bool_env('ENTERPRISE_ENABLED')
+
 
 class CloudEditionConfig(Config):
 

api/controllers/__init__.py

@@ -1,4 +1,3 @@
-# -*- coding:utf-8 -*-
 
 
 

api/controllers/console/__init__.py

@@ -1,22 +1,57 @@
 from flask import Blueprint
+
 from libs.external_api import ExternalApi
 
 bp = Blueprint('console', __name__, url_prefix='/console/api')
 api = ExternalApi(bp)
 
 # Import other controllers
-from . import admin, apikey, extension, feature, setup, version, ping
+from . import admin, apikey, extension, feature, ping, setup, version
+
 # Import app controllers
-from .app import (advanced_prompt_template, annotation, app, audio, completion, conversation, generator, message,
-                  model_config, site, statistic, workflow, workflow_run, workflow_app_log, workflow_statistic, agent)
+from .app import (
+    advanced_prompt_template,
+    agent,
+    annotation,
+    app,
+    audio,
+    completion,
+    conversation,
+    generator,
+    message,
+    model_config,
+    site,
+    statistic,
+    workflow,
+    workflow_app_log,
+    workflow_run,
+    workflow_statistic,
+)
+
 # Import auth controllers
 from .auth import activate, data_source_oauth, login, oauth
+
 # Import billing controllers
 from .billing import billing
+
 # Import datasets controllers
 from .datasets import data_source, datasets, datasets_document, datasets_segments, file, hit_testing
+
+# Import enterprise controllers
+from .enterprise import enterprise_sso
+
 # Import explore controllers
-from .explore import (audio, completion, conversation, installed_app, message, parameter, recommended_app,
-                      saved_message, workflow)
+from .explore import (
+    audio,
+    completion,
+    conversation,
+    installed_app,
+    message,
+    parameter,
+    recommended_app,
+    saved_message,
+    workflow,
+)
+
 # Import workspace controllers
-from .workspace import account, members, model_providers, models, tool_providers, workspace
+from .workspace import account, members, model_providers, models, tool_providers, workspace