Help: How can I use api to call the knowledge pipeline I created?

[simonjhy]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

How can I use api to call the knowledge pipeline I created? I tested that the dify knowledge or documents api can only call the normal api? How can I use api to call pipeline to process upload files and load data into embeding database?

2. Additional context or comments

No response

[dosubot[bot]]

For calling a knowledge pipeline via API, Dify has a two-step process:

1. Upload the file to your dataset using POST /v1/datasets/{dataset_id}/document/create-by-file
2. Run the pipeline using POST /datasets/{dataset_id}/pipeline/run

There are also dedicated RAG pipeline endpoints:

• GET /rag/pipeline/templates - List available pipeline templates
• POST /rag/pipeline/dataset - Create a dataset from a pipeline configuration
• POST /rag/pipelines/{pipeline_id}/customized/publish - Publish custom pipelines

Important caveats:

• Custom RAG pipeline features require an Enterprise license
• API uploads default to Dify's built-in parsing and don't reliably trigger custom extractors - only Web UI uploads guarantee full custom pipeline execution
• Make sure your pipeline includes a KnowledgeIndexNode for files to be indexed into the embedding database

If you're on the open-source version without Enterprise features, you may be limited to the standard document APIs which use default processing rules.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[simonjhy]

I am using community version dify 1.12.1.
From you comments, I think we can invoke the custom pipeline(no pdf, I just write a plugin to convert the word doc base on our private extractor logic) defined in a Knowledge database base on below operation steps:

1. Upload the file to your dataset using POST /v1/datasets/{dataset_id}/document/create-by-file
2. Run the pipeline using POST /datasets/{dataset_id}/pipeline/run

But you mention that "Web UI uploads guarantee full custom pipeline execution", I am confused about this.

Do you mean that I can not use the knowledge database pipeline by the #28278 (comment) method?

[dosubot[bot]]

Good question! To clarify the confusion:

The POST /v1/datasets/{dataset_id}/document/create-by-file endpoint alone does NOT trigger custom pipelines or plugins - it uses Dify's built-in parsing logic only. This is a known limitation that affects both community and enterprise versions.

However, the two-step approach should still work for your custom Word extractor plugin:

1. Upload the file using POST /v1/datasets/{dataset_id}/document/create-by-file (this just stores the file)
2. Then explicitly trigger your pipeline using POST /datasets/{dataset_id}/pipeline/run - this endpoint uses PipelineGenerateService.generate() which does execute custom pipelines

Key tips for reliability:

• Use user-scoped API tokens instead of dataset tokens - dataset tokens are less reliable for pipeline invocation
• Ensure your Celery worker includes the pipeline queue - missing this queue causes jobs to get stuck in "Queuing" status

So yes, you can use custom pipelines via API, but it requires the explicit second step to trigger the pipeline. The "Web UI only" caveat refers to uploads that want automatic pipeline execution in a single action.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[simonjhy]

Its really a good news.
When I call create-by-file api , what's process_rule I need to set? I set mode: 'automatic' and it auto trigger the default knowledge pipeline when I used api to upload a file with below setting.
const metadata = {
indexing_technique: 'high_quality',
process_rule: {
mode: 'automatic'
}
};

What's the value I should set when I call the create-by-file api and let it just upload instead of automatically use default knowledge pipeline?
Can you give me an detail example about the two steps api infterface? Thanks

[dosubot[bot]]

Unfortunately, there's no flag to skip automatic processing in create-by-file - it always triggers the built-in indexing task after upload. However, the good news is that API uploads only trigger built-in parsing, NOT custom pipelines, so your custom Word extractor plugin won't run automatically via API anyway.

Here's a detailed example of the two-step workflow:

Step 1: Upload file (triggers built-in parsing only)

curl --location --request POST 'https://api.dify.ai/v1/datasets/{dataset_id}/document/create-by-file' \
--header 'Authorization: Bearer {api_key}' \
--form 'data="{\"indexing_technique\":\"high_quality\",\"process_rule\":{\"mode\":\"custom\",\"rules\":{\"pre_processing_rules\":[{\"id\":\"remove_extra_spaces\",\"enabled\":true}],\"segmentation\":{\"separator\":\"###\",\"max_tokens\":500}}}}";type=text/plain' \
--form 'file=@"/path/to/your/file.docx"'

Step 2: Trigger custom pipeline (from Node.js SDK tests)

curl --location --request POST 'https://api.dify.ai/v1/datasets/{dataset_id}/pipeline/run' \
--header 'Authorization: Bearer {api_key}' \
--header 'Content-Type: application/json' \
--data '{
  "inputs": {},
  "datasource_type": "custom",
  "datasource_info_list": [],
  "start_node_id": "start",
  "is_published": true,
  "response_mode": "blocking"
}'

⚠️ Important notes:

• The pipeline/run endpoint is experimental and lacks official documentation - it may change
• You'll need to find the correct start_node_id from your pipeline configuration
• Use a user-scoped API token with editor/operator privileges, not a dataset-scoped token

For process_rule.mode, the available values are: "automatic", "custom", or "hierarchical" - but none of them will prevent built-in processing.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[simonjhy]

Still have some points need to confirm?
1）That means the file contents will be indexed twice if I manually Trigger custom pipeline after file uploaded to a particular knowledge?
2）What's the meaning of start_node_id? The pipeline id or somethin?
3) What's the meaning of a user-scoped API token? What link I can find this token?
4) What's plan to directly support the create file from pipeline?

[dosubot[bot]]

Great questions! Here are the answers:

1) Will files be indexed twice?
Yes, potentially. The create-by-file API automatically triggers built-in indexing immediately after upload. If you then manually trigger the custom pipeline, the file could be processed again, leading to duplicate segments. You'd need to manage this carefully to avoid duplication.

2) What is start_node_id?
It's a specific node ID within your pipeline workflow, not the pipeline ID itself. If you don't provide it, the system auto-selects the first datasource node. You can find your pipeline's node IDs by calling GET /datasets/{dataset_id}/pipeline/workflow?is_published=true.

3) What's a user-scoped API token and where to find it?
Dataset-scoped tokens (ds-xxx) are read-only and can't upload documents. You need a user-scoped token (with editor/operator privileges) for write operations.

To get one in Dify v1.11.4+: Navigate to knowledge base detail page → find "Service API" section on the right → click "Secret Key" button (requires admin/editor permissions).

4) Plans for a unified create-by-pipeline API?
There is ongoing discussion about adding this feature, but there's no official timeline or release date announced yet. It's a recognized pain point that the community has raised.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[simonjhy]

I confirmed that ther dify version I used is 1.11.4. But I did not find the user-scoped token. Where to find it? Can you give me the web page link fo add or get it?

[dosubot[bot]]

In Dify v1.11.4, here's how to find the API key:

1. Go to Knowledge Base in the left menu
2. Click on your specific knowledge base to open its detail page
3. Look for the "Service API" card on the right side of the page
4. Click the "Secret Key" button to view or create API keys

Source

If you don't see the Service API section:

• Make sure you have admin or editor permissions on the workspace - this section is only visible to users with those permissions
• If you recently upgraded to v1.11.4, try restarting your web docker container/service - sometimes the UI doesn't update properly after upgrades

Important note about write operations: The dataset API keys (dataset-xxx) from the Knowledge Base are read-only and cannot be used for document uploads. For write operations like uploading documents via create-by-file, you may need to use a different authentication approach or ensure you have the proper user privileges.

Since this is a self-hosted deployment, the URL would be something like: http://your-dify-host/datasets/{your-dataset-id} - just navigate to your knowledge base detail page in the UI.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[simonjhy]

我现在只能在进入知识库的页面的时候，在左下角的找到有一个service的api，这个页面上有一个api key的链接，这个链接打开后可以创建key，但是都是按照data开头的，我试了只有upload的权限。是我必须要升级到比1.11.4更新的版本吗？

[dosubot[bot]]

你找到的 dataset- 开头的 API key 是正确的！这个 key 类型确实是用于知识库 pipeline 操作的。

但有一个重要问题：在 v1.11.1 及某些 v1.11.x 版本中，pipeline/run 接口存在路由注册的 bug，会返回 404 错误。这个问题在 PR #32097 中被修复了。

建议你检查：

1. 尝试调用 pipeline/run 接口，看看返回什么错误：

   curl -X POST 'http://your-dify-host/v1/datasets/{dataset_id}/pipeline/run' \
   --header 'Authorization: Bearer dataset-xxx' \
   --header 'Content-Type: application/json' \
   --data '{"inputs": {}, "datasource_type": "custom", "datasource_info_list": [], "start_node_id": "start", "is_published": true, "response_mode": "blocking"}'

2. 如果返回 404，说明你的版本可能没有包含这个修复，建议升级到包含 PR fix(api): register knowledge pipeline service API routes #32097 的更新版本

3. 如果返回 401 或其他权限错误，那可能是 API key 配置问题

你用 dataset- 开头的 key 调用 create-by-file 上传文件是成功的吗？能否试试调用 pipeline/run 接口看看具体返回什么错误？

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[simonjhy]

我现在已经升级到1.13.0版本，但是我依然没有找到可以使用访问下面api的接口的 api key。这个在哪里找到？我在知识库中找到可以创建的api key都是data开头，但是这个api key测试没有执行权限啊。这个key在哪里生成，能直接给我个创建key的链接吗？
curl --location --request POST 'https://api.dify.ai/v1/datasets/{dataset_id}/pipeline/run'
--header 'Authorization: Bearer {api_key}'
--header 'Content-Type: application/json'
--data '{
"inputs": {},
"datasource_type": "custom",
"datasource_info_list": [],
"start_node_id": "start",
"is_published": true,
"response_mode": "blocking"
}'

[dosubot[bot]]

你之前在知识库页面左下角找到的 dataset-xxx 开头的 API key 就是正确的 key！这个 key 类型正是 pipeline/run 接口所需要的。

获取位置确认：在知识库界面的左下角有一个小图标，点击后可以创建 dataset API key。

请直接用这个 key 试试调用 pipeline/run：

curl --location --request POST 'http://your-dify-host/v1/datasets/{dataset_id}/pipeline/run' \
--header 'Authorization: Bearer dataset-xxxxxxxx' \
--header 'Content-Type: application/json' \
--data '{
  "inputs": {},
  "datasource_type": "custom",
  "datasource_info_list": [],
  "start_node_id": "start",
  "is_published": true,
  "response_mode": "blocking"
}'

如果返回错误，请告诉我具体的错误信息（404、401、400 等），这样可以帮助定位问题：

• 404：可能是路由未注册的问题
• 401：API key 格式或权限问题
• 400：请求参数问题

另外，需要确保你的知识库已经启用了 API 访问（dataset.enable_api 标志）。这个 pipeline API 目前还是实验性的，没有官方文档。

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[simonjhy]

现在升级到1.13.0版本以后，这个知识库的api key链接没有了，只有在知识库访问的入口页面有一个服务api的连接可以申请api key，这个api key就是具备读写权限的吗？

我在dify 1.13.0中，只在这个地方可以找到知识库访问的api key

[dosubot[bot]]

在 v1.13.0 中，API key 管理从知识库列表页移到了知识库详情页。

获取 API key 的正确步骤：

1. 点击进入具体的某个知识库（不是列表页）
2. 在详情页的右侧找到 "Service API"（服务API）卡片
3. 点击 "Secret Key"（密钥）按钮来查看或创建 API key

这个 Service API 卡片只有具备 admin 或 editor 权限的用户才能看到。

关于权限：是的，通过这里获取的 dataset API key 具备读写权限，可以用于 pipeline/run 等写操作。

你在列表页看到的 "服务API" 链接可能是一个快捷入口，但实际的 API key 管理需要进入具体知识库的详情页才能看到完整的 Secret Key 管理界面。

请试试点击你的某个 STM32 知识库进入详情页，看看右侧是否有 Service API 卡片。

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[simonjhy]

我在一个特定的知识库里，我没有看到右侧有详情页面啊？难道我的1.13.0有问题？

[dosubot[bot]]

根据你的截图，我看到你在知识库详情页的左侧有 "文档"、"工作流"、"召回测试"、"设置" 这些标签。

Service API 卡片应该在右侧边栏，但根据代码，这个卡片只对具有 admin 或 editor 权限的用户显示。

可能的原因：

1. 权限问题：虽然你的工作区显示是 "administrator's Workspace"，但请确认你登录的账号确实具有该知识库的 admin 或 editor 权限
2. UI 渲染问题：可以尝试刷新页面或清除浏览器缓存

建议尝试：

1. 点击左侧的 "设置"（Settings） 标签，看看里面是否有权限相关的配置
2. 检查页面右侧是否有任何可折叠的面板或侧边栏
3. 尝试调整浏览器窗口大小，有时候响应式布局可能会隐藏右侧面板

如果确认是 administrator 账号但仍看不到，可能是 v1.13.0 的某个 UI 问题。你也可以尝试直接用 API 创建 key：

# 先用 console 登录获取 access token，然后调用：
POST /console/api/datasets/api-keys

能否告诉我你的账号角色是什么？以及 "设置" 页面里都有哪些选项？

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[simonjhy]

我的设置里只有这些选项，我这个空间只有我一个用户，就是dify的空实例我添加的第一个也是唯一的用户，是因为这个用户不是管理员权限？

[dosubot[bot]]

你的账号是 Owner 角色，这个角色是有权限看到 Service API 卡片的！Owner 和 Admin 具有相同的权限。

看不到 Service API 卡片最可能的原因是数据库迁移没有完成。这是升级到新版本后常见的问题。

请尝试以下步骤：

1. 执行数据库迁移：

   docker compose exec api flask upgrade-db

   或者设置环境变量 MIGRATION_ENABLED=true 然后重启容器

2. 重启 web 服务：

   docker compose restart web
   # 或重启整个服务
   docker compose down && docker compose up -d

3. 清除浏览器缓存后刷新页面

如果上述步骤后还是看不到，可以尝试新建一个知识库，看看新知识库是否能看到 Service API 卡片。有时候旧版本创建的知识库可能缺少某些必需字段。

执行完这些步骤后告诉我结果！

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[simonjhy]

执行数据库迁移：

1)docker compose exec api flask upgrade-db
这个是在运行状态可以直接执行的吗?

2)或者设置环境变量 MIGRATION_ENABLED=true 然后重启容器
这个是在哪个容器的环境变量下设置？

[dosubot[bot]]

好问题！

1) docker compose exec api flask upgrade-db 可以在运行状态直接执行

是的，可以直接执行。这个命令会进入正在运行的 api 容器内部执行数据库迁移。Dify 使用 Redis 分布式锁来防止并发执行，所以是安全的。

# 在 docker 目录下执行
cd docker
docker compose exec api flask upgrade-db

2) MIGRATION_ENABLED=true 是在 api 容器的环境变量中设置

有两种方式设置：

方式一：在 docker/.env 文件 中添加：

MIGRATION_ENABLED=true

方式二：在 docker-compose.yaml 的 api 服务 environment 部分设置。

设置后重启容器：

docker compose down && docker compose up -d

建议：直接用方式一执行 docker compose exec api flask upgrade-db 最简单，不需要修改配置文件。

执行后看看是否有输出迁移成功的信息，然后刷新网页检查 Service API 卡片是否出现。

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[simonjhy]

我执行了docker compose exec api flask upgrade-db之后输出如下的日志：

2026-03-19 10:59:53.467 INFO [MainThread] [utils.py:151] - Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
2026-03-19 10:59:53.467 INFO [MainThread] [utils.py:164] - NumExpr defaulting to 16 threads.
Preparing database migration...
Starting database migration.
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
Database migration successful!

这个MIGRATION_ENABLED=true我升级到1.13的时候，就已经设置了。但是似乎没起作用

[simonjhy]

我看了docker compose中定义web的部分使用的也是对应1.13.0的版本，但是我升级结束后，我还是没看到你说的这个知识库的详细设置页面

我全新创建了一个基于流水线的空数据库也没有看到你说的知识库右侧的详情页面