Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/main'
Browse files Browse the repository at this point in the history
  • Loading branch information
@Evil0ctal
Evil0ctal committed May 6, 2024
2 parents 286418c + 4190bd7 commit a1f26e8
Show file tree
Hide file tree
Showing 2 changed files with 46 additions and 84 deletions.
64 changes: 22 additions & 42 deletions README.en.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,61 +13,30 @@

</div>

## 🔊 V4.0.0 version refactoring

> ALL:
- Removed outdated bilibili code and needs someone to rewrite it.
- Someone in the group wants to add the analysis of Kuaishou and Xigua videos.
- The readme is outdated and needs to be rewritten.
- Make PyPi package
- The config.yaml file needs to be trimmed.
- Add parsing of user homepage.
- iOS shortcuts need to be updated to be compatible with the latest API responses and paths.
- Desktop downloaders or browser plug-ins can be developed if necessary.
- Solve the problem of crawler cookie risk control.

> Change
- Run Pywebio as a sub-APP of FastAPI.
- Rewritten the interfaces of Douyin and TikTok, thank you[@johnserf-seed](https://github.com/Johnserf-Seed)
- The file download endpoint has been rewritten and now uses asynchronous file IO.
- 对所有端点进行了注解和演示值的添加。
- Organize the project file structure.

> Remark
If you are interested in writing this project together, please add us on WeChat`Evil0ctal`Note: Github project reconstruction, everyone can communicate and learn from each other in the group. Advertising and illegal things are not allowed. It is purely for making friends and technical exchanges.

> Private interface service
Discord:[Tikhub discord](https://discord.com/invite/aMEAS8Xsvz)

Free Douyin/TikTok API:[Tikhub Beta Opi](https://beta.tikhub.io/)

## 👻Introduction

> 🚨If you need to use a private server to run this project, please refer to the deployment method\[[Docker deployment](./README.md#%E9%83%A8%E7%BD%B2%E6%96%B9%E5%BC%8F%E4%BA%8C-docker),[One-click deployment](./README.md#%E9%83%A8%E7%BD%B2%E6%96%B9%E5%BC%8F%E4%B8%80-linux)]
> 🚨If you need to use a private server to run this project, please refer to:[Deployment preparations](./README.md#%EF%B8%8F%E9%83%A8%E7%BD%B2%E5%89%8D%E7%9A%84%E5%87%86%E5%A4%87%E5%B7%A5%E4%BD%9C%E8%AF%B7%E4%BB%94%E7%BB%86%E9%98%85%E8%AF%BB),[Docker deployment](./README.md#%E9%83%A8%E7%BD%B2%E6%96%B9%E5%BC%8F%E4%BA%8C-docker),[One-click deployment](./README.md#%E9%83%A8%E7%BD%B2%E6%96%B9%E5%BC%8F%E4%B8%80-linux)
This project is based on[PyWebIO](https://github.com/pywebio/PyWebIO)，[FastAPI](https://fastapi.tiangolo.com/)，[HTTPX](https://www.python-httpx.org/), fast and asynchronous[Tik Tok](https://www.douyin.com/)/[TikTok](https://www.tiktok.com/)Data crawling tool, and realizes online batch analysis and downloading of videos or photo albums without watermarks through the Web, data crawling API, iOS shortcut command without watermark downloads and other functions. You can deploy or modify this project yourself to achieve more functions, or you can call it directly in your project[scraper.py](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/Stable/scraper.py)or install an existing[pip package](https://pypi.org/project/douyin-tiktok-scraper/)As a parsing library, it is easy to crawl data, etc.....

_Some simple application scenarios:_

_Download prohibited videos, perform data analysis, download without watermark on iOS (with[Shortcut command APP that comes with iOS](https://apps.apple.com/cn/app/%E5%BF%AB%E6%8D%B7%E6%8C%87%E4%BB%A4/id915249334)Cooperate with the API of this project to achieve in-app downloads or read clipboard downloads), etc....._

#### ⚠️Note:
## 🔊 V4 version notes

- You need to solve crawler cookie risk control issues by yourself, otherwise the interface may become unusable.
- Douyin web cookie (obtain and replace the cookie in the configuration file below):
- <https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/30e56e5a7f97f87d60b1045befb1f6db147f8590/crawlers/douyin/web/config.yaml#L7>
- TikTok web-side cookies (obtain and replace the cookies in the configuration file below):
- <https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/30e56e5a7f97f87d60b1045befb1f6db147f8590/crawlers/tiktok/web/config.yaml#L6>
- I turned off the online download function of the demo site. The video someone downloaded was so huge that it crashed my server. I just right-clicked on the web page parsing result and saved it...
- The cookies of the demo site are my own and are not guaranteed to be valid for a long time. They only serve as a demonstration. If you deploy it yourself, please obtain the cookies yourself.
- If you are interested in writing this project together, please add us on WeChat`Evil0ctal`Note: Github project reconstruction, everyone can communicate and learn from each other in the group. Advertising and illegal things are not allowed. It is purely for making friends and technical exchanges.
- used in this project`X-Bogus`The algorithm can still call Douyin and TikTok APIs normally.`A-Bogus`The algorithm will not be open source for the time being.
- Due to Douyin's risk control, after deploying this project, please**Obtain the cookie of Douyin website in the browser and replace it in config.yaml.**
- Please read the document below before raising an issue. Solutions to most problems will be included in the document.
- This project is completely free, but when using it, please comply with:[Apache-2.0 license](https://github.com/Evil0ctal/Douyin_TikTok_Download_API?tab=Apache-2.0-1-ov-file#readme)
- This project has a closed source branch version, which contains more interfaces and services. Please see the information below for details.
- Discord:[Tikhub discord](https://discord.com/invite/aMEAS8Xsvz)
- Free Douyin/TikTok API:[Tikhub Beta Opi](https://beta.tikhub.io/)

## 🖥Demo site: I am very vulnerable...please do not stress test (·•᷄ࡇ•᷅ )

> 😾The online download function of the demo site is closed.
> 😾The online download function of the demo site has been turned off, and due to cookie reasons, the availability of Douyin's parsing and API services cannot be guaranteed on the Demo site.
🍔Web APP:<https://douyin.wtf/>

Expand Down Expand Up @@ -265,6 +234,17 @@ Online:<https://api.douyin.wtf/docs>

**_For more demonstrations, please see the documentation..._**

## ⚠️Preparation work before deployment (please read carefully):

- You need to solve crawler cookie risk control issues by yourself, otherwise the interface may become unusable.
- Douyin web cookie (obtain and replace the cookie in the configuration file below):
- <https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/30e56e5a7f97f87d60b1045befb1f6db147f8590/crawlers/douyin/web/config.yaml#L7>
- TikTok web-side cookies (obtain and replace the cookies in the configuration file below):
- <https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/30e56e5a7f97f87d60b1045befb1f6db147f8590/crawlers/tiktok/web/config.yaml#L6>
- I turned off the online download function of the demo site. The video someone downloaded was so huge that it crashed the server. You can right-click on the web parsing results page to save the video...
- The cookies of the demo site are my own and are not guaranteed to be valid for a long time. They only serve as a demonstration. If you deploy it yourself, please obtain the cookies yourself.
- here is one**Video tutorial**You can refer to:**_<https://www.bilibili.com/video/BV1vE421j7NR/>_**

## 💻Deployment (Method 1 Linux)

> 💡Tips: It is best to deploy this project to a server in the United States, otherwise strange BUGs may occur.
Expand Down
66 changes: 24 additions & 42 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,41 +31,9 @@

</div>

## 🔊 V4.0.0版本重构

> TODO:
- 移除了过时的bilibili代码，需要有人重写。
- 群里有人想添加快手以及西瓜视频的解析。
- 自述文件已经过时，需要进行重写。
- 进行PyPi包制作
- config.yaml文件需要进行修整。
- 添加对用户主页的解析。
- iOS快捷指令需要更新兼容最新的API响应和路径。
- 桌面端下载器或浏览器插件有需要可以进行开发。
- 解决爬虫Cookie风控问题。

> 更改
- 将Pywebio作为FastAPI的子APP一起运行。
- 重写了抖音以及TikTok的接口，感谢 [@johnserf-seed](https://github.com/Johnserf-Seed)
- 重写了文件下载的端点，现在使用异步文件IO。
- 对所有端点进行了注解和演示值的添加。
- 整理项目文件结构。

> 备注
感兴趣一起写这个项目的给请加微信`Evil0ctal`备注github项目重构，大家可以在群里互相交流学习，不允许发广告以及违法的东西，纯粹交朋友和技术交流。

> 私有接口服务
Discord: [TikHub Discord](https://discord.com/invite/aMEAS8Xsvz)

Free Douyin/TikTok API: [TikHub Beta API](https://beta.tikhub.io/)

## 👻介绍

> 🚨如需使用私有服务器运行本项目，请参考部署方式[[Docker部署](./README.md#%E9%83%A8%E7%BD%B2%E6%96%B9%E5%BC%8F%E4%BA%8C-docker), [一键部署](./README.md#%E9%83%A8%E7%BD%B2%E6%96%B9%E5%BC%8F%E4%B8%80-linux)]
> 🚨如需使用私有服务器运行本项目，请参考：[部署准备工作](./README.md#%EF%B8%8F%E9%83%A8%E7%BD%B2%E5%89%8D%E7%9A%84%E5%87%86%E5%A4%87%E5%B7%A5%E4%BD%9C%E8%AF%B7%E4%BB%94%E7%BB%86%E9%98%85%E8%AF%BB), [Docker部署](./README.md#%E9%83%A8%E7%BD%B2%E6%96%B9%E5%BC%8F%E4%BA%8C-docker), [一键部署](./README.md#%E9%83%A8%E7%BD%B2%E6%96%B9%E5%BC%8F%E4%B8%80-linux)
本项目是基于 [PyWebIO](https://github.com/pywebio/PyWebIO)，[FastAPI](https://fastapi.tiangolo.com/)，[HTTPX](https://www.python-httpx.org/)，快速异步的[抖音](https://www.douyin.com/)/[TikTok](https://www.tiktok.com/)数据爬取工具，并通过Web端实现在线批量解析以及下载无水印视频或图集，数据爬取API，iOS快捷指令无水印下载等功能。你可以自己部署或改造本项目实现更多功能，也可以在你的项目中直接调用[scraper.py](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/Stable/scraper.py)或安装现有的[pip包](https://pypi.org/project/douyin-tiktok-scraper/)作为解析库轻松爬取数据等.....

Expand All @@ -74,18 +42,20 @@ Free Douyin/TikTok API: [TikHub Beta API](https://beta.tikhub.io/)
*下载禁止下载的视频，进行数据分析，iOS无水印下载（搭配[iOS自带的快捷指令APP](https://apps.apple.com/cn/app/%E5%BF%AB%E6%8D%B7%E6%8C%87%E4%BB%A4/id915249334)
配合本项目API实现应用内下载或读取剪贴板下载）等.....*

#### ⚠️注意：
## 🔊 V4 版本备注

- 感兴趣一起写这个项目的给请加微信`Evil0ctal`备注github项目重构，大家可以在群里互相交流学习，不允许发广告以及违法的东西，纯粹交朋友和技术交流。
- 本项目使用的`X-Bogus`算法依旧可以正常调用Douyin以及TikTok的API，`A-Bogus`算法暂时不会开源。
- 由于Douyin的风控，部署完本项目后请在**浏览器中获取Douyin网站的Cookie然后在config.yaml中进行替换。**
- 请在提出issue之前先阅读下方的文档，大多数问题的解决方法都会包含在文档中。
- 本项目是完全免费的，但使用时请遵守：[Apache-2.0 license](https://github.com/Evil0ctal/Douyin_TikTok_Download_API?tab=Apache-2.0-1-ov-file#readme)
- 本项目有一个闭源的分支版本，包含更多的接口和服务，详情请查看下方的信息。
- Discord: [TikHub Discord](https://discord.com/invite/aMEAS8Xsvz)
- Free Douyin/TikTok API: [TikHub Beta API](https://beta.tikhub.io/)

- 你需要自行解决爬虫Cookie风控问题，否则可能会导致接口无法使用。
- 抖音网页端Cookie（自行获取并替换下面配置文件中的Cookie）：
- https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/30e56e5a7f97f87d60b1045befb1f6db147f8590/crawlers/douyin/web/config.yaml#L7
- TikTok网页端Cookie（自行获取并替换下面配置文件中的Cookie）：
- https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/30e56e5a7f97f87d60b1045befb1f6db147f8590/crawlers/tiktok/web/config.yaml#L6
- 演示站点的在线下载功能被我关掉了，有人下的视频巨大无比直接给我服务器干崩了，自己在网页解析结果里右键保存吧...
- 演示站点的Cookie是我自己的，不保证长期有效，只起到演示作用，自己部署的话请自行获取Cookie。
## 🖥演示站点: 我很脆弱...请勿压测(·•᷄ࡇ•᷅ ）

> 😾演示站点的在线下载功能已关闭。
> 😾演示站点的在线下载功能已关闭，并且由于Cookie原因，Douyin的解析以及API服务在Demo站点无法保证可用性。
🍔Web APP: [https://douyin.wtf/](https://douyin.wtf/)

Expand Down Expand Up @@ -287,6 +257,18 @@ https://www.tiktok.com/@evil0ctal/video/7156033831819037994

***更多演示请查看文档内容......***


## ⚠️部署前的准备工作(请仔细阅读)：

- 你需要自行解决爬虫Cookie风控问题，否则可能会导致接口无法使用。
- 抖音网页端Cookie（自行获取并替换下面配置文件中的Cookie）：
- https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/30e56e5a7f97f87d60b1045befb1f6db147f8590/crawlers/douyin/web/config.yaml#L7
- TikTok网页端Cookie（自行获取并替换下面配置文件中的Cookie）：
- https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/30e56e5a7f97f87d60b1045befb1f6db147f8590/crawlers/tiktok/web/config.yaml#L6
- 演示站点的在线下载功能被我关掉了，有人下的视频巨大无比直接给我服务器干崩了，你可以在网页解析结果页面右键保存视频...
- 演示站点的Cookie是我自己的，不保证长期有效，只起到演示作用，自己部署的话请自行获取Cookie。
- 这里有一个**视频教程**可以参考：***[https://www.bilibili.com/video/BV1vE421j7NR/](https://www.bilibili.com/video/BV1vE421j7NR/)***

## 💻部署(方式一 Linux)

> 💡提示：最好将本项目部署至美国地区的服务器，否则可能会出现奇怪的BUG。
Expand Down

0 comments on commit a1f26e8

Please sign in to comment.