Several Changes See Changelog For More

.gitignore

@@ -108,3 +108,5 @@ venv.bak/
 .mypy_cache/
 
 ##my files
+temptsfiles
+merged.ts

CHANGELOG.md

@@ -1,18 +1,38 @@
 # CHANGELOG.md
 
-## 1.0.55 (Unreleased)
+## 1.0.76 (24/09/2021)
+
+Features:
+
+- Now supports resumable downloading
+- Retry support on encountering error
+- Use custom headers configration json file
+- Use custom http and https proxies
+- Pre ffmpeg path check
+
+Changes:
+
+- No single threaded downloads, merged with thread pool executor with default as 5 workers
+- Now using a requests session for making a get request
+- Now KeyboardInterrupt is not handled, you have to kill the running script manually
+- `--blob` option renamed to `--baseurl`
+- `--user-agent` option removed
 
 Bug Fixes:
 
 - Unnecessary clean up task for single threaded downloads
 
+Added:
+
+- Makefile to deploy a windows executable
+
 ## 1.0.52 (29/07/2021)
 
 Features:
 
 - Typer CLI instead of argsparse 
 - Improved downloading with multiple threads
-- Ffmpeg ts conversion integration
+- ffmpeg ts conversion integration
 - Download speed meter
 - Data console output for **vsdownload capture** command
 

CLI-API.md

@@ -1,6 +1,6 @@
 # `vsdownload`
 
-command line extension to download hls video streams from websites, m3u8 files and urls
+command line program to download hls video streams from websites, m3u8 files and urls
 
 **Usage**:
 
@@ -10,15 +10,15 @@ $ vsdownload [OPTIONS] COMMAND [ARGS]...
 
 **Options**:
 
-* `-v, --version`: show installed version of vsdownload
+* `-v, --version`: show current version of vsdownload
 * `--install-completion`: Install completion for the current shell.
 * `--show-completion`: Show completion for the current shell, to copy it or customize the installation.
 * `--help`: Show this message and exit.
 
 **Commands**:
 
 * `capture`: capture m3u8 urls from a website
-* `save`: download and save ts file from m3u8 url or...
+* `save`: download m3u8 stream from m3u8 url or file
 
 ## `vsdownload capture`
 
@@ -36,14 +36,14 @@ $ vsdownload capture [OPTIONS] URL
 
 **Options**:
 
-* `--output TEXT`: output website m3u8 capture logs in which path  [default: log.json]
-* `--driver TEXT`: path of chrome driver for selenium
-* `--scan-ext TEXT`: scan network logs until --scan-ext extension is found in any one of url  [default: ts]
+* `--output mysite_log.json`: output website m3u8 capture logs in which path  [default: log.json]
+* `--driver chromedriver.exe`: path of chrome driver for selenium
+* `--scan-ext m3u8/ts/mp4`: scan network logs until --scan-ext extension is found in any one of the request url  [default: ts]
 * `--help`: Show this message and exit.
 
 ## `vsdownload save`
 
-download and save ts file from m3u8 url or file
+download m3u8 stream from m3u8 url or file
 
 **Usage**:
 
@@ -53,17 +53,20 @@ $ vsdownload save [OPTIONS] INPUT
 
 **Arguments**:
 
-* `INPUT`: url | .m3u8 | log.json  [required]
+* `INPUT`: url|.m3u8|log.json  [required]
 
 **Options**:
 
-* `-o, --output TEXT`: path for output for downloaded video stream file  [default: merged.ts]
-* `-t, --threads INTEGER`: download multiple ts files in parallel threads  [default: 1]
-* `-b, --blob TEXT`: endpoint base url for all segments
-* `--user-agent TEXT`: by default make a request to url with default user agent header
-* `--chunk-size INTEGER`: chunk size for downloading ts files in kilobytes  [default: 1024]
-* `--ffmpeg-path TEXT`: path of ffmpeg binary  [default: ffmpeg]
-* `--tempdir TEXT`: directory for saving temporary ts files when downloading in mutiple threads  [default: temptsfiles]
-* `--timeout INTEGER`: waiting time for post tasks to perform after downloading stream (in seconds)  [default: 5]
+* `-o, --output merged.ts/merged.mp4/merged.mkv`: path for output of downloaded video stream file  [default: merged.ts]
+* `-b, --baseurl https://xyz.com/playlist7/`: base url or endpoint for all segments
+* `-t, --threads 1-32`: max thread count for parallel threads to download segments  [default: 5]
+* `--cleanup / --no-cleanup`: delete temporary downloaded segments, add --no-cleanup flag to use resume capabilities  [default: True]
+* `--chunk-size INTEGER`: chunk size for downloading ts files (in kilobytes)  [default: 1024]
+* `--headers headers.json`: path of header defining json file which will update headers
+* `--proxy-address http://127.0.0.1:8000`: http or https proxy address to use
+* `--ffmpeg-path c:\ffmpeg\bin\ffmpeg.exe`: path of ffmpeg binary  [default: ffmpeg]
+* `--tempdir directory`: path of directory for saving temporary files while downloading  [default: temptsfiles]
+* `--retry-count INTEGER`: retry count for downloading segment  [default: 15]
+* `--timeout time`: waiting time for post tasks to perform after downloading (in seconds)  [default: 5]
 * `--pre-select INTEGER`: pre select a url from log.json file
 * `--help`: Show this message and exit.

Makefile

@@ -0,0 +1,25 @@
+VenvActivate := "./env/Scripts/activate.bat"
+
+.PHONY: requirements docs venv package
+
+requirements:
+	@pip install --upgrade pip
+	@pip install virtualenv
+	@pip install -r requirements.txt
+	@pip install typer-cli
+
+docs:
+	@typer vsdownload/vsdownload.py utils docs --output CLI-API.md
+
+venv:
+	@python -m venv env
+	@$(VenvActivate) && python -m pip install --upgrade pip
+	@$(VenvActivate) && python -m pip install pyinstaller
+	@$(VenvActivate) && python -m pip install -r requirements.txt
+
+package: venv
+	@$(VenvActivate) && pyinstaller main.py --name vsdownload --onefile
+	@powershell -C "Remove-Item env -Recurse"
+	@powershell -C "Remove-Item __pycache__ -Recurse"
+	@powershell -C "Remove-Item build -Recurse"
+	@powershell -C "Remove-Item vsdownload.spec"

README.md

@@ -1,6 +1,6 @@
 # VS Download - Video Stream Download
 
-command line extension to download hls video streams from websites, m3u8 files and urls.
+command line program to download hls video streams from websites, m3u8 files and urls.
 
 <p align="center">
   a compact lightweight m3u8 downloader
@@ -24,6 +24,20 @@ command line extension to download hls video streams from websites, m3u8 files a
   <img src="https://raw.githubusercontent.com/360modder/vsdownload/master/images/vsdownload.gif">
 </p>
 
+## Features Implemented
+
+- [x] resume support
+- [x] capturing m3u8 links and urls from a website
+- [x] realtime file size prediction and download speed
+- [x] downloding in multiple threads
+- [x] master m3u8 playlist parsing
+- [x] custom headers and proxies
+- [x] retry on error
+- [x] ffmpeg conversion integration
+- [x] platform independent
+- [ ] encrypted playlists
+- [ ] separate video and audio stream 
+
 ## Important Declaration
 
 If you are distributing downloaded video streams, first ensure that you have rights for those video streams or files.
@@ -46,6 +60,8 @@ Or install from github repository.
 pip install https://github.com/360modder/vsdownload/archive/master.zip
 ```
 
+Or you can also find a windows [executable](https://github.com/360modder/vsdownload/releases/download/v1.0.76/vsdownload_v1.0.76.exe) from [releases](https://github.com/360modder/vsdownload/releases).
+
 ## Usage
 
 - Capturing m3u8 files from website and downloading hls streams
@@ -61,6 +77,8 @@ vsdownload save log.json
 vsdownload save <m3u8 url or file> -o video.ts
 ```
 
+> Add **--no-cleanup** flag to use resume capabilities
+
 ## How to convert .ts to .mp4, .mkv etc. ?
 
 First download and install [ffmpeg](https://www.ffmpeg.org/download.html) and then in output flag specify your desired format.
@@ -78,7 +96,7 @@ vsdownload save <m3u8 url or file> --ffmpeg-path <path to ffmpeg binary> -o vide
 
 ## How to speed up downloading speed ?
 
-Downloading ts files with multiple threads would be a good option and also using a higher chunk size will reduce the file i/o operations. vsdownload can handle those things by using some flags.
+Downloading ts files with more multiple worker threads would be a good option and also using a higher chunk size will reduce the file i/o operations. vsdownload can handle those things by using some flags.
 
 Downloading m3u8 files with 16 threads with 4k chunk size.
 
@@ -90,11 +108,11 @@ vsdownload save <m3u8 url or file> --chunk-size 4096 -t 16 -o video.ts
 
 This problem arises when vsdownload makes get request to ts file and it returns a bad response body which results in corrupted downloads.
 
-This error maybe caused by incorrect segment uri which is auto parsed by vsdownload. You can manually override it by following the given steps. 
+This error maybe caused by incorrect segment url which is auto parsed by vsdownload. You can manually override it by following the given steps. 
 
 Steps resolve this error:
 
-1. Check for m3u8 uri and other ts files uri/s from your browser's network logs or use **vsdownload capture** command. you will notice a comman uri attached to very ts file uri, then note it down as **blob**.
+1. Check for m3u8 uri and other ts files uri/s from your browser's network logs or use **vsdownload capture** command. you will notice a comman url attached to very ts file uri, then note it down as **baseurl**.
 
 ```
 Example:
@@ -104,13 +122,13 @@ Example:
 5. https://xyz.in/283678-293/stream-hls/stream_0_high_02.ts
 4. https://xyz.in/283678-293/stream-hls/stream_0_high_03.ts
 
-Blob (comman uri / base endpoint): https://xyz.in/283678-293/stream-hls/
+Baseurl (comman url / base endpoint): https://xyz.in/283678-293/stream-hls/
 ```
 
-2. After getting a blob uri try to download stream by using this command.
+2. After getting a baseurl try to download stream by using this command.
 
 ```bash
-vsdownload save <m3u8 url or file> -b <blob (comman uri)> -o video.ts
+vsdownload save <m3u8 url or file> -b <baseurl> -o video.ts
 ```
 
 ```bash
@@ -125,6 +143,7 @@ Some extra things to try for websites which don't make their hls streams publicl
 
 1. Maintain a connection to server by playing stream in browser or from any other means.
 2. Check for maximum supported parallel connections from server. If it can use more than 1 connection, then streams maybe downloaded by following the above steps.
+3. Use **--proxy-address** flag.
 
 ## How to merge separate hls video and audio streams ?
 
@@ -139,10 +158,6 @@ ffmpeg -i merged_video_stream.ts -i merged_audio_stream.ts -c copy merged_video_
 - [CLI-API.md](CLI-API.md)
 - [CHANGELOG.md](CHANGELOG.md)
 
-## Todos
-
-- pause/resume support
-
 ## License
 
 © 2021 360modder

main.py

@@ -1,3 +1,10 @@
 import vsdownload.vsdownload
 
 vsdownload.vsdownload.console_script()
+
+# some test commands (root)
+# python main.py --help
+# python main.py save https://multiplatform-f.akamaihd.net/i/multi/will/bunny/big_buck_bunny_,640x360_400,640x360_700,640x360_1000,950x540_1500,.f4v.csmil/index_0_av.m3u8
+
+# resumable downloading
+# python main.py save https://multiplatform-f.akamaihd.net/i/multi/will/bunny/big_buck_bunny_,640x360_400,640x360_700,640x360_1000,950x540_1500,.f4v.csmil/index_0_av.m3u8 --no-cleanup

setup.py

@@ -29,7 +29,7 @@ def get_version() -> str:
 setup(
     name="vsdownload",
     version=get_version(),
-    description="command line extension to download hls video streams from websites, m3u8 files and urls",
+    description="command line program to download hls video streams from websites, m3u8 files and urls",
     long_description=README,
     long_description_content_type="text/markdown",
     keywords=["m3u8", "ts", "video", "stream", "downloader", "m3u8downloader"],
@@ -38,13 +38,6 @@ def get_version() -> str:
     author_email="apoorv9450@gmail.com",
     license="MIT",
     python_requires=">=3.6",
-    classifiers=[
-        "License :: OSI Approved :: MIT License",
-        "Programming Language :: Python :: 3.6",
-        "Programming Language :: Python :: 3.7",
-        "Programming Language :: Python :: 3.8",
-        "Programming Language :: Python :: 3.9",
-    ],
     packages=["vsdownload", "vsdownload/commands"],
     include_package_data=True,
     install_requires=REQUIREMENTS,

test/headers.json

@@ -0,0 +1,3 @@
+{
+    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
+}

vsdownload/commands/capture.py

@@ -24,7 +24,7 @@ def command_capture(args):
 
     m3u8_links = utils.find_urls_by_ext(f"{logs}", "m3u8")
     json_data = {"m3u8_urls": m3u8_links}
-    json_data["blob"] = utils.find_blob_by_urls(m3u8_links, "m3u8")
+    json_data["baseurl"] = utils.find_baseurl_by_urls(m3u8_links, "m3u8")
 
     pprint(json_data)