Split files above 2 Gb & merge splitted files

[EDM115]

• Temporary workaround : files above 2 Gb are uploaded to BayFiles instead of Telegram
• If a file is above 2 Gb, it's splitted first then uploaded
  Will based on this (c) TTK
• User send /merge, then send all the splitted files, then /merge again, bot downloads, merge and extract
  References : 1, 2, 3

[pr13260]

How about uploading files as zip or unziped to public clouds like anonfiles or user provided
You can use APIs or Rclone it

[EDM115]

@pr13260 something like an unzip to cloud (all unzipped files are separately uploaded to anonfiles or so) ?
Or something like merge the files then upload merged one to cloud ?

[pr13260]

@pr13260 something like an unzip to cloud (all unzipped files are separately uploaded to anonfiles or so) ?
Or something like merge the files then upload merged one to cloud ?

any would work
not all
files with **size more than 2gb should not be splitted **
and be uploaded to any cloud services
this will make bot 🔥

[EDM115]

Alright, I can work on this now

[EDM115]

References :

• BayFiles API
• JSON in Python
• Bot will execute curl command easily in terminal, but will work with requests later

[pr13260]

Alright, I can work on this now

Awesome 👍👍

[EDM115]

2023-07-01T18:02:33.207948+00:00 app[worker.1]: 2023-07-01 18:02:33,169 - INFO - unzipper - asyncio_1 - 
2023-07-01T18:02:33.207964+00:00 app[worker.1]: 7-Zip [64] 17.04 : Copyright (c) 1999-2021 Igor Pavlov : 2017-08-28
2023-07-01T18:02:33.207965+00:00 app[worker.1]: p7zip Version 17.04 (locale=C.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs x64)
2023-07-01T18:02:33.207966+00:00 app[worker.1]: 
2023-07-01T18:02:33.207966+00:00 app[worker.1]: Scanning the drive for archives:
2023-07-01T18:02:33.207966+00:00 app[worker.1]: 1 file, 2431906561 bytes (2320 MiB)
2023-07-01T18:02:33.207967+00:00 app[worker.1]: 
2023-07-01T18:02:33.207967+00:00 app[worker.1]: Extracting archive: /app/Downloaded/854158484/archive_from_854158484.zip
2023-07-01T18:02:33.207967+00:00 app[worker.1]: --
2023-07-01T18:02:33.207967+00:00 app[worker.1]: Path = /app/Downloaded/854158484/archive_from_854158484.zip
2023-07-01T18:02:33.207968+00:00 app[worker.1]: Type = zip
2023-07-01T18:02:33.207968+00:00 app[worker.1]: Physical Size = 2431906561
2023-07-01T18:02:33.207968+00:00 app[worker.1]: 
2023-07-01T18:02:33.207969+00:00 app[worker.1]: Everything is Ok
2023-07-01T18:02:33.207969+00:00 app[worker.1]: 
2023-07-01T18:02:33.207969+00:00 app[worker.1]: Folders: 1
2023-07-01T18:02:33.207969+00:00 app[worker.1]: Files: 2
2023-07-01T18:02:33.207969+00:00 app[worker.1]: Size:       2431905867
2023-07-01T18:02:33.207970+00:00 app[worker.1]: Compressed: 2431906561
2023-07-01T18:02:44.082203+00:00 app[worker.1]: 2023-07-01 18:02:44,082 - INFO - unzipper - MainThread - ext_a paths : ['/app/Downloaded/854158484/extracted/Interstellar (2014) (2014) [1080p]/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4', '/app/Downloaded/854158484/extracted/Interstellar (2014) (2014) [1080p]/WWW.YTS.RE.jpg']
2023-07-01T18:02:45.141825+00:00 app[worker.1]: 2023-07-01 18:02:45,141 - WARNING - pyrogram.session.session - MainThread - [UnzipperBot] Waiting for 3 seconds before continuing (required by "messages.SendMedia")
2023-07-01T18:03:16.618827+00:00 app[worker.1]: 2023-07-01 18:03:16,594 - WARNING - pyrogram.session.session - MainThread - [UnzipperBot] Waiting for 3 seconds before continuing (required by "messages.SendMessage")
2023-07-01T18:03:40.174314+00:00 app[worker.1]: 2023-07-01 18:03:40,173 - INFO - unzipper - asyncio_0 - 
2023-07-01T18:03:40.174336+00:00 app[worker.1]: 7-Zip [64] 17.04 : Copyright (c) 1999-2021 Igor Pavlov : 2017-08-28
2023-07-01T18:03:40.174337+00:00 app[worker.1]: p7zip Version 17.04 (locale=C.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs x64)
2023-07-01T18:03:40.174338+00:00 app[worker.1]: 
2023-07-01T18:03:40.174339+00:00 app[worker.1]: Scanning the drive:
2023-07-01T18:03:40.174340+00:00 app[worker.1]: 1 file, 2431697820 bytes (2320 MiB)
2023-07-01T18:03:40.174341+00:00 app[worker.1]: 
2023-07-01T18:03:40.174350+00:00 app[worker.1]: Creating archive: /app/Downloaded/splitted/854158484/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4
2023-07-01T18:03:40.174350+00:00 app[worker.1]: 
2023-07-01T18:03:40.174350+00:00 app[worker.1]: Items to compress: 1
2023-07-01T18:03:40.174351+00:00 app[worker.1]: 
2023-07-01T18:03:40.174351+00:00 app[worker.1]: 
2023-07-01T18:03:40.174351+00:00 app[worker.1]: Files read from disk: 1
2023-07-01T18:03:40.174352+00:00 app[worker.1]: Archive size: 2431698052 bytes (2320 MiB)
2023-07-01T18:03:40.174352+00:00 app[worker.1]: Everything is Ok
2023-07-01T18:03:40.767309+00:00 app[worker.1]: 2023-07-01 18:03:40,767 - WARNING - unzipper - MainThread - Can't upload files bigger than 2000 MiB
2023-07-01T18:03:40.890045+00:00 app[worker.1]: 2023-07-01 18:03:40,889 - WARNING - unzipper - MainThread - Failed to decode "/app/Downloaded/splitted/854158484/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4.002". The value does not represent an existing local file, HTTP URL, or valid file id.
2023-07-01T18:03:40.894168+00:00 app[worker.1]: 2023-07-01 18:03:40,890 - ERROR - pyrogram.dispatcher - MainThread - [Errno 2] No such file or directory: '/app/Downloaded/splitted/854158484'
2023-07-01T18:03:40.894169+00:00 app[worker.1]: Traceback (most recent call last):
2023-07-01T18:03:40.894169+00:00 app[worker.1]:   File "/venv/lib/python3.11/site-packages/pyrogram/file_id.py", line 228, in decode
2023-07-01T18:03:40.894169+00:00 app[worker.1]:     file_type = FileType(file_type)
2023-07-01T18:03:40.894170+00:00 app[worker.1]:                 ^^^^^^^^^^^^^^^^^^^
2023-07-01T18:03:40.894170+00:00 app[worker.1]:   File "/usr/lib/python3.11/enum.py", line 714, in __call__
2023-07-01T18:03:40.894170+00:00 app[worker.1]:     return cls.__new__(cls, value)
2023-07-01T18:03:40.894170+00:00 app[worker.1]:            ^^^^^^^^^^^^^^^^^^^^^^^
2023-07-01T18:03:40.894171+00:00 app[worker.1]:   File "/usr/lib/python3.11/enum.py", line 1130, in __new__
2023-07-01T18:03:40.894171+00:00 app[worker.1]:     raise ve_exc
2023-07-01T18:03:40.894171+00:00 app[worker.1]: ValueError: -60183811 is not a valid FileType
2023-07-01T18:03:40.894171+00:00 app[worker.1]: 
2023-07-01T18:03:40.894172+00:00 app[worker.1]: During handling of the above exception, another exception occurred:
2023-07-01T18:03:40.894172+00:00 app[worker.1]: 
2023-07-01T18:03:40.894172+00:00 app[worker.1]: Traceback (most recent call last):
2023-07-01T18:03:40.894173+00:00 app[worker.1]:   File "/venv/lib/python3.11/site-packages/pyrogram/utils.py", line 49, in get_input_media_from_file_id
2023-07-01T18:03:40.894173+00:00 app[worker.1]:     decoded = FileId.decode(file_id)
2023-07-01T18:03:40.894173+00:00 app[worker.1]:               ^^^^^^^^^^^^^^^^^^^^^^
2023-07-01T18:03:40.894173+00:00 app[worker.1]:   File "/venv/lib/python3.11/site-packages/pyrogram/file_id.py", line 230, in decode
2023-07-01T18:03:40.894174+00:00 app[worker.1]:     raise ValueError(f"Unknown file_type {file_type} of file_id {file_id}")
2023-07-01T18:03:40.894174+00:00 app[worker.1]: ValueError: Unknown file_type -60183811 of file_id /app/Downloaded/splitted/854158484/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4.002
2023-07-01T18:03:40.894174+00:00 app[worker.1]: 
2023-07-01T18:03:40.894174+00:00 app[worker.1]: During handling of the above exception, another exception occurred:
2023-07-01T18:03:40.894174+00:00 app[worker.1]: 
2023-07-01T18:03:40.894174+00:00 app[worker.1]: Traceback (most recent call last):
2023-07-01T18:03:40.894175+00:00 app[worker.1]:   File "/app/unzipper/modules/ext_script/up_helper.py", line 144, in send_file
2023-07-01T18:03:40.894175+00:00 app[worker.1]:     await unzip_bot.send_document(
2023-07-01T18:03:40.894175+00:00 app[worker.1]:   File "/venv/lib/python3.11/site-packages/pyrogram/methods/messages/send_document.py", line 178, in send_document
2023-07-01T18:03:40.894175+00:00 app[worker.1]:     media = utils.get_input_media_from_file_id(document, FileType.DOCUMENT)
2023-07-01T18:03:40.894175+00:00 app[worker.1]:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-07-01T18:03:40.894176+00:00 app[worker.1]:   File "/venv/lib/python3.11/site-packages/pyrogram/utils.py", line 51, in get_input_media_from_file_id
2023-07-01T18:03:40.894176+00:00 app[worker.1]:     raise ValueError(
2023-07-01T18:03:40.894177+00:00 app[worker.1]: ValueError: Failed to decode "/app/Downloaded/splitted/854158484/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4.002". The value does not represent an existing local file, HTTP URL, or valid file id.
2023-07-01T18:03:40.894177+00:00 app[worker.1]: 
2023-07-01T18:03:40.894178+00:00 app[worker.1]: During handling of the above exception, another exception occurred:
2023-07-01T18:03:40.894178+00:00 app[worker.1]: 
2023-07-01T18:03:40.894178+00:00 app[worker.1]: Traceback (most recent call last):
2023-07-01T18:03:40.894178+00:00 app[worker.1]:   File "/venv/lib/python3.11/site-packages/pyrogram/dispatcher.py", line 240, in handler_worker
2023-07-01T18:03:40.894178+00:00 app[worker.1]:     await handler.callback(self.client, *args)
2023-07-01T18:03:40.894178+00:00 app[worker.1]:   File "/app/unzipper/modules/callbacks.py", line 667, in unzipper_cb
2023-07-01T18:03:40.894179+00:00 app[worker.1]:     await send_file(
2023-07-01T18:03:40.894179+00:00 app[worker.1]:   File "/app/unzipper/modules/ext_script/up_helper.py", line 181, in send_file
2023-07-01T18:03:40.894179+00:00 app[worker.1]:     shutil.rmtree(full_path)
2023-07-01T18:03:40.894179+00:00 app[worker.1]:   File "/usr/lib/python3.11/shutil.py", line 722, in rmtree
2023-07-01T18:03:40.894179+00:00 app[worker.1]:     onerror(os.lstat, path, sys.exc_info())
2023-07-01T18:03:40.894179+00:00 app[worker.1]:   File "/usr/lib/python3.11/shutil.py", line 720, in rmtree
2023-07-01T18:03:40.894180+00:00 app[worker.1]:     orig_st = os.lstat(path, dir_fd=dir_fd)
2023-07-01T18:03:40.894180+00:00 app[worker.1]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-07-01T18:03:40.894180+00:00 app[worker.1]: FileNotFoundError: [Errno 2] No such file or directory: '/app/Downloaded/splitted/854158484'

I assume that the 2g file size is slightly too big

[EDM115]

2023-07-01T18:49:52.288423+00:00 app[worker.1]: 2023-07-01 18:49:52,288 - INFO - unzipper - MainThread - ext_a paths : ['/app/Downloaded/854158484/extracted/Interstellar (2014) (2014) [1080p]/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4', '/app/Downloaded/854158484/extracted/Interstellar (2014) (2014) [1080p]/WWW.YTS.RE.jpg']
2023-07-01T18:50:38.187771+00:00 app[worker.1]: 2023-07-01 18:50:38,186 - INFO - unzipper - asyncio_1 - 
2023-07-01T18:50:38.187783+00:00 app[worker.1]: 7-Zip [64] 17.04 : Copyright (c) 1999-2021 Igor Pavlov : 2017-08-28
2023-07-01T18:50:38.187784+00:00 app[worker.1]: p7zip Version 17.04 (locale=C.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs x64)
2023-07-01T18:50:38.187784+00:00 app[worker.1]: 
2023-07-01T18:50:38.187785+00:00 app[worker.1]: Scanning the drive:
2023-07-01T18:50:38.187785+00:00 app[worker.1]: 1 file, 2431697820 bytes (2320 MiB)
2023-07-01T18:50:38.187785+00:00 app[worker.1]: 
2023-07-01T18:50:38.187786+00:00 app[worker.1]: Creating archive: /app/Downloaded/splitted/854158484/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4
2023-07-01T18:50:38.187786+00:00 app[worker.1]: 
2023-07-01T18:50:38.187786+00:00 app[worker.1]: Items to compress: 1
2023-07-01T18:50:38.187787+00:00 app[worker.1]: 
2023-07-01T18:50:38.187787+00:00 app[worker.1]: 
2023-07-01T18:50:38.187787+00:00 app[worker.1]: Files read from disk: 1
2023-07-01T18:50:38.187787+00:00 app[worker.1]: Archive size: 2431698052 bytes (2320 MiB)
2023-07-01T18:50:38.187787+00:00 app[worker.1]: Everything is Ok
2023-07-01T18:50:38.188058+00:00 app[worker.1]: 2023-07-01 18:50:38,187 - INFO - unzipper - MainThread - ['/app/Downloaded/splitted/854158484/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4.001', '/app/Downloaded/splitted/854158484/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4.002']

Solved with c1528bc and 4afea91

Now it's time to fine tune this, as uploaded files are 1907,3 Mb