Last Release

Unfortunately, this will be the final release for the project. I haven't made changes to the project in some time and won't be for some time. I've merged all pending pull requests. This project will be archived soon. Thank you all for using this project.

v0.6.2.3 Reupload

WARNING: DOWNLOAD FROM #29 AS IT FIXES A CRITICAL BUG CAUSED BY A KEMONO PARTY UPDATE. DO NOT USE THE FILES PROVIDED IN THIS RELEASE. VIEW #33 FOR MORE DETAILS.

Took the release down for a bit to remove instances of "Published" in the file name which was a result of a recent Kemono UI update I wasn't aware of. Sorry for any inconveniences this may have caused.

Pretty significant update this time around aimed at fixing the issues of dupe files and adding --RENAME switch which took a lot of work. If you encounter any issues with this update, submit an issue and use the last version which only has bug fixes and no major code changes like this release. I will spend some time this week working on the wiki.

1. --RENAME switch

Straightforward switch, if you download an works and the program finds these works already exists, they will be renamed accordingly. I will include some cases where --RENAME switch is used:

Default case

We begin with this case. The switches used will be shown on the top of each case. URL used is "https://kemono.party/patreon/user/574050?q=dog+girl" (R-18 warning):
"C:/Users/Jeff Chen/AppData/Local/Microsoft/WindowsApps/python3.10.exe" m:/Projects/KMP-Kemono-Downloader/KMPDownloader.py -d ".\Sample"

Rename case

Rename the files but do not include id or date:
"C:/Users/Jeff Chen/AppData/Local/Microsoft/WindowsApps/python3.10.exe" m:/Projects/KMP-Kemono-Downloader/KMPDownloader.py -d ".\Sample" --RENAME --id --date

Note that if unzipped files exist in the old directory, it will not be moved and another zip file will be downloaded and unzipped in the new directory. This also means that the old directory will not be deleted since it is not empty. --RENAME only works with --id and --date and cannot be used to change unpacked to packed file structure and vice versa.

Revert case

If I want to revert back to the directories shown in the default case, I will need to use the --RENAME switch. Not using the switch results in the following directories during program execution:

However, after execution, you will end up with what you started with:

Now with the --RENAME switch, we now get the result we want:
"C:/Users/Jeff Chen/AppData/Local/Microsoft/WindowsApps/python3.10.exe" m:/Projects/KMP-Kemono-Downloader/KMPDownloader.py -d ".\Sample" --RENAME

2. Improved duplicate works downloading

Added another duplication check layer during the URL scraping phase which fixed the issues of overwriting encountered when downloading works with the same name. For example, "https://kemono.su/patreon/user/574050?q=toriel" (R-18 warning) has 5 works with the same name. When downloaded with --id and --date switches, we now get:

3. --id switch

Prepends work ID to the beginning of the file or folder depending on unpacked or packed download structure

4. Default switches

--id and --date is now on by default. Using --id or --date will turn off the switch instead of enabling it.

5. Automatic dupe file resolution

If a file was downloaded multiple times mistakenly, it will now automatically removed or renamed according to the circumstance. For example:

Folder

• 0.jpg (100b)
• 0 (0).jpg (100b)

When running the program, if it happens to download 0.jpg (100b) again, it will remove the 0 (0).jpg, resulting in:
Folder

• 0.jpg (100b)

6. UPDATE database updated for different prefixes

When downloading works with different prefixes (.party, .su), prefixes are ignored in the update database and only the artist id and service portion of the url is taken into account. Databases with the same artist with multiple prefixes are automatically removed when the artist work is updated.

7. Fixed bug for dupe entries in file preload system

Reduces memory use and fixes bugs from the duplicate entries.

8. Bad switch reminders

When entering a switch incorrectly, the program will report this to the user instead of crashing outright. This feature needs more work to cover all the cases but works for many cases as for this release.

9. Zip error reporting

Zip errors are now reported to the log instead of being skipped.

V0.6.2.2.1-pre

Fixes various bugs:

• Fixes infinite retries due to broken content link
• Fixes date issue for Gumroad
• Fixes issue with embedded divs in post content
• Fix for zip file extractions with trailing spaces

Next release is nearly ready to go, it introduces some new features as well as a slight overhaul to duplicate file matching system to account for different variations in file names as well as introducing a switch that allows renaming of files automatically if they exists on your system already. It should be released in around a week.

0.6.2.1-pre

Fixed issue mentioned in #20

v0.6.2-pre

A small update adding some extra switches and changes to program behavior:

• Added a switch to change logging level. Accessible by using the "-q --logging <#>" switch
• Added disable file prescan switch. Accessible by using the "-k --disableprescan" switch. This will save lots of memory and lower project runtime for a lower level of duplicate file detection.
• Added a custom url prefix switch addressing the kemono banner addressing users to bookmark kemono.su. Accessible by using the "-j --prefix " switch.
• UPDATE now ignores kemono url prefixes and uses suffixes only. Basically, UPDATE switch will always work regardless of the URL prefix used unless there is a change in the suffix which is highly unlikely.
• Added a switch to append or insert artist work date to downloads. For normal download mode, this is the folder. For unpacked download mode, this is the individual download items. Accessible through the "-w --date" switch Addresses #14
• Improved scanning files terminal output, will display what is being scanned currently.
• Old default download method has been replaced by the EXPERIMENTAL method. You may continue to use the old download method by using the "--DEPRECATED" switch.

As always, I will edit the wiki shortly after this release.

Some Updates

I have been thinking recently about opening a donation option for this repo. Nothing will be paywalled, it's more of a way for you all to express your appreciation for this project. This project has been a quality learning experience for me and it has exceeded my expectations as my first maintained open source project. If you have an issue with me adding a way to donate to this project, submit an issue and I'll have a look over it.

On a other note, the project now has a switch for every character in the English alphabet! I didn't even know I had that many switches until I had to count the alphabet several times over for the above updates. If you any issues you come across, please submit an issue and possibly revert to the last release as that is stable.

v0.6.1.4-pre

Added pip reqs file and bat script to handle dependency issues mentioned here #16. I will update the wiki and other documentation in a future. Follow the instructions below if you are having any issues with concerning #16.

No issues

You do not need to do anything, continue using v0.6.1.3. No code changes have been made.

You are having issues mentioned in #16

Open CMD or an equivalent shell and enter "pip uninstall urllib3". Then enter "pip install urllib3==1.26.14". The problem should go away afterwards.

If the above method does not work

Navigate to the program directory and open CMD. Enter "pip install -r requirements.txt". This method will only work if dependency related libraries do not exists in your system. You may reference installed dependencies with "pip freeze" and uninstall them using "pip uninstall ".

If the above method does not work or you are a new user.

Clone a fresh instance of the repository. Open program files and open "install_requirements.bat". This will create a virtual environment, install all correct packages to it, and transfer the patched patoolib files. This will be the recommended way for users to install dependencies from this point on but existing users do not need to follow this process. Note that you will always have to enter "venv\Scripts\activate" before running the program. You can check to see if venv is activate if "(venv)" can be seen from the prompt in the CMD or terminal you are using.

Updates

I'm planning to keep working on this project until version 0.8 or 0.9. Although details of these versions are not mentioned in the roadmap, you can expect to see some sort of working GUI. I will also be making several changes to future proof the application by having an alternate download route using Kemono's API and more. I won't be able to dedicate to much time to this project but I do intend to start working on it weekly. Here are some things that will change as the project comes to a close:

1. Deprecation of basic download mode. Program has 2 ways to download, it can download post by post or aggregate all posts together and download it all in the end (--EXPERIMENTAL switch).
2. Discontinued Discord Download support: I won't be providing any updates to the Discord download feature. Unless there is enough support, I won't consider updating it in the future.

What to expect from the GUI

My plans of the GUI is to convert to wrap the existing KMPDownloader code into a Flask API that will be run concurrently with a JS Web Application. This approach shouldn't require too much modification for the code to accommodate a front end web app. Download and running requirements may have to change drastically but it is a worthy tradeoff. Currently, I have most of the static website finished. However, a fully functional and connected GUI is still months away from being finished and released.

v0.6.1.3-pre

Update to the track feature as it was left off the switches tested in v0.6.1. You do not need this version if you are not using the --track switch. v0.6.1.2 will be fine.

• Fix to --track switch broken in v0.6.1

v0.6.1.2-pre

Hotfix for issue #11

v0.6.1.1

Here are some changes I made in this release:

• Improved file preload performance significantly
• Changed hashing algorithm which reduces collisions in file scanning use case significantly
• Added reupdate feature, this will redownload everything from the update db
• Added switch to use different db name
• Added switch to change download format types
• Updater ignores update entries with a nonexistant path
• Slightly decreased memory usage for file preload (~9% decrease)
• Now only scans text files created by the program.

Dependencies:

• pip install mmh3

Preload should be good to go now. The main issue was due to the program's hash algorithm not being good for similar case strings. This has been replaced with mmh3 which performs significantly better. The other issues associated with the previous release has been fixed.

Moving forward, new features will not be added for a while. I plan on rewriting and moving around some parts of the code which will naturally decrease bugs.

v0.6.1-pre

There is currently an issue with post content data gathering, download the previous release instead of this one! Next release will be coming in a few days to address several concerns

Lots of things in this update pertaining to reliability.

• HTTP retries no longer infinite by default
• Post content with embedded containerlized elements like embedded GDrive files links are saved to file
• Duplicate files with now have space in between filename and () to conform to windows standards.
• Duplicate file algorithm now takes into account locally available files, this includes files with old dupe and new dupe naming scheme.
• Duplicate file algorithm no longer overwrites local files
• Temporarily removed resume download when stopped due to HTTP errors, download now completely restarts
• Fixed issue where program hangs up on certain post attachments due to not being supported for download
• Database is now backed up when database related switches are used
• Database updated to contain 2 new entries "artist" and "config", old databases are automatically updated to this new format
• Database is now updated at the end of program execution to lower locking issues
• Increased download thread ceiling to 5
• HTTP code 502 added to HTTP retry list

Program should run a lot more reliably now. I was able to test a significant number of downloads without encountering into any download ending issue.

So far, --UPDATE seems to be working fine; however, depending on the number of directories used, it can start to consume a significant amount of ram due to an oversight in my hashtable in file prescan. This oversight was due to the table not being built to handle huge number of items (100s of thousands). There was also a small issue with the hash function as it was the first ever python I wrote and the porting over from Java code was not done properly. This has been fixed and will be released in the next update in a few days.

All documentation and information on how to use the program has been moved to the wiki