Ideas for v0.3

[Avnsx]

• add scraping content from messages functionality
• maybe change update checks to be scraped from the config file on the github repo, instead of the tag version for executables
• find a way around compability with os.startfile #3
• re-write or improve main scraping function (line 127 and beyond), so that less duplicates are scraped in first place instead of using image hashing to reduce them
• generally improve code performance - in terms of speed

[simmons2714]

If you can, you could check the MD5 of an image before you download it against the the MD5 in the already downloaded directory and if that file exists skip it.

[simmons2714]

Odd because all the duplicate images have the same checksum.

[Avnsx]

Odd because all the duplicate images have the same checksum.

That is correct, but some images are duplicates in terms of having completely different filebytes(and with that also a completely different md5 hash), but they are still representing the same image. The reasoning behind this, are re-uploads of the same image. For example if creators just screenshot and cut the screenshot down to the original image and then upload it again. This process renders using file byte comparison or md5 comparison entirely useless, since the image is the same but it's not really the same in terms of md5 / raw filebytes. Same accounts for videos. Re-uploading a mp4 in a different video format, shortened by a second or edited with a new text on it; still makes it the same video for a human, but not the same in filebytes/md5.

[simmons2714]

On the one hand archivists would want all versions. However, that is a cop out.
If an creator edits a post does it have the same ID? If so, would it be possible to check against already downloaded post ID's? Similar to how youtube-dl compares against URLs. Assuming the div element ID doesn't change randomly.

This would prob be the best way to check not based on hashes.
https://www.pyimagesearch.com/2014/09/15/python-compare-two-images/

But again that and other functions depend on having the image on the system.
It seems as if the best way to solve this is to save a list of the image IDs either by post title or by div ID and compare that against what has already been downloaded.
Another jank way would to be download images after the last time the program was run. However, this method doesn't download if a post was edited.

[Avnsx]

No one wants the same Image twice. It doesn't even have to be drastically different -so not noticeable by humans that it's a different variation- literally editing a single pixel differently or a very slight loss of quality, a not noticeable cropdown already changes the filebytes entirely.

There's no unique identifiers that the API responds with.

For titles the API makes it even harder, by just allowing creators to upload content with no utf-8 title, which then end up having no name at all in the api response or media with the same title; representing a different image/video.

[simmons2714]

Yeah sure if you compare pixel by pixel the program will think two images are different if only one pixel is changed. However, there are a few basic equations that could filter out minor differences like a crop or edit.
Also there's the whole AI aspect but that becomes very resource heavy very quickly.

[simmons2714]

https://github.com/simmons2714/InfiniDrive-OpenStego/blob/main/test.py

Selenium could be an option for scraping the messages. It can be programmed to function similar to a user using a browser. You'd need to pair it with your scraping code tho, I don't think you can just right click save as in private messages.

[Avnsx]

I'm keeping this library as lightweight as possible, I do not want to use selenium yet. Not until I am being forced to by the fansly developers, in terms of; if they somehow manage to make continously reversing the the fansly API a not viable option in the future.

For messages I'm just going to do this: #2 (comment)
I'm aware of this since he pointed it out, but I haven't had the time yet to do any updates in general on this code, since I'm bussy developing two other projects. I would be looking forward to contributions if you're invested into this 😊

[KBNovice12]

Would it be possible to add a way around paywalls? I don't want to offend any sense of ethics but I'm sure I'm not the only one who has been burned with subpar content at premium prices 🚶🤷‍♂️. there's also a scrapper for another popular site that does get around paywalls. But there's no executable version like you have (btw as a noncoder this is very much appreciated). I would contribute if this scrapper could get around the paywalls. Anyway, thanks for taking the time regardless. Cheers!

[Avnsx]

I haven't even tried to bypass their subscription algorithm and there are multiple reasons why I won't in the future;

1. It's very illegal, and I'm not endorsing any illegal activity
2. It would be very damaging to fansly developers, which is not my intention, I'm aiming at co-existing in terms of if I didn't write this repo someone else would've
3. It would force them to constantly patch the bypasses I find, and I would need to counteract on that, which would waste their and my time