Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File Path Mapping is inconsistent / incorrecr #30

Open
twist3dimages opened this issue Jan 10, 2024 · 7 comments
Open

File Path Mapping is inconsistent / incorrecr #30

twist3dimages opened this issue Jan 10, 2024 · 7 comments
Assignees

Comments

@twist3dimages
Copy link

When I run the script I get a lot of these messages (running on Windows now)

Info: There's a file in this OnlyFans metadata database that we couldn't find in your Stash database but the file IS on your filesystem.
Try running a Scan Task in Stash then re-running this script or changing your Search Specificity mode to Low.

  • user_data/sites5/OnlyFans<username>/Posts/Free/Videos\0h9j9rz7ipd9bb7nfn3kp_source.mp4

Info: There's a file in this OnlyFans metadata database that we couldn't find in your Stash database but the file IS on your filesystem.
Try running a Scan Task in Stash then re-running this script or changing your Search Specificity mode to Low.

  • user_data/sites5/OnlyFans<username>/Posts/Free/Videos\0hakgfjsx4unbkgs2v2dg_source.mp4

Notice every slash is a / except for before the filename

image

I have confirrmed these files exist

@ALonelyJuicebox
Copy link
Owner

Oh that's interesting. Did you scrape OF on a *nix based operating system and then run this script while on Windows?
Under the hood, the metadata database stores the directory and the filename separately, and this script needs the full path to the media. To handle that, this script makes an educated guess as to what directory delineator is appropriate (/ or ) based on the operating system that you're currently running the script on, which partially explains what you're seeing.

It's strange though that there seem to be folders inside your filename or something...

We can double check my theory though-- are you willing to help troubleshoot a bit? (otherwise I'll need access to that metadata database of yours)

  • Download DB Browser for SQLite https://sqlitebrowser.org/
  • Click on the Browse Data tab
  • Under the dropdown for Table, click on Medias
  • Under the column for filename there's an empty row that will say "Filter" in it. Enter "0hakgfjsx4unbkgs2v2dg_source.mp4"
  • The program should now just show you a single row of results.
  • Send over the directory and the filename from that row. (you can obfuscate as much as you need)

@twist3dimages
Copy link
Author

twist3dimages commented Jan 10, 2024

Oh that's interesting. Did you scrape OF on a *nix based operating system and then run this script while on Windows? Under the hood, the metadata database stores the directory and the filename separately, and this script needs the full path to the media. To handle that, this script makes an educated guess as to what directory delineator is appropriate (/ or ) based on the operating system that you're currently running the script on, which partially explains what you're seeing.

It's strange though that there seem to be folders inside your filename or something...

We can double check my theory though-- are you willing to help troubleshoot a bit? (otherwise I'll need access to that metadata database of yours)

  • Download DB Browser for SQLite https://sqlitebrowser.org/
  • Click on the Browse Data tab
  • Under the dropdown for Table, click on Medias
  • Under the column for filename there's an empty row that will say "Filter" in it. Enter "0hakgfjsx4unbkgs2v2dg_source.mp4"
  • The program should now just show you a single row of results.
  • Send over the directory and the filename from that row. (you can obfuscate as much as you need)

Haha yes, downloaded on Linux and ran on Windows until DigitalCriminal stopped updating it, so using OF-DL on my windows machine. Here's the screenshot you requested. I am not sure why the format is like that

image

Notice the \ and / where I marked over

I am not sure if this is already implemented, but i had to end up creating a bunch of directories like

user_data/sites5
user_data/sites

etc. Maybe a feature request to see if the file exists and match off the filename, for example i may have moved files from user_data/sites to user_data/sites5

@ALonelyJuicebox
Copy link
Owner

Yeah I can't quite explain why your filesystem looks the way that it does unless the OF scraper just...doesn't handle linux filepaths correctly. There's no reason I can think of that the OF scraper would put the wrong folder delineation character after the username of a performer but then put normal delineation characters for everything after that. Very strange indeed. 👀
Anyway, that's an oddity outside of this script, and it shouldn't technically matter tbh.

Two more quick questions--
1 - What did you set your Scan Specificity mode to? If you're unsure, just open the OFMetadataToStash_Config file in a text editor

2 - While you confirmed that the files exist on the filesystem, are you sure they actually exist in Stash?

The reason why I'm asking is that if you have your scan mode set to normal, it matches based on ensuring the performer name is somewhere in the path and that the filesize matches between what's in Stash and what's in the metadata database.
Based on what you've shown me so far, I don't see any reason why the performer name wouldn't be in the path somewhere, and there shouldn't be an issue matching the filesize.

Alternatively you could set the scan mode to "low" and see how that goes.

@twist3dimages
Copy link
Author

Yeah I can't quite explain why your filesystem looks the way that it does unless the OF scraper just...doesn't handle linux filepaths correctly. There's no reason I can think of that the OF scraper would put the wrong folder delineation character after the username of a performer but then put normal delineation characters for everything after that. Very strange indeed. 👀 Anyway, that's an oddity outside of this script, and it shouldn't technically matter tbh.

Two more quick questions-- 1 - What did you set your Scan Specificity mode to? If you're unsure, just open the OFMetadataToStash_Config file in a text editor

2 - While you confirmed that the files exist on the filesystem, are you sure they actually exist in Stash?

The reason why I'm asking is that if you have your scan mode set to normal, it matches based on ensuring the performer name is somewhere in the path and that the filesize matches between what's in Stash and what's in the metadata database. Based on what you've shown me so far, I don't see any reason why the performer name wouldn't be in the path somewhere, and there shouldn't be an issue matching the filesize.

Alternatively you could set the scan mode to "low" and see how that goes.

  1. It's set to low

  2. they do

I know it's fucked haha, is there any way you can add something to check the file system and if windows replace every / with \ in the directory and vice versa on Linux?

@ALonelyJuicebox
Copy link
Owner

This wouldn't address the problem though-- on "low" this script just asks the Stash db if there's a file in the Stash database that matches the filesize of the file in your OF DB. No filepaths are involved (outside of the error message doing its own file system check to try and help the user sort out what the deal is)

Let me do some digging and see if I can replicate this in some form while having this script set to low.

@ALonelyJuicebox ALonelyJuicebox self-assigned this Jan 16, 2024
@twist3dimages
Copy link
Author

twist3dimages commented Jan 27, 2024

The new update correctly identifies the images (even with the \ and / being wonky) but not videos anymore. I just scanned. Doing it on my linux server and windows

Info: There's a file in this OnlyFans metadata database that we couldn't find in your Stash database but the file IS on your filesystem.
Try running a Scan Task in Stash then re-running this script or changing your Search Specificity mode to Low.

  • user_data/sites5/OnlyFans<username>/Posts/Free/Videos/0hakh4igch3q5s2supy2m_source.mp4

Info: There's a file in this OnlyFans metadata database that we couldn't find in your Stash database but the file IS on your filesystem.
Try running a Scan Task in Stash then re-running this script or changing your Search Specificity mode to Low.

  • user_data/sites5/OnlyFans<username>/Posts/Free/Videos/0h4x53e7owkhxlzn7nqpa_source.mp4

Info: There's a file in this OnlyFans metadata database that we couldn't find in your Stash database but the file IS on your filesystem.
Try running a Scan Task in Stash then re-running this script or changing your Search Specificity mode to Low.

  • user_data/sites5/OnlyFans<username>/Posts/Free/Videos/0h75zfmqxevnlceah7bzd_source.mp4

  • Added metadata to Stash's database for the following file:
    user_data/sites5/OnlyFans<username>/Posts/Free/Images/2316x3064_7c6812e6016a3decc3364b4a5e8aefba.jpg

  • Added metadata to Stash's database for the following file:
    user_data/sites5/OnlyFans<username>/Posts/Free/Images/2316x3088_0d56f9459851ccd51f65cf85d0b878ca.jpg

  • Added metadata to Stash's database for the following file:
    user_data/sites5/OnlyFans<username>/Posts/Free/Images/2316x3088_1d33e528dbc39fb995b81f3d880a6efb.jpg

  • Added metadata to Stash's database for the following file:
    user_data/sites5/OnlyFans<username>/Posts/Free/Images/2316x3088_2415ced324ce9484b9dfcbcbb8daa2da.jpg

  • Added metadata to Stash's database for the following file:
    user_data/sites5/OnlyFans<username>/Posts/Free/Images/2316x3088_2c8207aad6bb80c7e80be79c5b7ede21.jpg

  • Added metadata to Stash's database for the following file:
    user_data/sites5/OnlyFans<username>/Posts/Free/Images/2316x3088_2cbf02595a2cec9cda38a232d076d646.jpg

  • Added metadata to Stash's database for the following file:
    user_data/sites5/OnlyFans<username>/Posts/Free/Images/2316x3088_6497f80930abb0cd81485883776176f7.jpg

  • Added metadata to Stash's database for the following file:
    user_data/sites5/OnlyFans<username>/Posts/Free/Images/2316x3088_c33edec86703c2d1419c1a7a5817db83.jpg

  • Added metadata to Stash's database for the following file:
    user_data/sites5/OnlyFans<username>/Posts/Free/Images/2316x3088_ed0c3f7915f02c010d371c68972616b8.jpg

@ALonelyJuicebox
Copy link
Owner

Hmm...the plot thickens. Can you go back to normal specificity mode one more time?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants