What is “Unsupported URL Path” in Scrape errors?

[baccccccc]

I've run CDL against a forum thread. The summary has the following section:

Scrape Failures:
  404 Not Found: 8
      Error-rateLimit: 8
      Unsupported URL Path: 24

my Scrape_Error_URLs.csv has 16 rows (except the header obviously) which correspond to the first two error types.

there's not a single entry for an “Unsupported URL Path”, and I don't see anything relevant in downloader.log either.

the whole Unsupported_URLs.csv is only 14 rows, which is less than 24.

I don't remember seeing these type of errors before. Is it something new and not logged properly yet?

[NTFSvolume]

CDL throws that error when it gets an URL from a supported site but with an unrecognized structure or format. It doesn't know how to scrape it

ex: https://www.reddit.com/settings/

Not logging the URL was a bug. It was fixed by #577. Fix will be included on the next release

[NTFSvolume]

The main log file should still have the messages with the actual URLs

[baccccccc]

can you give an example? I tried to search by the same string (Unsupported URL Path) and did not find anything. Chances are, the log uses a different wording, and I could not grasp it.

[NTFSvolume]

On v6.5.0, the actual log message is Unknown URL path

https://github.com/jbsparrow/CyberDropDownloader/blob/52f59a84df7c500b902ad055273d1d342a71b9fa/cyberdrop_dl/scraper/crawler.py#L297

[baccccccc]

gotcha. Found it:

                    ERROR    Scrape Failed: Unknown URL path: https://cyberfile.su                                                                                                                                                  logger.py:27
                    ERROR    Scrape Failed: Unknown URL path: https://bunkr.cr                                                                                                                                                      logger.py:27

it's hard to find those links at forum, but who the hell would just post links like this? Smells like a parser error to me. But I'll try find out where exactly it finds it.

[baccccccc]

nah, I'm wrong. No parser error. There's a message on the forum that literally says:

use https://bunkr.cr/ or https://cyberfile.su/

so we're good. But I do think that we have to handle it better somehow.

[baccccccc]

and at very least, please keep it consistent. It should be either Unsupported URL Path or Unknown URL path across all the logs. Makes it hella difficult to troubleshoot.

[NTFSvolume]

That was fixed in the same PR i mentioned before. The message was hardcoded as a variable to make sure is the same in the UI, csv logs and the main log

For the next release, the wording will be Unknown URL path

https://github.com/jbsparrow/CyberDropDownloader/blob/2428ec5a3486143190d6c1978adb29f05878569e/cyberdrop_dl/scraper/crawler.py#L29

I can't guarantee that it won't change in a future release, but it will be consistent across logs.