Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug Report] Possibly outdated chromedp causes unmarshaling error when scraping with CDP (unknown PrivateNetworkRequestPolicy value) #3733

Closed
devnsfw opened this issue May 15, 2023 · 5 comments · Fixed by #4107
Labels
bug Something isn't working

Comments

@devnsfw
Copy link

devnsfw commented May 15, 2023

Describe the bug
Can't use CDP with some scrapers on macOS

To Reproduce
Steps to reproduce the behavior:

  1. Run Stash on macOS
  2. Add a scene by URL with a scraper that uses CDP (such as Brazzers or Kink)
  3. A red banner with an error appears briefly complaining about not being able to process OuterHTML
  4. In the log, several instances of ERROR: could not unmarshal event: unknown PrivateNetworkRequestPolicy value are logged

Expected behavior
The scenes can be parsed normally and no extraneous messages are added to the log

Stash Version: (from Settings -> About): 0.20.2 (build hash e685f80)

Desktop (please complete the following information):

  • OS: macOS 12.5.1
  • Browser: Firefox Developer Edition
  • Version: 114.0b3

Smartphone (please complete the following information):
No smartphone involved

Additional context
Tried with CDP based on Brave, Chromium and Chrome (all newest versions)
After looking up the error message, it appears that is related to old chromedp not being able to handle a new value for the PrivateNetworkRequestPolicy, but these changes appear to be at least 8 months old. And not seeing this issue being reported here before makes me suspect it is the Mac build of Stash which is made with an outdated version of chromedp. It could be that it is cross compiled and this part is not properly tested by the people preparing the builds.

@devnsfw devnsfw added the bug report Bug reports that are not yet verified label May 15, 2023
@jeremymeyers
Copy link
Collaborator

Seeing this on Windows with 0.21.0 and chromedp headless Chrome/114.0.5735.134"

Attempting to do a scrape on a brazzers URL leads to this error and this redbox:

Screenshot 2023-06-25 233950

@DefinitelyNotSpicy
Copy link

It seems stash is just using a chromedp version that's too old.
This comment says we need at least v0.7.7 but Stash is using v0.7.3.

It is a bit weird that the issue isn't more widely reported here. Maybe it only happens under certain circumstances?

@devnsfw
Copy link
Author

devnsfw commented Oct 8, 2023

yeah, it's possible that only certain content triggers it, or that some users decide that this technical stuff is too complex already and they'll just wait until they can pull from stashdb.

@DingDongSoLong4
Copy link
Collaborator

@its-josh4 Would you be able to update chromedp in #4107 to fix this issue? Just a bump to v0.7.8 seems like it would do the job, although the latest v0.9.2 would of course be the best if you're willing to make any necessary changes.

its-josh4 added a commit to its-josh4/stash that referenced this issue Oct 8, 2023
@its-josh4
Copy link
Contributor

@its-josh4 Would you be able to update chromedp in #4107 to fix this issue? Just a bump to v0.7.8 seems like it would do the job, although the latest v0.9.2 would of course be the best if you're willing to make any necessary changes.

@DingDongSoLong4 done

@WithoutPants WithoutPants added bug Something isn't working and removed bug report Bug reports that are not yet verified labels Oct 26, 2023
@WithoutPants WithoutPants added this to the Version 0.24.0 milestone Oct 26, 2023
WithoutPants pushed a commit that referenced this issue Oct 26, 2023
* Update a number of dependencies (incl. CVE fixes)

Includes some dependencies that were upgraded in #4106 as well as a few more dependencies.

Some deps that have been upgraded had CVEs.

Notably, upgrades deprecated dependencies such as:
- `github.com/go-chi/chi` (replaced with `/v5`)
- `github.com/gofrs/uuid` (replaced with `/v5`)
- `github.com/hashicorp/golang-lru` (replaced with `/v2` which uses generics)

* Upgraded a few more deps

* lint

* reverted yaml library to v2

* remove unnecessary mod replace

* Update chromedp

Fixes #3733
halkeye pushed a commit to halkeye/stash that referenced this issue Sep 1, 2024
* Update a number of dependencies (incl. CVE fixes)

Includes some dependencies that were upgraded in stashapp#4106 as well as a few more dependencies.

Some deps that have been upgraded had CVEs.

Notably, upgrades deprecated dependencies such as:
- `github.com/go-chi/chi` (replaced with `/v5`)
- `github.com/gofrs/uuid` (replaced with `/v5`)
- `github.com/hashicorp/golang-lru` (replaced with `/v2` which uses generics)

* Upgraded a few more deps

* lint

* reverted yaml library to v2

* remove unnecessary mod replace

* Update chromedp

Fixes stashapp#3733
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants