Diagnose and fix/suppress chromedp errors

[jpahm]

Currently in the process of scraping, chromedp will log various errors that don't seem to have any actual negative impact on the scraping process, i.e.:

ERROR: could not unmarshal event: unknown ClientNavigationReason value: initialFrameNavigation

and other, similar errors. These errors clutter the logs and should either be fixed or suppressed.

This issue involves performing two tasks:

• Identify the cause of the errors -- particularly, whether they're due to an error on our part or a problem with chromedp itself
• Fix or suppress the errors accordingly

[sreenidhi-gs]

Hello! This is my first task and I would love to try and fix this issue.

[jpahm]

Hello! This is my first task and I would love to try and fix this issue.

Sounds great, it's all yours!

[jpahm]

Hello! This is my first task and I would love to try and fix this issue.

Hi, is there any update on this?

[sreenidhi-gs]

Hello! I am currently working on this, but was not able to get much done as I was studying for finals. I will try getting it done by end of this week!

[jpahm]

@sreenidhi-gs I'm going to open this back up since it's been a while -- feel free to claim it again if you're still interested in it.

[SoggyRhino]

Hi, I just joined and I think I found the cause.

Page.ClientNavigationReason is marked as Experimental in the Chrome Devtools Protocol and it looks like value initialFrameNavigation was only recently added to the spec.

Cdproto added support on August 14, 2024 with Commit b931b75

// cdproto/page/types.go 

const (
	ClientNavigationReasonAnchorClick            ClientNavigationReason = "anchorClick"
	ClientNavigationReasonFormSubmissionGet      ClientNavigationReason = "formSubmissionGet"
	ClientNavigationReasonFormSubmissionPost     ClientNavigationReason = "formSubmissionPost"
	ClientNavigationReasonHTTPHeaderRefresh      ClientNavigationReason = "httpHeaderRefresh"
	ClientNavigationReasonInitialFrameNavigation ClientNavigationReason = "initialFrameNavigation"    // <- this is new 
	ClientNavigationReasonMetaTagRefresh         ClientNavigationReason = "metaTagRefresh"
	ClientNavigationReasonOther                  ClientNavigationReason = "other"
	ClientNavigationReasonPageBlockInterstitial  ClientNavigationReason = "pageBlockInterstitial"
	ClientNavigationReasonReload                 ClientNavigationReason = "reload"
	ClientNavigationReasonScriptInitiated        ClientNavigationReason = "scriptInitiated"
)

// MarshalEasyJSON satisfies easyjson.Marshaler.
func (t ClientNavigationReason) MarshalEasyJSON(out *jwriter.Writer) {
	out.String(string(t))
}

// MarshalJSON satisfies json.Marshaler.
func (t ClientNavigationReason) MarshalJSON() ([]byte, error) {
	return easyjson.Marshal(t)
}

// UnmarshalEasyJSON satisfies easyjson.Unmarshaler.
func (t *ClientNavigationReason) UnmarshalEasyJSON(in *jlexer.Lexer) {                  
	v := in.String()
	switch ClientNavigationReason(v) {
	case ClientNavigationReasonAnchorClick:
		*t = ClientNavigationReasonAnchorClick
	case ClientNavigationReasonFormSubmissionGet:
		*t = ClientNavigationReasonFormSubmissionGet
	case ClientNavigationReasonFormSubmissionPost:
		*t = ClientNavigationReasonFormSubmissionPost
	case ClientNavigationReasonHTTPHeaderRefresh:
		*t = ClientNavigationReasonHTTPHeaderRefresh
	case ClientNavigationReasonInitialFrameNavigation: // <- this is new 
		*t = ClientNavigationReasonInitialFrameNavigation                                
	case ClientNavigationReasonMetaTagRefresh:
		*t = ClientNavigationReasonMetaTagRefresh
	case ClientNavigationReasonOther:
		*t = ClientNavigationReasonOther
	case ClientNavigationReasonPageBlockInterstitial:
		*t = ClientNavigationReasonPageBlockInterstitial
	case ClientNavigationReasonReload:
		*t = ClientNavigationReasonReload
	case ClientNavigationReasonScriptInitiated:
		*t = ClientNavigationReasonScriptInitiated

	//older versions dont support initialFrameNavigation so we reach the default and error
	default:                                                                               
		in.AddError(fmt.Errorf("unknown ClientNavigationReason value: %v", v)) 
	}
}

What is happening is that we are reaching the default case and erroring since we our version of cdproto doesn't support the reason initialFrameNavigation.

Chromedp version 10.0.0 (what we are using) requires Cdproto v0.0.0-20240801214329-3f85d328b335, which was released August 1, 2024, before the update.

Chromedp versions 10.0.1+ use updated versions of Cdproto

Basically we just need to update Chromedp, I tested with the latest version (12.0.1) and the errors were gone. I don't see any reason not to use the latest version. On the same note there are newer versions of mongo-driver and goquery.

[jpahm]

Awesome, thank you for looking into this! Feel free to make a PR that performs the necessary updates if you'd like proper credit for the fix.

[democat3457]

This issue should be fixed with #49

[jpahm]

This issue should be fixed with #49

Right, missed this, thank you!