Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to get content dynamically generated by js #19

Closed
noecs opened this issue Aug 9, 2020 · 1 comment
Closed

Unable to get content dynamically generated by js #19

noecs opened this issue Aug 9, 2020 · 1 comment

Comments

@noecs
Copy link

noecs commented Aug 9, 2020

Hello, the webpage I want to crawl, because it is a node dynamically generated by js. So I try to use Geziyor to get the content I want, and I have two questions.
First, I used the official example to get the attribute value of the src of the video tag, and it was still unable to print out the attribute value (code is below).
Second, whether the content dynamically generated by js can open multiple requests like getting a static page.

package main

import (
	"fmt"
	"github.com/geziyor/geziyor"
	"github.com/geziyor/geziyor/client"
)

func main() {
	geziyor.NewGeziyor(&geziyor.Options{
		// set UserAgent
		UserAgent: "Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1",
		//
		StartRequestsFunc: func(g *geziyor.Geziyor) {
			g.GetRendered("https://m.douyu.com/100", g.Opt.ParseFunc)
		},
		// Handle response
		ParseFunc: func(g *geziyor.Geziyor, r *client.Response) {
			if src, ok := r.HTMLDoc.Find("#html5player-video").Attr("src"); ok {
				fmt.Println("src is:", src)
			}
		},
	}).Start()
}
@musabgultekin
Copy link
Collaborator

Hi, in https://m.douyu.com/100 page, I don't see src value of video player. Are they changed the html?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants