-
Notifications
You must be signed in to change notification settings - Fork 65
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments for List Google Pages Indexed for SEO: Two Step How To #234
Comments
I'd suggest using curl and bash's {} grouping operators to stream content to sed like this. You could also check out my post on poor man's concurrency with bash to run several of these processes at once. { for i in http://www.google.com http://www.backcountry.com;do curl $i;done; } | sed -e '/stuff/d' |
Hi Shane, Thanks for the suggestion. However, running: { for i in "http://www.google.com/search?num=100&as_sitesearch=www.endpoint.com"; do wget $i; done; } | ... or { for i in "http://www.google.com/search?num=100&as_sitesearch=www.endpoint.com"; do curl $i; done; } | ... triggers a 403 (forbidden) response, so it would require some hacking to get around forbidden script requests to Google. Perhaps I'll play around with the User Agent settings and find a way to successfully make curl requests in the future. ~Steph |
Good point. Google seems to require the user agent string. Using something as simple as seems to work. The linkscape api is a great tool for this sort of thing also. |
Yes, I've been thinking about working with the Linkscape API. According to the API docs, from the free (limited) API, you can grab: I would love to integrate the Linkscape API into my SEO workflow. |
this may be a stupid question, but how do i run the command? Is it done through the browser? |
Worked great just by typing the command where you would otherwise enter the URL address in your browser window. |
Hi Steph, I ran the sed command in terminal but don't know how to view the output of the url's you showed. Would you have any advice for a novice??? Thanks, |
Reiki, The results of the sed command should output directly into the terminal. If you want, you can output them into a file by appending "> filename" to the end of the command, and using a text editor (vi, emacs, notepad, gedit, etc.) to read the file. ~Steph |
Comments for https://www.endpointdev.com/blog/2009/12/google-pages-indexed-seo/
By Steph Skardal
To enter a comment:
The text was updated successfully, but these errors were encountered: