Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sending patterns and urls to the instance does not mean they will be crawled? #87

Open
scaraliu opened this issue Jan 21, 2019 · 0 comments

Comments

@scaraliu
Copy link

When sending the allowed patterns and the URL list to the instance/index, does not start the crawl process on that list. I had to use the OpenSearchServer\Crawler\Web\Crawl() request to actually start the crawl process on that list of URLs. Just injecting the urls and setting the allowed patterns doesn't guarantee that those URL will be processed as configured in the running index.

I thought just by injecting urls and setting them as allowed patterns, the instance will process them.
So, to index some urls I have to do as follows:

  • Inject url list OpenSearchServer\Document\Put();
  • Inject those URLs as allowed path/s OpenSearchServer\Crawler\Web\Patterns\Inclusion\Insert();
  • and also to do a crawl request on each URL OpenSearchServer\Crawler\Web\Crawl();

Am I missing something?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant