Add a repeat option for extractors to WebsiteAgent
#1769
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
knu
force-pushed
the
website_agent_repeat_option
branch
2 times, most recently
from
33c914d
to
f67da0b
Compare
cantino
reviewed
Nov 2, 2016
| @@ -42,7 +44,8 @@ class WebsiteAgent < Agent | |||
| "extract": { | |||
| "url": { "css": "#comic img", "value": "@src" }, | |||
| "title": { "css": "#comic img", "value": "@title" }, | |||
| "body_text": { "css": "div.main", "value": "string(.)" } | |||
| "body_text": { "css": "div.main", "value": "string(.)" }, | |||
| "page_title": { "css": "title", "value": "string(.)" } | |||
There was a problem hiding this comment.
Oh yes, that's what I intended. I'll fix it.
knu
force-pushed
the
website_agent_repeat_option
branch
from
051082e
to
d72d773
Compare
This allows user to include a value that only appears once in a content in all events created from the content.
knu
force-pushed
the
website_agent_repeat_option
branch
from
d72d773
to
bd9455d
Compare
|
This allows user to include a value that only appears once in a resource in all events created from it, for all extraction types, which has been possible only in the html/xml extraction types owing to the power of XPath, but never in regexp/json. @cantino What do you think? |
cantino
approved these changes
Nov 19, 2016
| num_tuples = output.each_value.first.size | ||
| num_tuples = output.each_value.inject(nil) { |num, value| | ||
| case size = value.size | ||
| when Float::INFINITY |
There was a problem hiding this comment.
This is what Repeater#size returns, meaning it is an infinite list. num_tuples is being set to a finite number that is the size of finite lists, so it is counted out here.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is useful such as when you want to include the title of a page in all events created from the page. In combination with the
templateoption (#1743) and the_loop_variable (#1768), I guess you could finally achieve crawling of paginated contents properly, by adding the link to the next page to the last event created by a check, incrementing the nest level, feeding it back to the agent via a TriggerAgent with conditions on the presence of the said link and the nest level.The implementation with regard to Repeater may be too tricky, and the wording of documentation seemed far beyond me, and I'm not 100% certain as to the naming of the option, so any feedback is welcome!