Skip to content

Actions

Jump to bottom
sixcious edited this page Mar 22, 2024 · 13 revisions

Actions allow you to find the next page in a series of pages.

These are the four available Actions:

  1. Next Link
  2. Click Element
  3. Increment URL
  4. URL List

Note: Actions are only compatible with certain append modes. See the compatibility chart for more details.

How to decide on which action to use?

This depends on the website, as each site is designed differently.

Many websites feature a link to the next page, so you'll probably want to first try using the Next Link action. More modern AJAX sites that don't have links will require you to use the Click Element action. If the URL has an incrementable sequence like a page=1 in it, you can use the Increment URL action. Finally, if you already have a list of URLs to navigate to, you can use the URL List action.

These actions cover practically every possible use-case you might run into.

Next Link

Many pages have a link that take you to the next page.

Automatically Finding the Next Link

Infy will automatically try to find the next link using its algorithm, so you may not need to configure anything. The algorithm first attempts to use the Next Link Path (Selector or XPath). The default Selector path [rel='next'] works on many websites. If it can't find a link using the path, it then tries using the Next Keywords (found in the Options Page) to parse the entire page for a link that has one of those keywords. You can edit the path in Infy's Window and Infy will re-try to find the next link automatically, giving you total control.

Manually Finding the Next Link

If it's not finding the next link automatically from the default path, you'll need to manually set it yourself. There's two ways to do this:

  1. DevTools
  2. Element Picker

Important: You should also make sure there actually is a link on the page. Some websites are actually Single-page Applications that use AJAX techniques, not links. You'll need to inspect the page using DevTools to verify there's a link.

Prev Link

Need to follow the Previous Link instead? You can change the Link Type from Next to Prev and Infy will start using the Prev Link Path and Keywords.

Finding the Next Link Path

  1. On the page you are viewing, right click on the link
  2. Inspect to open your Browser's DevTools
  3. In DevTools, under the Elements tab, Right-click the HTML link element you found. Hint: next links are usually inside an a element
  4. Copy selector or Copy Xpath to get the path for the link element. You may want to alter the path if it seems too verbose/specific

A Note on Generated Next Links

If you generated the path using DevTools and it uses nth-child (Selector) or [Number] (XPath), this path may fail to find the link on the next page in the HTML Markup. For example, imagine a paginated list of numbered links like Page 1, Page 2, Page 3, .... Now, say you're on Page 1 and the generated Selector found Page 2 to be the 2nd element in the list and therefore used nth-child(2). While this will work to find Page 2, it won't work to find Page 3 (and future pages) because Page 3 is the 3rd element, or nth-child(3), and your path, which was using nth-child(2), won't work anymore. In this case, you will need to write a custom path that takes into account sibling and/or parent relationships.

Next Prev Link Algorithm (Paths and Keywords)

Infy's algorithm is designed to try to find the next or prev link first via the Selector or XPath expression, and then fall back to parsing the page using the Keywords. However, this "test" on whether to use the keywords is only done the first time it tries to find the link.

Next Link Algorithm: An Example

For example, say Infy found the next link using the path initially (on page 1), and then fails to find the next link using the path on page 2. Infy won't fall back on the keywords in that situation and will give up. In other words, Infy will only try using Keywords if it failed to find the next page the very first time on page 1 (but it will continue to keep using the keywords on subsequent pages since it knows it failed to do it using the path on page 1).

After spending several hours testing several websites, I found this strategy to hit that "just right" Goldilocks approach and keep it from being too aggressive.

Database URLs Won't Default to Keywords

This principle also applies to Database URLs, which are always designed to have an XPath expression to find the next link. You'll never see Infy fall back to using keywords on these URLs.

Overriding The Algorithm

That said, you can always override this and have Infy always fall back on Keywords no matter what. You can do this by manually going into Infy's Popup Window and entering the Setup and checking the Look for links using keywords if the path fails checkbox.

Click Element

If the website you're dealing with doesn't have a next link or incrementable URL, this will be your next-best option. Some websites have an annoying Load More Button, and Infy can click that button for you as you scroll. You may also be on an AJAX site and will need Infy to click a Next button. You'll need to provide the path for the button the same way you did for finding the next link.

Finding the Click Element Path

  1. On the page you are viewing, right click on the button
  2. Inspect to open your Browser's DevTools
  3. In DevTools under the Elements tab, Right-click the HTML button element you found. It's usually going to either be a button, div, or a element
  4. Copy selector or Copy Xpath to get the path for the button element. You may want to alter the path if it seems too verbose/specific

Increment URL

If the URL has a number or something that can be incremented like a page=1, you can try using this action. Incrementing a URL is fairly straightforward: simply select the part of the URL you want incremented (e.g. the 1 in the above example), and Infy will increment it based on the settings you set. The Base Type should be Default (or Base 10) for most numbers, however Infy can increment all sorts of things, from Dates to Roman Numerals and even Custom Alphabets you define!

Decrementing URLs

Need to decrement instead? Just negate the Interval and use a negative number like -1.

URL List

Already have a list of URLs you want Infy to append? Simply paste them in the input and Infy will append each one as you scroll down. Just make sure the following requirements are met:

  1. The page you are currently on matches the same domain of all the pages in your list. Infy can only append URLs that belong to one domain, and you need to be already on that domain. In other words, your list can't feature URLs from two different domains like google.com and microsoft.com, it's only one or the other.
  2. The URLs must be full URLs and start with a protocol, like https://. This is so Infy doesn't get confused and treat your URLs like they're relative URLs. For example, a full URL should look something like: https://www.google.com/page1, not www.google.com/page1 or google.com/page1.
  3. The URLs must be separated by line breaks.

Infy can also auto-append all the URLs if you toggle on Auto. In this case, you can put any value in the Times input since Infy will automatically use the length of your URL List as the Times.

URL List Options

Click the Links Button to find all the links on the page and click the A-Z button to sort them alphabetically.

Clone this wiki locally