Content extraction pipelines for indexing Project Helix websites
The following components are required before installation:
- A GitHub repository with your
helix-query.yamlconfiguration
If all of this is given, deploy the OpenWhisk action in a package of your choice. For more information on the parameters expected by the action, see the next section.
The following parameters should be passed when invoking the action:
owner: GitHub repository ownerrepo: GitHub repository nameref: GitHub repository reference or branchpath: path to the HTML document to be indexed
Note: the first three parameters also determine the location where the helix-query.yaml configuration file is downloaded from.
In your helix-query.yaml 1, you can define one or more index definitions. A sample index definition looks as follows:
indices:
mysite:
source: html
fetch: https://{ref}--{repo}--{owner}.project-helix.page/{path}
properties:
author:
select: main > div:nth-of-type(3) > p:nth-of-type(1)
value: |
match(el, 'by (.*)')
The select property is a CSS selector that grabs HTML elements out of your document. To verify that a CSS selector entered
is selecting what you expect, you can test it in your browser's Javascript console, e.g. for the author selector shown above,
enter the following expression:
document.querySelectorAll('main > div:nth-of-type(3) > p:nth-of-type(1)');
The value property contains an expression to apply to all HTML elements selected. That expression can contain
a combination of functions and variables:
Returns the HTML content of an element.
Returns the text content of the selected element, and all its descendents.
Returns the value of the attribute with the specified name of an element.
Matches a regular expression containing parentheses to capture items in the passed element.
In the author example above, the actual contents of the <p> element selected might
contain by James Brown, so it would capture everything following by .
Useful for teasers, this selects a range of words out of an HTML element.
Replaces all occurrences of a substring in a text with a replacement.
Parses a timestamp given as string, and returns its value as number of seconds since 1 Jan 1970.
Returns the HTML elements selected by the select property.
Returns the path of the HTML document being indexed.
Returns the value of the HTTP response header with the specified name, at the time the HTML document was fetched.
[1]: The full definition of the helix-query.yaml is available here: https://github.com/adobe/helix-shared/blob/main/docs/indexconfig.md