Skip to content

Commit

Permalink
Fixes neo4j-contrib#1372: apoc.load.html ability to read runtime stru…
Browse files Browse the repository at this point in the history
…cture of the page
  • Loading branch information
vga91 committed May 24, 2022
1 parent 64aacce commit c73b445
Show file tree
Hide file tree
Showing 21 changed files with 913 additions and 159 deletions.
89 changes: 37 additions & 52 deletions docs/asciidoc/modules/ROOT/partials/usage/apoc.load.html.adoc
Expand Up @@ -226,50 +226,39 @@ a|
----
|===

We can also pass an HTML string into the 1st parameter by putting as a config parameter `htmlString: true`, for example:
https://github.com/vga91/neo4j-apoc-procedures/blob/cherry-picks_may_22_4.4/docs/asciidoc/modules/ROOT/partials/usage/apoc.load.html.adoc

[source,cypher]
----
CALL apoc.load.html("<!DOCTYPE html> <html> <body> <p class='firstClass'>My first paragraph.</p> </body> </html>",{metadata:"meta", h2:"h2"}, {htmlString: true});
----
== Runtime js generated html

The jsoup class https://jsoup.org/apidocs/org/jsoup/nodes/Element.html[org.jsoup.nodes.Element]
provides a set of functions that can be used.
Anyway, we can emulate all of them using the appropriate css/jQuery selectors in these ways
(except for the last one, we can substitute the `*` with a tag name to search into it instead of everywhere. Furthermore, by removing the `*` selector will be returned the same result):
If we have a `.html` file with a jQuery script like:

[source,html]
----
<!DOCTYPE html>
<head>
<script type="text/javascript">
$(() => {
var newP = document.createElement("strong");
var textNode = document.createTextNode("This is a new text node");
newP.appendChild(textNode);
document.getElementById("appendStuff").appendChild(newP);
});
</script>
<meta charset="UTF-8"/>
</head>
<body onLoad="loadData()" class="mediawiki ltr sitedir-ltr mw-hide-empty-elt ns-0 ns-subject page-Aap_Kaa_Hak rootpage-Aap_Kaa_Hak skin-vector action-view">
<div id="appendStuff"></div>
</body>
</html>
----

[opts="header"]
|===
| jsoup function | css/jQuery selector | description
| `getElementById(id)` | `#id` | Find an element by ID, including or under this element.
| `getElementsByTag(tag)` | `tag` | Finds elements, including and recursively under this element, with the specified tag name.
| `getElementsByClass(className)` | `.className` | Find elements that have this class, including or under this element.
| `getElementsByAttribute(key)` | `[key]` | Find elements that have a named attribute set.
| `getElementsByAttributeStarting(keyPrefix)` | `*[^keyPrefix]` | Find elements that have an attribute name starting with the supplied prefix. Use data | to find elements that have HTML5 datasets.
| `getElementsByAttributeValue(key,value)` | `*[key=value]` | Find elements that have an attribute with the specific value.
| `getElementsByAttributeValueContaining(key,match)` |`*[key*=match]` | Find elements that have attributes whose value contains the match string.
| `getElementsByAttributeValueEnding(key,valueSuffix)` | `*[class$="test"]` | Find elements that have attributes that end with the value suffix.
| `getElementsByAttributeValueMatching(key,regex)` |`*[id~=content]` | Find elements that have attributes whose values match the supplied regular expression.
| `getElementsByAttributeValueNot(key,value)` |`*:not([key="value"])` | Find elements that either do not have this attribute, or have it with a different value.
| `getElementsByAttributeValueStarting(key,valuePrefix)` |`*[key^=valuePrefix]` | Find elements that have attributes that start with the value prefix.
| `getElementsByIndexEquals(index)` |`*:nth-child(index)` | Find elements whose sibling index is equal to the supplied index.
| `getElementsByIndexGreaterThan(index)` |`*:gt(index)` | Find elements whose sibling index is greater than the supplied index.
| `getElementsByIndexLessThan(index)` |`*:lt(index)` | Find elements whose sibling index is less than the supplied index.
| `getElementsContainingOwnText(searchText)` |`*:containsOwn(searchText)` | Find elements that directly contain the specified string.
| `getElementsContainingText(searchText)` |`*:contains('searchText')` | Find elements that contain the specified string.
| `getElementsMatchingOwnText(regex)` |`*:matches(regex)` | Find elements whose text matches the supplied regular expression.
| `getElementsMatchingText(pattern)` |`*:matchesOwn(pattern)` | Find elements whose text matches the supplied regular expression.
| `getAllElements()` |`*` | Find all elements under document (including self, and children of children).
|===

For example, we can execute:
we can read the generated js through the `browser` config.
Note that to use a browser, you have to install <<selenium-depencencies,this dependencies>>:

[source,cypher]
----
CALL apoc.load.html($url, {nameKey: '#idName'})
CALL apoc.load.html("test.html",{strong: "strong"}, {browser: "FIREFOX"});
----

.Results
[opts="header"]
|===
Expand All @@ -278,31 +267,27 @@ a|
[source,json]
----
{
"h6": [
"strong": [
{
"attributes": {
"id": "idName"
},
"text": "test",
"tagName": "h6"
"tagName": "strong",
"text": "This is a new text node"
}
]
}
----
|===

== Html plain text representation

Using the same syntax and logic as `apoc.load.html`,
we can get a plain text representation of the whole document, using the `apoc.load.htmlPlainText(URL_OR_TEXT, QUERY_MAP, CONFIG_MAP)` procedure, for example:
If we can parse a tag from a slow async call, we can use `wait` config to waiting for 10 second (in this example):

[source,cypher]
----
CALL apoc.load.htmlPlainText($urlOrString, {nameKey: 'body'})
CALL apoc.load.html("test.html",{asyncTag: "#asyncTag"}, {browser: "FIREFOX", wait: 10});
----

or of some elements, with a selector:
[source,cypher]
----
CALL apoc.load.htmlPlainText($urlOrString, {nameKey: 'div'})
----
[[selenium-depencencies]]
== Dependencies

To use the `apoc.load.html` procedures with `browser` config (not `NONE`), you have to add additional dependencies.

This dependency is included in https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/download/{apoc-release}/apoc-selenium-dependencies-{apoc-release}.jar[apoc-selenium-dependencies-{apoc-release}.jar^], which can be downloaded from the https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/tag/{apoc-release}[releases page^].
Once that file is downloaded, it should be placed in the `plugins` directory and the Neo4j Server restarted.
@@ -1,10 +1,19 @@
The procedure support the following config parameters:

.Config parameters
[opts=header]
[opts="header",cols="1m,2m,1m,4"]
|===
| name | type | default | description
| charset | String | "UTF-8" | the character set of the page being scraped
| browser | Enum [NONE, CHROME, FIREFOX] | NONE | If it is set to "CHROME" or "FIREFOX", is used https://www.selenium.dev/documentation/en/webdriver/[Selenium Web Driver] to read the dynamically generated js.
In case it is "NONE" (default), it is not possible to read dynamic contents.
Note that to use the Chrome or Firefox driver, you need to have them installed on your machine and you have to download additional jars into the plugin folder. <<selenium-depencencies, See below>>
| wait | long | 0 | If greater than 0, it waits until it finds at least one element for each of those entered in the query parameter
(up to a maximum of defined seconds, otherwise it continues execution).
Useful to handle elements which can be rendered after the page is loaded (i.e. slow asynchronous calls).
| charset | String | "UTF-8" | the character set of the page being scraped, if `http-equiv` meta-tag is not set.
| headless | boolean | true | Valid with `browser` not equal to `NONE`, allow to run browser in https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md[headless mode],
that is without actually opening the browser UI (recommended).
| acceptInsecureCerts | boolean | true | If true, allow to read html from insecure certificates
| baseUri | String | "" | Base URI used to resolve relative paths
| failSilently | Enum [FALSE, WITH_LOG, WITH_LIST] | FALSE | If the parse fails with one or more elements, using `FALSE` it throws a `RuntimeException`, using `WITH_LOG` a `log.warn` is created for each incorrect item and using `WITH_LIST` an `errorList` key is added to the result with the failed tags.
|htmlString | boolean | true | to use a string instead of an url as 1st parameter
Expand Down
25 changes: 25 additions & 0 deletions extra-dependencies/selenium/build.gradle
@@ -0,0 +1,25 @@
plugins {
id 'com.github.johnrengelman.shadow' version '4.0.3'
}

java {
sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_11
}

archivesBaseName = 'apoc-selenium-dependencies'
description = """APOC Selenium Dependencies"""

jar {
manifest {
attributes 'Implementation-Version': version
}
}

dependencies {
// currently we cannot update to the latest version due to guava minimum version required (31.0.1-jre)
compile group: 'org.seleniumhq.selenium', name: 'selenium-java', version: '3.141.59', {
exclude group: 'com.google.guava', module: 'guava'
}
compile group: 'io.github.bonigarcia', name: 'webdrivermanager', version: '4.4.3'
}
Empty file.
@@ -0,0 +1,6 @@
#Tue Feb 06 14:27:44 CET 2018
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-5.2.1-bin.zip
183 changes: 183 additions & 0 deletions extra-dependencies/selenium/gradlew
@@ -0,0 +1,183 @@
#!/usr/bin/env sh

#
# Copyright 2015 the original author or authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

##############################################################################
##
## Gradle start up script for UN*X
##
##############################################################################

# Attempt to set APP_HOME
# Resolve links: $0 may be a link
PRG="$0"
# Need this for relative symlinks.
while [ -h "$PRG" ] ; do
ls=`ls -ld "$PRG"`
link=`expr "$ls" : '.*-> \(.*\)$'`
if expr "$link" : '/.*' > /dev/null; then
PRG="$link"
else
PRG=`dirname "$PRG"`"/$link"
fi
done
SAVED="`pwd`"
cd "`dirname \"$PRG\"`/" >/dev/null
APP_HOME="`pwd -P`"
cd "$SAVED" >/dev/null

APP_NAME="Gradle"
APP_BASE_NAME=`basename "$0"`

# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'

# Use the maximum available, or set MAX_FD != -1 to use that value.
MAX_FD="maximum"

warn () {
echo "$*"
}

die () {
echo
echo "$*"
echo
exit 1
}

# OS specific support (must be 'true' or 'false').
cygwin=false
msys=false
darwin=false
nonstop=false
case "`uname`" in
CYGWIN* )
cygwin=true
;;
Darwin* )
darwin=true
;;
MINGW* )
msys=true
;;
NONSTOP* )
nonstop=true
;;
esac

CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar

# Determine the Java command to use to start the JVM.
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD="$JAVA_HOME/jre/sh/java"
else
JAVACMD="$JAVA_HOME/bin/java"
fi
if [ ! -x "$JAVACMD" ] ; then
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
else
JAVACMD="java"
which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi

# Increase the maximum file descriptors if we can.
if [ "$cygwin" = "false" -a "$darwin" = "false" -a "$nonstop" = "false" ] ; then
MAX_FD_LIMIT=`ulimit -H -n`
if [ $? -eq 0 ] ; then
if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ] ; then
MAX_FD="$MAX_FD_LIMIT"
fi
ulimit -n $MAX_FD
if [ $? -ne 0 ] ; then
warn "Could not set maximum file descriptor limit: $MAX_FD"
fi
else
warn "Could not query maximum file descriptor limit: $MAX_FD_LIMIT"
fi
fi

# For Darwin, add options to specify how the application appears in the dock
if $darwin; then
GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\" \"-Xdock:icon=$APP_HOME/media/gradle.icns\""
fi

# For Cygwin or MSYS, switch paths to Windows format before running java
if [ "$cygwin" = "true" -o "$msys" = "true" ] ; then
APP_HOME=`cygpath --path --mixed "$APP_HOME"`
CLASSPATH=`cygpath --path --mixed "$CLASSPATH"`
JAVACMD=`cygpath --unix "$JAVACMD"`

# We build the pattern for arguments to be converted via cygpath
ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null`
SEP=""
for dir in $ROOTDIRSRAW ; do
ROOTDIRS="$ROOTDIRS$SEP$dir"
SEP="|"
done
OURCYGPATTERN="(^($ROOTDIRS))"
# Add a user-defined pattern to the cygpath arguments
if [ "$GRADLE_CYGPATTERN" != "" ] ; then
OURCYGPATTERN="$OURCYGPATTERN|($GRADLE_CYGPATTERN)"
fi
# Now convert the arguments - kludge to limit ourselves to /bin/sh
i=0
for arg in "$@" ; do
CHECK=`echo "$arg"|egrep -c "$OURCYGPATTERN" -`
CHECK2=`echo "$arg"|egrep -c "^-"` ### Determine if an option

if [ $CHECK -ne 0 ] && [ $CHECK2 -eq 0 ] ; then ### Added a condition
eval `echo args$i`=`cygpath --path --ignore --mixed "$arg"`
else
eval `echo args$i`="\"$arg\""
fi
i=`expr $i + 1`
done
case $i in
0) set -- ;;
1) set -- "$args0" ;;
2) set -- "$args0" "$args1" ;;
3) set -- "$args0" "$args1" "$args2" ;;
4) set -- "$args0" "$args1" "$args2" "$args3" ;;
5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;;
6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;;
7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;;
8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;;
9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;;
esac
fi

# Escape application args
save () {
for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done
echo " "
}
APP_ARGS=`save "$@"`

# Collect all arguments for the java command, following the shell quoting and substitution rules
eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS"

exec "$JAVACMD" "$@"

0 comments on commit c73b445

Please sign in to comment.