Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
cd31417
Merge remote-tracking branch 'neo4j/3.5' into 3.5
bradnussbaum Mar 25, 2020
8377817
use versions from /getting-started docs which does work
mneedham Mar 26, 2020
b025b96
wrong gradle file
mneedham Mar 26, 2020
7f52be4
TTL procs
mneedham Mar 30, 2020
e211fb5
apoc spanning tree
mneedham Mar 30, 2020
dfd1d11
missing header
mneedham Mar 30, 2020
bdd8a0f
bump version
mneedham Mar 30, 2020
26636d0
document branching policy (#1465)
sarmbruster Mar 30, 2020
99321e1
apoc.uuid.install yeild batchComputationResult (#1454)
lqst Mar 20, 2020
180c7f9
Start phase-out process for apoc.date.field* to use native dates in N…
JMHReif Mar 25, 2020
71cc35a
Move ttl procs out of date section and under ttl
JMHReif Mar 26, 2020
e6e878c
fix compilation issue
mneedham Mar 30, 2020
636dc27
Merge branch '3.5' of github.com:neo4j-contrib/neo4j-apoc-procedures …
mneedham Mar 30, 2020
9aa88f4
missing imports
mneedham Mar 30, 2020
8f4f7bc
attempt 3
mneedham Mar 30, 2020
d444f41
doesn't seem like inheriting fields works in 3.5
mneedham Mar 31, 2020
8fdadc1
exclude TTLLifeCycle
mneedham Mar 31, 2020
941e33a
fixes #1257: apoc.load.json file roundtrip from apoc.export.json.all …
conker84 Apr 8, 2020
ab6717e
fixes #1458: apoc.load.jdbcUpdate doesn't work with apoc.periodic.ite…
conker84 Apr 8, 2020
a7b20f3
apoc.coll.min and max comparsion operator logic like cypher (#1473)
lqst Apr 8, 2020
c38364c
fix deadlock situation in concurrent executions (#1462)
sarmbruster Apr 10, 2020
c7424c4
Only register with kernel if there are triggers
jakewins Apr 9, 2020
e7ec3ab
fix failing tests: TriggerHandler needs to be config aware
sarmbruster Apr 15, 2020
ef3974f
fix broken test!
mneedham Apr 16, 2020
910a738
attempt to fix flacky test
sarmbruster Apr 16, 2020
33c1a54
TTL procs in own file
mneedham Mar 27, 2020
3775b87
bump version
mneedham Apr 16, 2020
6d9250e
downgrading jackson-databind to a jdk8 compliant version
sarmbruster Apr 16, 2020
92afe05
fix failing tests: use proper uid for docker
sarmbruster Apr 16, 2020
b1b3a14
bump neo4j version
sarmbruster Apr 16, 2020
e0e2499
Merge remote-tracking branch 'neo4j/3.5' into 3.5
bradnussbaum May 1, 2020
72340bd
Merge 3.5
bradnussbaum May 1, 2020
11ac7d6
Reverts params/config changes in Trigger.java
alexiudice May 5, 2020
4168a60
Updates Trigger test classes to use updated parameter syntax.
alexiudice May 5, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion CONTRIBUTING.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,23 @@ If there are multiple commits, and the PR is fine to merge online, use *Squash a

Using the issue id in the summary line will allow us to keep track of commits belonging together.

=== Branching Policy

We do have a branch for each major version of Neo4j, e.g. `3.5`, `4.0` and `4.1`.
As a contributor you're asked to use newest possible branch for you PR.

When your PR is accepted and merged it's the responsibility of the maintainers who merged it to cherry-pick that changes to any newer branch.
Once cherry-picking is done, the maintainers should mark that PR with the `cherry-picked` label.

Please indicate in your PR message text if your PR needs a different behaviour - e.g. if the feature you're fixing has been removed in a newer branch or APIs have changed too much and you have a separate PR for the newer branch.

EXAMPLE:

1. You're fixing a bug being reported for 3.5.x.x.
2. You're choosing the 3.5 branch as base for your PR branch.
3. Once you're done you send a PR.
4. When a maintainer merges that PR, they also take care to cherry-pick it to 4.0 and any more recent branches.

=== Handling pull requests

Be polite.
Expand Down Expand Up @@ -84,4 +101,4 @@ curl $PULL_REQUEST_URL.patch | git am --ignore-whitespace
Make sure to push changes to a PR to the original remote branch.
This will cause the pull request UI in GitHub show and link those commits.

This guideline document is based on the https://github.com/spring-projects/spring-data-build/blob/master/CONTRIBUTING.adoc[spring-data guidelines], thanks @olivergierke.
This guideline document is based on the https://github.com/spring-projects/spring-data-build/blob/master/CONTRIBUTING.adoc[spring-data guidelines], thanks @olivergierke.
5 changes: 3 additions & 2 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ plugins {
id "io.codearte.nexus-staging" version "0.20.0"
id 'maven-publish'
id 'antlr'
id "org.sonarqube" version "2.7"
}
asciidoctorj {
version = '1.6.1'
Expand Down Expand Up @@ -39,7 +40,7 @@ description = """neo4j-apoc-procedures"""

ext {
// NB: due to version.json generation by parsing this file, the next line must not have any if/then/else logic
neo4jVersion = "3.5.16"
neo4jVersion = "3.5.17"
// instead we apply the override logic here
neo4jVersionEffective = project.hasProperty("neo4jVersionOverride") ? project.getProperty("neo4jVersionOverride") : neo4jVersion
testContainersVersion = '1.11.0'
Expand Down Expand Up @@ -157,7 +158,7 @@ dependencies {
testCompile group: 'com.couchbase.client', name: 'java-client', version: '2.7.2'

compileOnly group: 'org.neo4j', name: 'neo4j', version: neo4jVersionEffective
compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.10.3'
compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.9.10.4'
compile group: 'com.opencsv', name: 'opencsv', version: '4.2'
compileOnly group: 'org.ow2.asm', name: 'asm', version: '5.0.2'
compile group: 'com.github.javafaker', name: 'javafaker', version: '0.10'
Expand Down
164 changes: 137 additions & 27 deletions docs/asciidoc/graph-updates/ttl.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,69 +3,179 @@

[abstract]
--
This section describes procedures that can be used to remove nodes from the database once a time limit has been reached.
This section describes procedures that can be used to remove nodes from the database once a time or time limit has been reached.
--

Some nodes are not meant to live forever.
That's why with APOC you can specify a time by when they are removed from the database, by utilizing a schema index and an additional label.
A few convenience procedures help with that.
A few procedures help with that.

ifdef::backend-html5[]
++++
<iframe width="560" height="315" src="https://www.youtube.com/embed/e9aoQ9xOmoU" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
++++
endif::[]
This section includes:

Enable TTL with setting in `neo4j.conf` : `apoc.ttl.enabled=true`
* <<ttl-available-procedures, Available Procedures>>
* <<ttl-config-parameters, Configuration and Parameters>>
//* <<ttl-handson-video, Time-To-Live Hands-On>>
* <<ttl-examples, Examples>>
** <<ttl-expireAt, Expire at Specified Time>>
** <<ttl-expireIn, Expire after Amount of Time>>
* <<ttl-process, Manual Process: How TTL Works>>

There are some convenience procedures to expire nodes.
[[ttl-available-procedures]]
== Available Procedures

You can also do it yourself by running
The table below describes the available procedures:

[separator=¦,opts=header,cols="1,1m,1m,5"]
|===
include::../../../build/generated-documentation/apoc.ttl.csv[]
|===

[[ttl-config-parameters]]
== Configuration and Parameters

For configuration, you will need to enable time-to-live functionality with the following settings in `neo4j.conf`:

.neo4j.conf
[source,properties]
----
apoc.ttl.enabled=true

# Optional: controls the repeat frequency
# apoc.ttl.schedule=5
----

In the available procedures listed above, there are several parameters with specific values.
The table below outlines values and formats for the valid parameters.

[options="header"]
|===
| Parameter | Description | Possible Values | Examples
| `node` | The entity or entities to add the label and property of time-to-live (previous selection statement needed) | Any node or group of nodes fitting desired criteria | `n`, `person`, `group`
| `epochTime` | The datetime value of when the node(s) should expire | Any value in epoch seconds or millisecond format | `1540944000`, `1582209630000`
| `time-unit` | Measurement of units for input value | `ms, s, m, h, d` (long forms: `millis, milliseconds, seconds, minutes, hours, days`) | `milliseconds`, `h`
|===

//[[ttl-handson-video]]
//== Time-To-Live Hands-On
//
//ifdef::backend-html5[]
//++++
//<iframe width="560" height="315" src="https://www.youtube.com/embed/e9aoQ9xOmoU" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
//++++
//endif::[]

[[ttl-examples]]
== Examples: Time-To-Live

This section includes examples showing how to use the time-to-live procedures.
These examples are based on a movies dataset, which can be imported by running the following Cypher query:

include::../export/createExportGraph.adoc[]

The Neo4j Browser visualization below shows the imported graph:

image::play-movies.png[title="Movies Graph Visualization"]

[[ttl-expireAt]]
=== Expire node(s) at specified time

The `apoc.ttl.expireAtInstant` procedure deletes a node or group of nodes after the datetime specified.

To remove a single node or set of nodes, we can use a selection query prior to calling the procedure that defines which nodes we want to apply the time-to-live label and property.
We then call the procedure and pass in the selected node(s), the future datetime at which we want the nodes to be removed, and the specificity of the datetime (seconds, milliseconds, etc).

[source,cypher]
----
SET n:TTL
SET n.ttl = timestamp() + 3600
MATCH (movie:Movie)<-[produced:PRODUCED]-(person:Person)
CALL apoc.ttl.expireAtInstant(person,1585176720,'s')
RETURN movie, produced, person
----

[cols="1m,5"]
.Results
[opts="header"]
|===
| CALL apoc.date.expire.in(node,time,'time-unit') | expire node in given time-delta by setting :TTL label and `ttl` property
| CALL apoc.date.expire(node,time,'time-unit') | expire node at given time by setting :TTL label and `ttl` property
| "movie" | "produced" | "person"
| {"title":"The Matrix","tagline":"Welcome to the Real World","released":1999} | {} | {"name":"Joel Silver","ttl":1585176720000,"born":1952}
|===

Optionally set `apoc.ttl.schedule=5` as repeat frequency.
After the point in time specified (in this case, after `2020-03-25 17:52:00`), the node(s) will be expired and deleted from the graph.
Running the statement below will return no results for our example graph.

[source,cypher]
----
MATCH (movie:Movie)<-[produced:PRODUCED]-(person:Person)
RETURN movie, produced, person
----

== Process
[[ttl-expireIn]]
=== Expire node(s) after specified time period

30s after startup an index is created:
The `apoc.ttl.expireAfterTimeLength` procedure deletes a node or group of nodes after the length of time specified.
//LEFT OFF HERE!
Just as with the similar procedure above, we can use a selection query prior to calling the procedure that defines which nodes we want to apply the time-to-live label and property.
We then call the procedure and pass in the selected node(s), the time delta from current time at which we want the nodes to be removed, and the specificity of the time amount (seconds, milliseconds, etc).

[source,cypher]
----
CREATE INDEX ON :TTL(ttl)
MATCH (movie:Movie)<-[produced:PRODUCED]-(person:Person)
CALL apoc.ttl.expireAfterTimeLength(person,1585176720,'s')
RETURN movie, produced, person
----

At startup a statement is scheduled to run every 60s (or configure in `neo4j.conf` - `apoc.ttl.schedule=120`)
.Results
[opts="header"]
|===
| "movie" | "produced" | "person"
| {"title":"The Matrix","tagline":"Welcome to the Real World","released":1999} | {} | {"name":"Joel Silver","ttl":120000,"born":1952}
|===

After the length of time specified has passed (in this case, after `120 seconds`), the node(s) will be expired and deleted from the graph.
Running the statement below will return no results for our example graph.

[source,cypher]
----
MATCH (t:TTL) where t.ttl < timestamp() WITH t LIMIT 1000 DETACH DELETE t
MATCH (movie:Movie)<-[produced:PRODUCED]-(person:Person)
RETURN movie, produced, person
----

The `ttl` property holds the *time when the node is expired in milliseconds since epoch*.
[[ttl-process]]
== Manual Process: How TTL Works

You can expire your nodes by setting the :TTL label and the ttl property:
You can also do the time-to-live process manually by running the following steps:

* Set the `:TTL` label and `ttl` property on the node(s) you want to expire.

[source,cypher]
----
MATCH (n:Foo) WHERE n.bar SET n:TTL, n.ttl = timestamp() + 10000;
SET n:TTL
SET n.ttl = timestamp() + 3600
----

There is also a procedure that does the same:
The `ttl` property holds the *time when the node is expired in milliseconds since epoch*.

* Create an index on the time-to-live label and property.

[source,cypher]
----
CALL apoc.date.expire(node,time,'time-unit');
CALL apoc.date.expire(n,100,'s');
CREATE INDEX ON :TTL(ttl)
----

When using the procedure, the index is created 30 seconds after startup.

* Remove node(s) that have passed the expiration time or length of time

[source,cypher]
----
MATCH (t:TTL) where t.ttl < timestamp() WITH t LIMIT 1000 DETACH DELETE t
----

When using the procedure, the deletion statement to remove nodes past expiration will run every 60 seconds.
You can also configure the schedule by adding the following setting in `neo4j.conf`:


.neo4j.conf
[source,properties]
----
# Optional: controls the repeat frequency
apoc.ttl.schedule=120
----
25 changes: 25 additions & 0 deletions docs/asciidoc/import/loadjson.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -195,3 +195,28 @@ You will receive the following response:
"weight": 0.02548018842935562
}
----

== Import JSON

If you used the `apoc.export.json.all` to export the graph, we provide the `apoc.import.json` to reimport that data.

[source,cypher]
----
CALL apoc.import.json($file, $config)
----

The `$config` parameter is a map

[opts=header,cols="m,m"]
|===
| name | default | description
| unwindBatchSize | 5000 | the batch size of the unwind
| txBatchSize | 5000 | the batch size of the transacttion
| importIdName | neo4jImportId | the name of the "id" field into the used for the import it refers to the "id" field into the root object of the json.
| nodePropertyMappings | {} | The mapping label/property name/property type for Custom Neo4j types (point date). I.e. { User: { born: 'Point', dateOfBirth: 'Datetime' } }
| relPropertyMappings | {} | The mapping rel type/property name/property type for Custom Neo4j types (point date). I.e. { KNOWS: { since: 'Datetime' } }
|===

N.b. Supported Neo4j types via config mappings are:

Point, Localdate, Localtime, Localdatetime, Duration, offsettime, Zoneddatetime
8 changes: 6 additions & 2 deletions docs/asciidoc/path-finding/config-params.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,12 @@ The procedures support the following config parameters:
| name | type | default | description
| minLevel | Long | -1 | the minimum number of hops in the traversal. Must be 0 or 1 if specified
| maxLevel | Long | -1 | the maximum number of hops in the traversal
| relationshipFilter | String | null | the relationship types and directions to traverse. See <<expand-subgraph-relationship-filters>>.
| labelFilter | String | null | the node labels to traverse. See <<expand-subgraph-label-filters>>.
| relationshipFilter | String | null | the relationship types and directions to traverse.

See <<expand-subgraph-relationship-filters>>.
| labelFilter | String | null | the node labels to traverse.

See <<expand-subgraph-label-filters>>.
| beginSequenceAtStart | Boolean | true | starts matching sequences of node labels and/or relationship types (defined in `relationshipFilter`, `labelFilter`, or `sequences`) one node away from the start node.
| bfs | Boolean | true | use Breadth First Search when traversing. Uses Depth First Search if set to `false`
| filterStartNode | Boolean | false | whether the `labelFilter` and `sequence` apply to the start node of the expansion.
Expand Down
Loading