Neo4j pre 1.9 Enterprise changelog
HA: o Configurable chunk size via ha.com_chunk_size for communication chunks between server and client. Also reduced default chunk size to 2M from 16M. A chunk size of at most 16M is still possible to configure. o Made read timeout for the "copy store" request twice that of the normal read timout to be more resilient to slow network or bogged down master
Kernel: o Removed contention around allocating and moving persistence windows so that a thread won't need to await another thread doing this refresh, instead just continue knowing that the windows will be optimally placed in a very near future. o Removed contention around translating a key (as String) into the ID by using copy-on-write map instead of a concurrent hash map. Used in property key as well as relationship type translation. o Fix for Node/Relationship#getPropertyValues() sometimes returning null values from the iterator.
Server: o Upgraded Jackson JAXRS to version 1.9.7 o Keeping the Cypher execution engine between calls makes it possible to re-use execution plans o added User-Agent header tracking to udc to determine rest-driver usage
Cypher: o Removed the /../ literal for regular expressions. Now a normal string literal is used instead o Concatenation handles types better o Changed how the graph-matching module is used, to make it safe for concurrent use o Better type error messages o Renamed iterable to collection o Fixed #795: so that WITH keeps parameters also for empty aggregation results o Fixed #772: Creating nodes/relationships goes before SET, so SET can run on already created elements o Added error when using != instead of <> o Fixed #787: Issue when comparing array properties with literal collections o Fixed #751: Better error message for some type errors o Fixed #818: Problem where properties could only have scalar values (they can be arrays as well) o Fixed #834: Gives relevant syntax exception when properties are entered in MATCH
HA: o Fixed bug where transactions committed through slaves in a mixed version cluster would fail o Master recovery in clusters with only one instance master capable will now always work as expected
Kernel: o Traversal framework backwards compatibility
- Cleaned up any breaking changes
- Removed Expander#addFilter o Kernel JMX bean instance identifier is now reused and can optionally be set explicitly via forced_kernel_id config setting
Server: o Consoles in webadmin can now be disabled.
Cypher:
o Added escape characters to string literals
o Renamed RELATE to CREATE UNIQUE
UDC: o Added edition information (community, advanced, enterprise) o Added a cluster-name hash so that stores originating from the same cluster can be aggregated o Fixed release version and revision o Changed precedence of database configuration over internal udc configuration o Added distribution information (dpkg, rpm, unknown)
HA: o Added transaction push factor that can be configured with amount of slaves to push transactions to. The master will optimistically push each transaction before tx.finish completes to reduce risk of branched data. o Added the ability for rolling upgrades from versions 1.5.3 onwards. o Changed the way master election notification and data gathering works, leading to massively reduced writing of data to the zookeeper service and a subsequent performance increase.
Kernel: o Deprecated AbstractGraphDatabase.transactionRunning() o Changed synchronization of applying transactions to prevent a deadlock scenario o Original cause can be extracted from a transaction RollbackException
Server: o Fixed issue that stopped the server from starting without the UDC-jars.
Cypher: o Fixes problem when graph elements are deleted multiple times in the same query o Fixes #625: Values passed through WITH have the wrong type o Fixes #654: Some paths are returned the wrong way
HA: o Changed master election process, now writing of latest txid for cluster members happens on demand, removing zookeeper as a bottleneck
Kernel: o Configurable amount of logical logs to keep, by for example number of days or size on disk. keep_logical_logs configuration is extended to support values such as: "10 days", "200M size" a.s.o. Regardless of configuration there will always be at least the latest non-empty logical log left. o Reduced synchronization while memory mapping files, leading to increased multithreaded performance
Server: o Added support for multi line Cypher queries in webadmin browser
Lucene-index:
- Removed lucene_writers_cache_size and have lucene_searcher_cache_size decide for both, since it's doesn't make sense to have a writer w/o a searcher and isn't possible to have a searcher w/o its writer.
- Loosened contention regarding getting index searchers for querying.
Cypher: o CREATE and RELATE can now introduce path identifiers o String literals can now contain escape characters o Fixes #600: Double optional with no matching relationships returns too many rows o Fixes #613: Missing dependencies not reported correctly for queries with RELATE/SET/DELETE o Fixes around optional paths
Kernel: o Additions to the traversal framework:
- Bidirectional traversals (with BidirectionalTraversalDescription). AllPaths/AllSimplePaths uses this feature and are now faster due to less relationships being traversed.
- Multiple start nodes (as well as multiple end nodes for bidirectional traversals).
- PathExpander (RelationshipExpander for Paths) which has the full Path passed in instead of just the end node of the path. It can also see and modify a user defined traversal branch state.
- Metadata about the traversal, for the time being: number of relationships traversal and number paths returned.
- Added Path#reverseNodes() and Path#reverseRelationships which gives all nodes/relationships in a path in reverse order (starting from the end node going back to the start node). More relevant in many scenarios as well as allowing for a more efficient implementation.
- Sorting of traversal results, specify with TraversalDescription#sort(Comparable).
- Some measure to reduce memory consumption and garbage generation during traversals.
Graph-algo: o AllPaths/AllSimplePaths uses the new bidirectional traversal feature in the traversal framework. Less relationships now needs to be traversed to find the requested paths. o Added an implementation of the shortest path algorithm with the bidirectional traversal feature.
Cypher: o CREATE now accepts full paths and not only single nodes and relationships o Path patterns can now be used as expressions and as predicates o Fixes bug where RELATE fails to compare array properties o Fixes #573: Arithmetics operations have wrong type o Fixes #567: Parameter maps coming through REST are not parsed correctly o Fixes #563: COUNT(*) does not return 0 for empty result o Fixes #561: RELATE doesn't work as expected with map-parameters
Kernel: o Changed array map to CHM in property index manager, better multithreaded performance Shell: o Added BEGIN TRANSACTION/COMMIT/ROLLBACK o Sessions now live on the server side instead of on the client, which means that not just serializable values can be set there. Paves the way for Cypher making use of it. Server: o keep_logical_logs is now respected if set to false - default is still true Cypher: o Added RELATE o Changed the CREATE syntax, so it looks more like RELATE o Fixes #506: delete head(rels) doesn't work o Fixes #508: length function on array properties doesn't work o Fixes #512: start n=node(*) match n-[r?]->() delete n,r not working o Fixes #514: Extract doesn't work with array properties o Fixes #523: String literals not parsed correctly in CREATE statement o Fixes #526: cypher filter in return clause o Fixes #536: SQRT only returns an integer o Fixes #543: Appending elements to collections High Availability: o Fixed possible but benign NPE when pulling updates o Really old instances when attempting to join but the tx is not in the master's log will now copy the store
Kernel: o Optimized short string and short array encoding algorithms o Fixed problem with SpringTransactionManager during HA master switch Shell: o Shell can now be exited with Ctrl-D Server: o Support for streaming results for http batch operations. o Proper encoding of Index URI key/value pairs. o Fixed Swedish character problem in webadmin string properties. o Webadmin chart labels have nice formatting now. Cypher: o Added the possibility to create nodes from an iterable of maps
- Byte array properties are handled in a more optimized way, increasing performance by a couple of times at least.
- Fix for an issue where update of cache when creating a relationship on a very dense node would take longer and longer time.
- Fix for an issue where a recovery would sometimes fail with it reporting that a start record already had been injected.
- Added "create" shell app for Cypher queries only doing creations. Server: o Added streaming to REST API. All representation implementations have been adapted to produce their data lazily. o Added HTTP logging. Cypher: o Added the possibility to return all graph elements using RETURN * o Added CREATE, SET, DELETE, FOREACH, WITH o Fixes issue #432, #446 High Availability and Backup: o Fix for a rare race condition in initialization code of zoo keeper client which could make a database unable to join its cluster sometimes.
- Moved BatchInserter to a different package.
- Fixed 'Record Not In Use' issue with creating and setting Node properties with the BatchInserter. Cypher: o Added the possibility to use multiple relationship types on patterns o Fixed bug when aggregation is used in ORDER BY and not listed in the RETURN clause Server: o Added scored index results to REST API o Improvements to installation docs for the server o Added auto index management API to REST, fixing #399 o Fixed unicode issues in batch operations API in windows and OS X o Server now disallows creating indexes with empty names. Closes #311 o Attempting to delete a non-existing index now returns 404, closes #349 High Availability and Backup: o Changed so branched data directories end up in a sub-directory. o Changed so informative log messages are not logged as errors. o Fixed issue with relationship types not being created via master if performed on a slave. o Improvements to shell script messages and documentation Management:
- Now contains management information for the new Garbage Collection Resistant Cache UDC: o Modified UDC to run one thread per DB, and to shut down background thread on unload. Closes #279 o UDC now sends the machines MAC adress to Neo Technology, to separate between instances behind firewalls o UDC now sends database registration ID to Neo Technology, if one is available o UDC now sends "tags" to Neo Technology, these contain information about the type of deployment, such as language, web-container, app-container, spring, ejb
- Removed old OSGi support in favor of the new and better one.
- Added possibility to use logback for logging.
- Renamed array cache to GC resistant cache (GCR) and moved it to enterprise.
- Fixed problem with GCR cache that could cause it to calculate incorrect size.
- Fixed problem with closing of messages.log on windows High Availablity and Backup: o Fixed problem with "Transaction already active" getting logged because of lock timeout. o Added GCR cache. o GCR cache arrays are now reused between restarts HA restarts.
- Added lock free atomic array cache that avoids triggering full GCs.
- Added GC monitor that will keep track on how much the JVM blocks application threads.
- Fix for an issue where upgrading indices with an unclean shutdown. High Availablity and Backup: o Fixed issue with message log not getting closed properly.
- Fixed bug in PropertyStore that during recovery would throw exception if the last record was incomplete.
- Fixes to transaction log start position caching which mitigates a performance issue and fixes a caching issue during rotation.
- Fixed a couple of issue around copying incomplete transaction to new log during a log rotation.
- Fixed a property cache poisoning bug and some stale references leaking.
- Fixed a data race issue between threads removing the same property while the node is evicted from cache.
- Fixed an issue where a property record in the logical log was missing its owner.
- Log messages in messages.log are now printed in UTC time zone so that it's the same across servers.
- Fix for an issue where a full rebuild of an id generator (rebuild_idgenerators_fast=false) could result in exception when adding free ids.
- Fix for issue which would close an id generator as clean if a startup failed at the wrong time, which would make the id generator diverge from the store it held ids for and possible truncate that store file the next clean shutdown. Indexing: o Added LowerCaseKeywordAnalyzer and a clear test for it which shows how to configure a case-insensitive exact index. Cypher: o Added literal collections and the IN predicate o Added HEAD, TAIL, LAST and FILTER o Added ! so that missing property can evaluate to false o BREAKING CHANGE: To check whether a property exists on a node, use the HAS(...) function o Arithmetic & string concatenation o An identifier can now be assigned all the nodes/rels in the graph - START a=node(*) o Predicates can be specified inside of a pattern relationship description Server: o Fixed a NPE in guarded database for the result of Index#putIfAbsent. o Added SSL support in the server. o SecurityRule implementations can now use simple wildcards to allow whole sub paths of URIs to be covered by the rule. o Added an "order" parameter to index REST queries to control result ordering. Shell: o dbinfo command now able to print simple values as well as arrays and compound values as JSON output. o A shell server which isn't remotely avaiable doesn't touch RMI, neither does a client which connects to a server within the same JVM. This makes for less RMI garbage when forgetting to call shutdown(). o Fixes problem with eval command in that it couldn't execute a single-line command, which made it impossible to do ./shell -c 'eval ...' from outside UDC: o Added MAC address to pings. High Availablity and Backup: o Added config option for ha.max_concurrent_transactions_on_master. There was already an equivalent for clients, so those two go hand in hand. o Failure to rotate logical log is treated as a kernel panic event (just as expected in this case). o While copying a store from the master the broker is shut down to cancel out any noisy events from it during copy. o Added Broker#restart and implemented in ZooKeeperBroker to handle ZK session expirations. o Made ZK session timeout configurable. o Removed a retry/wait which held a monitor on HighlyAvailableGraphDatabase which could delay a master switch by 'read timeout' seconds. o Restructuring to use DI, configuration interfaces, less setters/getters, removed unecessary lifecycle classes, added proper life cycle support for services and lots and lots of other cleanups. This also removes the need for wrapping an entire HA graphdb in a wrapper to have the entities survive an internal restart. o Now full and incremental backup fetches and writes to disk the latest transaction if not present, allowing for an HA cluster to bootstrap itself.
- New HA request type allows for copy of transaction interval from master to slave, to recover from missing logical log files.
- It is no longer required to run an update puller to trigger a master election when the master goes down abruptly.
- Multiple retries on store copy operations during startup, so that long copies that get interrupted have a chance to complete.
- Backup rotates messages.log with the timestamp of the backup operation
- New HA request type allows for copy of transaction interval from master to slave, to recover from missing logical log files.
- Corrected ZK Expired event handling. Now, instead of restarting the zookeeper client, the whole broker is restarted and newMaster is called. This leads to proper reconnect to the ZK quorum and parsing of the possibly updated cluster state.
- Added support for having one ZK cluster managing many HA clusters (controlled by ha.cluster_name config).
- Fixed issues with update puller being shutdown incorrectly and sometimes having references to stale objects.
- Added support for creating unique entities via Index#putIfAbsent.
- [cypher] BREAKING CHANGE: Changed the syntax for iterable predicated ALL/NONE/ANY/SINGLE to use WHERE instead of a colon
- [cypher] BREAKING CHANGE: ExecutionResult is now a read-once, forward only iterator.
- [server] DEPRECATION: Cypher execution is now part of the core REST API, the cypher plugin is deprecated.
- [com] Now closes connections if either side dies during streaming
- [ha] Added entry about HAProxy in manual.
- [ha] Fixed an issue where pulling updates would fail if no master had been elected. Now it tries to find a new master instead.
- [ha] Exposes information about branched (and stashed) stores through JMX.
- [server] Webadmin data browser now supports cypher queries
- [server] Updated to gremlin 1.4
- [server] Fixed bug in how auto indexes are represented
- [server] Max request execution time can now be limited
- [server] Hypermedia URLs returned by the server now set their host based on the HTTP host header
- [kernel] Lower memory usage of ImpermanentGraphDatabase.
- [kernel] Abstracted stores that stores id/name pairs into AbstractNameStore/AbstractNameRecord and removed lots of duplicated code.
- [kernel] Fixed a race condition in the property cache which cuold poison the cache.
- [kernel] Fixed an issue where a JVM crash in the wrong place would make the next startup rename the current logical log without incrementing the logVersion, making the next log rename fail.
- [kernel] Start records in the logical log aren't written until the transaction starts preparing/committing. This fixes an issue with the logical log not being rotated sometimes.
- [kernel] Added AbstractGraphDatabase#tx() which returns a TransactionBuilder possible of configuring the transaction to be "unforced" meaning that logical logs won't be forced when committing. Means better performance for small non-critical transactions.
- [kernel] Reduced number of proxy object instantiation and node record loading in general.
- [kernel] Added a wrapping graphdb abstraction, which makes instances survive even if the underlying db restarts.
- [cypher] Added allShortestPaths
- [cypher] Added COALESCE
- [cypher] Added column aliasing with AS
- [cypher] Fixed problems with optional graph elements
- Added entry about HAProxy in manual.
- Fixed an issue where pulling updates would fail if no master had been elected. Now it tries to find a new master instead.
- Exposes information about branched (and stashed) stores through JMX.
- Added default remote JMX access configuration and security setup options
- Added protocol version to com protocol to be able to fail fast and predictably if two different version tries to communicate with each other.
- Performance improvements in extraction of logs sent from master to slave in HA.
- Some deadlock scenarios in HA fixed.
- Performance improvements in coordinator communication in HA.
- Fixed some issues regarding monitoring of open channels and what resources they map to and occupy.
- First iteration of the Cypher Query Language included, with a section in the manual.
- Experimental support for batch operations in REST
- The Neo4j Manual now includes some examples, and the section on REST has been extended.
- Server logging has been changed, see the Server Configuration chapter in the manual for further information.
- Neo4j Community now licensed under GPL
- All known Windows problems in Neo4j fixed.
- rewritten Webadmin interface with graph visualization support
- Short string support in kernel for better
- 256 Billion database primitives
- support for Gremlin 0.9 and related Tinkerpop projects
- Better JMX and monitoring support (Neo4j Advanced)
- New backup solution with support
- New look and feel of the webadmin
- New visual data browser
- Updated to Gremlin 0.8
- Added docs for server plugin initialization.
- All manpages are included in the manual.
- Some fixes to be more Windows friendly.
- Added Dijkstra to the list of graph algorithms to be used when finding paths.
- Support for advanced index queries through REST.
- Numerous updates to the Webadmin tool to make it more usable and to fix some visual bugs.
- Removal of the properties service, replaced with a discovery service to allow third-party apps to bind to the server at runtime and discover what services are offered and where.
- Changed the configuration file format to make it simpler and less error prone (but it is not) backwards compatible with earlier versions.
- Separated out the server plugin API for easier development, making only the only development dependency for server a very thin layer and small jar.
- Gremlin updated to 0.7 and trimmed down to bare bones dependencies.
- Webadmin minor improvements in Console, Data browser and monitoring.
- better REST JSON property support in Arrays.
- server started in High Availability mode.
- Full online backup (no need to copy store files before hand) in HA and general support for requests with unlimited size.
- A couple of bugs fixed in lucene index.
- More memory efficient handling of transaction streams (extracting and applying).
- New 'eval' shell command which lets you execute JavaScript on the database.
- Server
- Ability to start the server in HA mode.
- Excludes the neo4j-index component (not used anyhow) to enable HA mode.
- Examples
- Added an example of an ordered traversal.
- New components:
- Neo4j Server, including Web Admin
- Neo4j High Availability
- Neo4j Graph Database Monitoring and Management tools moved to its own component
- Neo4j Index API integrated in the GraphDatabaseService API
- Neo4j Usage Data Collection
- Other changes:
-
Additional services (extensions) for the Neo4j kernel are loaded automatically or can be injected in a running instance.
-
Improved memory footprint and read performance.
-
A weak reference cache is now available for high load low latency workloads.
-
The old index API has been deprecated (but still included and have been updated to use Lucene version 3.0.1).
-
There is a new index API based on Lucene supporting multiple indexes both for nodes and relationships.
-
Path algos can now be performed using the shell.
-
All known bugs have been fixed. For more details see the individual milestone releases below.
- Kernel
- Fixed an issues with PruneEvaluators getting invoked with the start node as argument. PruneEvaluators was never invoked with the start node before the arrival of Evaluator interface.
- Added logging of recovery completion.
- Added TransactionData#isDeleted(Node) and #isDeleted(Relationship) and also made the collection to hold these a HashSet instead of ArrayList. This allows fast lookups for asking if a node/relationship is deleted in a transaction.
- More flexible caching (can instantiate an arbitrary Cache object).
- Shell
- Fixed a problem where ShellServerExtension assumed the shell was enabled during shutdown, which could result in NPE.
- Lucene-index
- More flexible caching (can instantiate an arbitrary Cache object).
- Merged the fast-writes branch. Basically it makes committing changes to lucene indexes faster and more throughput-friendly for multiple threads. The performance improvement for committing lucene transactions shows best for small transactions, but will improve all modifying operations to lucene indexes.
- Fixed bug which could close an IndexSearcher before it was supposed to when there was exactly one item in the result set, two or more threads did this at the same time as there was another thread doing an update... AND the reading threads all called IndexHits#getSingle() on the result.
- HA
- When a new master is chosen, slaves throws away the id allocations they got from the previous master. This fixes a problem where occupied ids where reused.
- Enabled (and fixed problem with) thread on master which checks for dead slave connections and rolls back those transactions, so that their locks are released. This fixes a problem where an HA cluster would seem to freeze up and become unresponsive to write requests.
- Adding Log4j and Netty license information.
- Also closes the executor containing the connections.
- Changed dependency scopes.
- Shuts down databases after verifying them.
- Server
- Bridge OSGi LogService to server Logger. Bundle log messages and stdout pipe through correctly.
- Refactoring of functional tests to remove static dependencies. Introduced a server builder to deal with it instead.
- Can now add performance-tweaking properties to the database hosted by the server. It uses the same neo4j.properties file as the embedded version, but you need to add a property: org.neo4j.server.db.tuning.properties into the neo4j-server.properties file.
- Webadmin: Removed component titles, added save button to data browser, minor change to data browser layout.
- Webadmin: Minor UI updates, added error message that shows up when server connection is lost.
- Webadmin: Moved charts into tabbed box, minor UI updates.
- Fixed an issue with initialization order of things in the server to make sure that RRD is initialized when it is needed.
- Webadmin: Added proper tab-like styling to chart tabs and scale selectors.
- Fixed bug in RRD memory usage sampler, making rounding happen at end of calculations, instead of in the midst of.
- Webadmin: Made 30minutes the default view in charts.
- Webadmin: Added kernel version to dashboard, fixed bug in chart drawing (was drawing when the charts are not visible under certain conditions).
- Webadmin: Disallow self relationships in UI, and show error message explaining this.
- Can now load 3rd party JAX-RS jars from the classpath.
- Updated static assembly to properly include both webadmin statics and documentation. Documentation now tested and ships properly down to neo4j-standalone.
- Webadmin: Made the current node show up by default when adding new relations in webadmin.
- Webadmin: Dashboard shows charts with tabs, and allows switching between charts.
- Added a basic readme with instructions for building, running, and functional testing.
- Updating the way to create temporary files to solve the functional test problem where lots of files stick around.
- re-implemented REST to expose indexing of nodes and relationships (new index API)
- single path algo works in RestfulGraphDatabase now
- fixed duplicate paths for delete node or relationship from index
- Http DELETE requires the proper mediatype to accept
- All indexes must be created to use them.
- Added the ability extend the REST API with server plugins.
- Added back links to the first page in the HTML format.
- updating to Gremlin 0.6
- Updated the component site documentation with the new index API.
-New components
- neo4j-ha, providing high availability
-Important changes, server
- updated to Jersey 1.4
- integrated index is now supported by the shell
- new Evaluator interface improves the traversal API
- support for index hit scores in neo4j-lucene-index
- index caching support added to BatchInserter
- added a server packaging as part of neo4j
- added more configuration options to neo4j-lucene-index
- generating javadocs for all components automatically to components.neo4j.org/{component-name}
- Added ImpermanentGraphDatabase to aid in testing
- Monitoring/management over JMX was moved to the new neo4j-management component.
- Added ability to get the name and configuration of existing integrated indexes.
-New components
- Lucene index, an implementation of the integrated index framework using lucene as backend. Planned to replace current index service.
-Important changes
- Fixed shutdown issue regarding UDC leaving a thread running.
- A new index framework integrated into the kernel API.
- Getting relationships is faster due to less cache lookups.
-New components
- Udc
-Important changes
- Index:
- Lucene version upgraded to 3.0.1
- Shell:
- Apps discoverable with service locator
- Added read-only mode on server
- 'host' and 'config' options
- Graph-algo:
- Find paths of certain length
- Kernel:
- Lower memory footprint
- Higher throughput/concurrency for reads
- Common interface for loading kernel extensions (f.ex. shell)
-New components
- Graph-algo
- Online-backup
-Initial components
- Kernel
- Index
- Remote-graphdb
- Shell
