forked from arangodb/arangodb
-
Notifications
You must be signed in to change notification settings - Fork 0
/
CHANGELOG
15379 lines (10184 loc) 路 602 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
devel
-----
* Improve readability of running and slow queries in web UI by properly left-
aligning the query strings.
* Allow changing collection properties for smart edge collections as well.
Previously, collection property changes for smart edge collections were not
propagated.
* Adjust arangodump integration test to desired behavior, and make sure
arangodump behaves as specified when invoking it with non-existing
collections.
* Fixed BTS-110: Fulltext index with minLength <= 0 not allowed.
* Disallow using V8 dependent functions in SEARCH statement.
* Remove superflous `%>` output in the UI modal dialog in case the JSON editor
was embedded.
* Fixed a misleading error message in AQL.
* Fix undistribute-remove-after-enum-coll which would allow calculations
to be pushed to a DBServer which are not allowed to run there.
* Fixed issue ES-609: "Transaction already in use" error when running
transaction.
Added option `--transaction.streaming-lock-timeout` to control the timeout in
seconds in case of parallel access to a streaming transaction.
* Returned `AQL_WARNING()` to emit warnings from UDFs.
* Fixed internal issue BTS-107, offset over the mainquery passed through a
subquery which has modification access to shards could yield incorrect
results, if shards are large enough and skipping was large enough, both to
overflow the batch-size in AQL for each shard individually.
The Following Query would be effected:
FOR x IN 1..100
LET sub = (
FOR y IN Collection
UPDATE y WITH {updated: true} IN Collection
RETURN new
)
LIMIT 5, 10
REUTRN {x,sub}
If a shard in Collection, has enough entries to fill a batch, the second shard
could run into the issue that it actually does not skip the first 5 main
queries, but reports their results in addition. This has the negative side
effect that merging the Subqueries back together was off.
* Correct some log entries.
* Allow removal of existing schemas by saving a schema of either `null` or `{}`
(empty object). Using an empty string as schema will produce an error in the
web interface and will not remove the schema.
The change also adjusts the behavior for the SCHEMA_VALIDATE AQL function in
case the first parameter was no document/object. In this case, the function
will now return null and register a warning in the query, so the user can
handle it.
* Internal issue BTS-71: Added a precondition to prevent creating a collection
with an invalid `distributeShardsLike` property.
* Internal issue BTS-71: In a cluster, for collections in creation, suspend
supervision jobs concerning replication factor until creation is completed.
Previously, it could cause collection creation to fail (e.g. when a server
failed during creation), even when it didn't have to.
* Internal issue BTS-71: Fixed error handling regarding communication with the
agency. This could in a specific case cause collection creation in a cluster
report success when it failed.
* Fixed internal issue #725: Added analyzers revision for _system database in
queries.
* Allow restoring collections from v3.3.0 with their all-numeric collection
GUID values, by creating a new, unambiguous collection GUID for them.
v3.3.0 had a bug because it created all-numeric GUID values, which can be
confused with numeric collection ids in lookups. v3.3.1 already changed the
GUID routine to produce something non-numeric already, but collections
created with v3.3.0 can still have an ambiguous GUID. This fix adjusts
the restore routine to drop such GUID values, so it only changes something
if v3.3.0 collections are dumped, dropped on the server and then restored
with the flawed GUIDs.
* Fixed bug in IResearchViewExecutor that lead to only up to 1000 rows being
produced.
* Changing the current users profile icon in the Web UI now renders the new
icon directly without the need of a full UI browser reload.
* The Web UI's replication view is now checking the replication state
automatically without the need of a manual reload.
* Fixed an error scenario where a call could miss count skip.
It was triggered in the case of Gather in Cluster, if we skipped over a full
shard, and the shard did actually skip, but there are more documents to skip
on another shard.
* Fixed hotbackup agency lock cleanup procedure.
* Only advance shard version after follower is reported in sync in agency.
* Fixed cluster behavior with HotBackup and non-existing backups on DB-Servers.
* Fixed that, when performing a graph AQL query while a (graceful) failover for
the leader of the system collections is in progress, ArangoDB would report a
"Graph not found error".
The real error, namely that an agency transaction failed, was swallowed in
the graph lookup code due to a wrong error code being used from Fuerte.
We now generate a more appropriate 503 - Service Unavailable error.
* added option `--log.use-json-format` to switch log output to JSON format.
Each log message then produces a seperate line with JSON-encoded log data,
which can be consumed by applications.
* added option `--log.process` to toggle the logging of the process id
(pid) in log messages. Logging the process id is useless when running
arangod in Docker containers, as the pid will always be 1. So one may
as well turn it off in these contexts.
* added option `--log.in-memory` to toggle storing log messages in memory,
from which they can be consumed via the `/_admin/log` and by the web UI. By
default, this option is turned on, so log messages are consumable via API
and the web UI. Turning this option off will disable that functionality and
save a tiny bit of memory for the in-memory log buffers.
* Allow for faster cluster shutdown. This should reduce the number of shutdown
hangers in case the agents are stopped already and then coordinators or
DB-Servers are shut down.
* Fixed issue ES-598. Web UI now shows correct permissions in case wildcard
database level and wildcard collection level permissions are both set.
* Fixed non-deterministic test failure in Pregel WCC test.
* Fixed unintentional connection re-use for cluster-internal communications.
* Fixed problem with newer replication protocol and ArangoSearch which could
lead to server crashes during normal operation.
* Fixed bad behavior that led to unnecessary additional revision tree rebuilding
on server restart.
* Allow AQL queries on DB-Servers again. This is not an official supported
feature, but is sometimes used for debugging. Previous changes made it
impossible to run a query on a local shard.
* Fix restoring old arangodumps from ArangoDB 3.3 and before, which had index
information stored in slightly different places in the dump.
* Fix invalid calls to `AgencyCommResult::slice()` method, which must only be
made in case of an agency result was retrieved successfully. In case the call
to the agency was not successful, `slice()` must not be called on it. This
change makes sure this invariant holds true.
* Fix internal test helper function `removeCost` to really remove costs.
* Fix potential AQL query shutdown hanger.
* Use smarter default value preset in web UI for replication factor in case
there are constraints established for the replication factor by the
startup options `--cluster.min-replication-factor` and
`--cluster.max-replication-factor`.
* Modified the exception handling in the RestHandler. Asynchronous communication
could lead to less detailed failure information.
* Added permissions check before trying to read data from `_analyzers`
collection.
If these permissions are not there, no load is performed (user can not use
analyzers from database anyway).
* Updated arangosync to 0.7.6.
* Make the reboot tracker catch a failed server and permanently removed
servers. This allows other servers in the cluster to move on more quickly,
when a server fails and does not immediately come back.
* Added WCC pregel algorithm for weakly connected components.
* Display "geoJson" value for geo indexes in index overview in web UI.
* Allow specifiying arbitrary vertex attributes when creating a vertex via the
graph viewer in the web UI. This is necessary for collections that have a
schema and that require new vertices to have certain attributes/values.
* Pass through AQL query errors to UI graph viewer without turning them into
"Internal Server Error"s.
* Augment nodes view in web UI with node id and full datetime of startup.
* Fixed "Create Debug Package" in web UI in cluster mode.
* Pick up selected "write concern" value entered in web UI when creating a
collection in the cluster. Previously, the "write concern" value was not
picked up correctly and collections were created with the default write
concern.
* Fixed the creation of edges in the web UI, which did not work due to a
JavaScript error.
* Fixed velocypack validator for proper check of keys.
* Handle missing "shardingStrategy" in Plan agency entries for collections.
"shardingStrategy" entries can be missing for collections created with
ArangoDB clusters in version 3.3 or earlier. A missing "shardingStrategy"
entry can cause issues for certain queries that need to check the shard
distribution on the DB-Servers.
This change transparently handles missing "shardingStrategy" entries.
* Taken collection dropping from fast track in maintenance. This avoids
blocking fast track maintenance threads when a shard cannot immediately
be dropped because of some pending lock.
* Updated ArangoDB Starter to 0.14.15.
* Fixed a spuriously occuring shutdown deadlock between AgencyCallbackRegistry
and AgencyCache.
* Fixed `"Plan" is not an object in agency` error messages when restarting
DB-Servers that contained ArangoSearch Views with links.
* Fixed misordering of skipped number of rows report. Only triggered if you do
a modification on a subquery nesting level 2 or more, e.g.:
```
LET s1 = (
LET s2 = (
INSERT {} INTO collection
)
COLLECT WITH COUNT INTO c
RETURN c
)
```
Here the c would be off, or a count on main query would be off.
* Fix crash in execution of non-spliced subqueries if remainder of subquery is
skipped.
* Added missing mutex to ConnectionPool::cancelConnections().
* Foxx API now respects "idiomatic" flag being explicitly set to false.
* Fixed crash in execution of non-spliced subqueries if remainder of subquery is
skipped.
* Made modification subqueries non-passthrough. The passthrough logic only works
if exactly as many output rows are produced as input rows are injected.
If the subquery with modification is skipped however this API is violated, we
perform the subquery with the modification, but then discard the input if
skipped).
* Optimized Web UI styling for ArangoSearch Views when in read-only mode.
* Fixed a bug in the internal `recalculateCount()` method for adjusting
collection counts. The bug could have led to count adjustments being
applied with the wrong sign.
* Removed external file loading in a used 3rd party library (Web UI).
* More substantial metrics in agency.
* arangodump and arangorestore will now fail when using the `--collection`
option and none of the specified collections actually exist in the database
(on dump) or in the dump to restore (on restore). In case some of the specified
collections exist, arangodump/restore will issue warnings about the invalid
collections, but will continue to work for the valid collections.
* Improved network send request for more robustness.
* Added multiple RocksDB configuration options to arangod:
* `--rocksdb.cache-index-and-filter-blocks` to make the RocksDB block cache quota
also include RocksDB memtable sizes
* `--rocksdb.cache-index-and-filter-blocks-with-high-priority` to use cache index
and filter blocks with high priority making index and filter blocks be less
likely to be evicted than data blocks
* `--rocksdb.pin-l0-filter-and-index-blocks-in-cache` make filter and index blocks
be pinned and only evicted from cache when the table reader is freed
* `--rocksdb.pin-top-level-index-and-filter` make the top-level index of partitioned
filter and index blocks pinned and only be evicted from cache when the table reader
is freed
* Don't move potentially expensive AQL function calls into loops in the
`remove-unnecessary-calculations-rule`.
For example, in the query
LET x = NOOPT(1..100)
LET values = SORTED(x)
FOR j IN 1..100
FILTER j IN values
RETURN j
there is only one use of the `values` variable. So the optimizer can remove
that variable and replace the filter condition with `FILTER j IN SORTED(x)`.
However, that would move the potentially expensive function call into the
inner loop, which could be a pessimization.
Now the optimizer will not move the calculation of values into the loop when
it merges calculations in the `remove-unnecessary-calculations` optimizer rule.
* Fixed modification executors inside of a subquery, where the subquery decided to
fetch all rows from upstream first and the amount of rows is higher then the
batch size.
* Fixed reporting of skipped number of documents if we have a LIMIT x, 0
right after the modification.
* Added exceptions catching in agency callbacks.
* Fixed issue #11104 case with complex condition on second branch of OR
operator.
* Fixed bad behavior when dropping followers. A follower should be dropped
immediately when it is officially FAILED, not only after a longish timeout.
* Fixed a bug in CollectionNameResolver which could lead to an extended
busy spin on a core when a collection was dropped, but documents of it
still remained in the WAL.
* Fixed return value of fuerte::H1Connection in case of timeout.
* Usability improvements for arangodump and arangorestore:
- when a dump is restricted to one or multiple collections using the
`--collection` option of arangodump, warnings are issued for all specified
collections that are not present in the database. Previously there were
no warnings, so missing a collection by misspelling its name could easily
go unnoticed.
- when a restore is restricted to one or multiple collections using the
`--collection` option of arangorestore, warnings are issued for all specified
collections that are not present in the dump. Previously there were
no warnings, so missing a collection by misspelling its name could easily
go unnoticed.
- when a dump was taken using the `--overwrite` option, there was no check
that validated whether the encryption mode used in the existing dump
directory was the same as the requested encryption mode. This could have
led to dump directories with both encrypted and unencrypted files. This
was only the case when using `--overwrite true`, which is not the default.
- when a restore was performed using the `--encryption.keyfile` option,
there was no check whether the to-be-restored files were actually encrypted.
Now this check is enforced and arangorestore will bail out with an error
if the requested encryption mode for restore is not the same as for the
stored dump files.
* Fixed traversal issue: If you have a traversal with different minDepth and maxDepth
values and filter on path elements that are larger then minDepth, in a way that a
shorter path would match the condition, the shorter paths would in some cases
not be returned, even if they are valid. e.g.
FOR v, e, p IN 1..3 OUTBOUND @start Graph "myGraph"
FILTER p.vertices[2].label != "foo"
RETURN p
In the above query, a path of length 1 would be valid. There p.vertices[2]
does not exist => Evaluated to `null`. `null`.label is again evaluated to `null`
=> `null != "foo"` is true, so the path is valid.
* Fixed traversal issue: If you have a filter on the path that is based on a
variable value, which could not be deduced as constant during runtime, in
a sharded GeneralGraph the filter was not applied correctly.
SmartGraphs and SingleServer traversals are not effected by this issue.
Also OneShard traversals are not effected.
* Added `securityScheme`, `securityScope` and `security` methods to Foxx
endpoints to allow defining Swagger security schemes.
* Foxx routes now always have a Swagger `operationId`. If the route is unnamed,
a distinct operationId will be generated based on the HTTP method and URL.
* Fixed, if you have a collection access within a Subquery, where the main query is fully skipped
and the "splice-subqueries" rule is active. The information of the skip was not transported
correctly. This could cause incorrect counting reports.
If splice-subqueries are disabled, or the main-query is only partly skipped, everything worked
as expected.
* Expanded -std=c++17 flag to all compilers.
* Fixed issue ES-600: log out of LDAP automatically.
Option `--ldap.allow-offline` was added with `false` value by default to let
server use cached offline credentials in case of LDAP server communication was
lost and ArangoDB can't prolong cache online.
* Include all requests to the web interface at endpoint `/_admin/aardvark` in
a node's requests statistics. Previously, requests going to `/_admin/aardvark`
were excluded from the request statistics if and only if the requested
database was the `_system` database.
* Fixed an issue with truncate of a collection after a dbserver was restarted
very quickly. This could block arangosync from making progress because the
_jobs collection could no longer be truncated.
* Web UI is now using pre-compiled templates.
* Fixed issue #11652: Error: AQL: missing variable #2 (1) for node #6
(FilterNode) while planning registers.
* Fixed internal issue #650: Added analyzer cache update before ArangoSearch
view creation/update.
* Update OpenLDAP to 2.4.50.
* Fixed issue ES-554: failed to build filter while querying ArangoSearch view.
This change provides global fix for ArangoSearch analyzers distribution and
management within a cluster.
* Fixed issue ES-545: failing modification operations led to random crashes with
"Index out of bound" error message.
* Fix edge cases in RocksDB primary index range lookups for operators >= and <.
* Fixed issue #11525: Address security vulnerability by updating Swagger-UI
dependency (upgraded Swagger UI to 3.25.1).
* Updated arangosync to 0.7.5.
* Fixed ability to edit graph edge in Graph Viewer of web UI.
* Fixed issue #10371: For k-shortest-paths queries on certain graphs a condition
to abort search was too eager and hence missed the shortest path between two
vertices and returned a longer one.
* Added feature: Disjoint SmartGraphs
SmartGraphs have been extended to a new subtype, called **Disjoint SmartGraphs**.
A Disjoint SmartGraph prohibits edges between different SmartGraph components.
In case the graph schema can be represented without the need of connected
SmartGraph components, a Disjoint SmartGraph should be used as this knowledge
can be used by the internal optimizer to gain even better query traversal
execution times.
* Fixed issue ES-605: arangodump broken when using encryption.
* Fixed a lockup in dropCollection due to a mutex being held for too long.
* Add an optimizer rule that enables execution of certain subqueries on a
DB Server. For this optimization to work, the subquery must contain exactly
one DISTRIBUTE/GATHER pair and only access at most one collection.
This proves particularly useful for traversals, shortest path, and k-shortest
paths queries on Disjoint SmartGraphs where the entire traversal is executed
on the DB Server without involvement of a coordinator.
* ClusterInfo does its own updating of plan / current caches.
* Properly forward requests based on the transaction header.
* Fix a bug in Agency poll, which would sometimes return a wrong firstIndex.
* Made several improvements to AQL's internal register planning. This will
improve query performance in several cases; mainly of large queries with many
variables, and of spliced subqueries. This also fixes the github issues #10853
and #11358 which reported performance degradation caused by subquery splicing.
* Fixed issue #11590: Querying for document by _key returning only a single
seemingly random property on entity ("old", in this case).
This fixes single-key document lookups in the cluster for simple by-key
AQL queries, such as `FOR doc IN collection FILTER doc._key == @key RETURN
doc` in case the document has either an "old" or a "new" attribute.
* Restored behavior of Foxx API documentation being expanded to show all routes
rather than collapsing all sections by default.
* Add optimization for subqueries for which only the number of results
matters. The optimization will be triggered for read-only subqueries that
use a full collection scan or an index scan, without any additional filtering
(early pruning or document post-filtering) and without LIMIT.
It will help in the following situation:
FOR doc IN ...
LET count = COUNT(
FOR sub IN subCollection
FILTER sub._from == doc._id
RETURN sub
)
...
The restrictions are that the subquery result must only be used with the
COUNT/LENGTH function and not for anything else. The subquery itself must
be read-only (no data-modification subquery), not use nested FOR loops nor
LIMIT, nor a FILTER condition or calculation that requires accessing the
document data. Accessing index data is supported for filtering, but not
for further calculations.
If the optimization is triggered, it will show up in the query execution
plan under the name `optimize-count`.
* Integrated libiresearch log topic properly into ArangoDB logging system.
* Fix a bad deadlock because transaction cleanup was pushed with too low
priority.
* Allow specifying graph names as unquoted string in an AQL graph traversal
query, e.g. `FOR ... IN ... GRAPH abc`. Previously, the graph name had
to be a bind parameter or a string enclosed in quotes.
* loadPlan and loadCurrent have been fixed to not miss out on increments.
* ClusterInfo and all data definition changes use the local agency cache.
* Coordinators and DB servers cache the agency's read key value store.
* Agency offers the new poll API to subscribe to the Raft log stream.
* Added option `--rocksdb.edge-cache` to toggle in-memory caching for
edges. The option is turned on by default. This normally helps with
performance in read-only and read-mostly workloads.
* Fixed a bug in Maintenance which could prevent collection creation from
working (made CreateCollection action idempotent).
* Fix potential undefined behavior in some operations issued to the REST
handler at `/_api/collection` in cluster mode.
* `--cluster.agency-prefix` marked as obsolete. Did never work and is not supported.
* Removed old AgencyComm.
* Improve continuation behavior of AQL queries. We post the continuation handler
on lane CLUSTER_AQL instead of CLIENT_AQL. This leads to MEDIUM prio instead
of LOW.
* Fixed a sleeping barber in fuerte. Added TLA+ model to prove that there
is not another one hiding somewhere.
* Fix spurious bugs in `resilience_move` tests due to replication context of
to-be-dropped collections lingering around until timeout.
* Fixed issue 11243: `db._isSystem()` returns true for a non-system DB.
* Fix REST API `/_admin/cluster/removeServer`.
This also fixes the removal of servers via the web interface.
* Updated OpenSSL to 1.1.1g.
* Lower the default values for `--rocksdb.block-cache-size` and
`--rocksdb.total-write-buffer-size` for systems with less than 1 GB of RAM.
For these systems the default value were presumably too high and needed manual
adjustment.
* Allow to override the detected number of available CPU cores via an environment
variable ARANGODB_OVERRIDE_DETECTED_NUMBER_OF_CORES.
* Fix a bug in the optimizer which handled combinations of SmartJoins with
query components that had to run on a coordinator wrong.
* Remove the scatter-arangosearch-view-in-cluster optimizer rule; it was
integrated into scatter-in-cluster.
* Issue #11447: expose extra RocksDB options `--rocksdb.target-file-size-base`
and `--rocksdb.target-file-size-multiplier` to make them configurable.
* Allow STARTS_WITH function to process several prefixes with min match count
parameter.
* Added collection property changes into auditing.
* Fix creation of example graphs when `--cluster.min-replication-factor` is set
and no replication factor was specified for a graph. In this case, the new
graph will use the configured minimum replication factor.
* Show some index details in index overview of the web interface, e.g. the
`minLength` attribute for fulltext indexes or the `expireAfter` attribute for
TTL indexes.
* Added startup option `--server.validate-utf8-strings`.
This option controls whether strings in incoming JSON or VPack requests are
validated for UTF-8 conformity. If set to `true` (which is the default value),
all strings are validated, and requests containing invalid UTF-8 data will be
rejected. Setting the option to `false` will turn off the UTF-8 string
validation for incoming requests. This mode should only be used for
deployments that are known to already contain invalid UTF-8 data and to keep
them operational until the wrong string encoding is fixed in the data.
* Fixed a bug which occurred if a DB-Server was shut down exactly
when it was supposed to resign from its leadership for a shard.
* Improve continuation behavior of AQL queries. We post the continuation
handler on lane CLUSTER_AQL instead of CLIENT_AQL. This leads to MEDIUM
prio instead of LOW.
* When relaying requests to other coordinators in a load-balanced setup, don't
forward the "http/1.1" HTTP header from the remote response. Forwarding that
header in the same way as any other header is not required, as each response
will store and set its status separately anyway. Adding it to the headers
again will only cause confusion and some client applications to choke.
* Fixed issue #11104: `ANALYZER` function doesn't correctly process `!= OR`
queries.
* Obsoleted startup option `--database.maximal-journal-size`. This option was
useful for the MMFiles storage engine only, but did not have an effect with
the RocksDB engine. Using this startup option is not an error, but has
no effect anymore.
* Added `JACCARD` AQL function.
* storedValues property is removed from ArangoSearch link properties output.
* Added primarySortCompression property to ArangoSearch Views.
* Added compression property to ArangoSearch View storedValues.
* Added overwrite mode "ignore" for document inserts. This mode allows
ignoring primary key conflicts on insert when the target document already
exists.
The following overwrite modes now exist:
* "ignore": if a document with the specified `_key` value exists already,
nothing will be done and no write operation will be carried out. The
insert operation will return success in this case. This mode does not
support returning the old document version using the `returnOld`
attribute. `returnNew` will only set the `new` attribute in the response
if a new document was inserted.
* "replace": if a document with the specified `_key` value exists already,
it will be overwritten with the specified document value. This mode will
also be used when no overwrite mode is specified but the `overwrite`
flag is set to `true`.
* "update": if a document with the specified `_key` value exists already,
it will be patched (partially updated) with the specified document value.
* "conflict": if a document with the specified `_key` value exists already,
return a unique constraint violation error so that the insert operation
fails. This is also the default behavior in case the overwrite mode is
not set, and the *overwrite* flag is *false* or not set either.
The overwrite mode "ignore" can also be used from AQL INSERT operations
by specifying it in the INSERT's `OPTIONS`, e.g.
INSERT { _key: ..., .... } INTO collection OPTIONS { overwriteMode: "ignore" }
Again, when the overwrite mode "ignore" is used from AQL, it does not
support returning the old document version. Using "RETURN OLD" in
an INSERT operation that uses the "ignore" overwrite mode will trigger
a parse error, as there will be no old version to return. "RETURN NEW"
will only return the document in case it was inserted. In case the
document already existed, "RETURN NEW" will return "null".
The main use case of inserting documents with overwrite mode "ignore" is
to make sure that certain documents exist in the cheapest possible way.
In case the target document already exists, the "ignore" mode is most
efficient, as it will not retrieve the existing document from storage and
not write any updates to it.
* Added feature: SatelliteGraphs
SatelliteGraphs are a new type of graph, added in addition to the existing
ones, General Graphs and SmartGraphs.
When doing joins involving graph traversals, shortest path or k-shortest paths
computation in an ArangoDB cluster, data has to be exchanged between different
servers. In particular graph traversals are usually executed on a Coordinator,
because they need global information. This results in a lot of network traffic
and potentially slow query execution.
SatelliteGraphs are the natural extension of the concept of
SatelliteCollections to graphs. All of the usual benefits and caveats apply.
SatelliteGraphs are synchronously replicated to all DB-Servers that are part of
a cluster, which enables DB-Servers to execute graph traversals locally. This
includes (k-)shortest path(s) computation and possibly joins with traversals
and greatly improves performance for such queries.
* Removed deprecated MMFiles storage engine and also the `arango-dfdb`
(datafile debugger) executable that could be used to validate MMFiles
datafiles.
This change also obsoletes all MMFiles-specific startup options in the
`--wal.*` section. Using these startup options is not an error, but has
no effect anymore.
* Fixed a bug in the agency supervision, which ignored the `failoverCandidates`
field.
* Fixed a bug in the agency supervision, which could declare an already FAILED
DB-Server temporarily as GOOD again after an agency leader change.
* Added `INTERLEAVE` AQL function.
* Upgraded bundled RocksDB library to version 6.8.0.
* Added AQL function `IN_RANGE`.
* Added startup option `--ssl.prefer-http1-in-alpn` to optionally let the
server prefer HTTP/1.1 over HTTP/2 in ALPN protocol negotiations.
* Compilation issues with wrong cv-qualifiers and unnecessary
temporary copying.
* Add DTRACE points to track a request through the infrastructure.
* Fixed issue #11275: indexes backward compatibility broken in 3.5+.
* Updated snowball dependency to the latest version.
More stemmers are available. Built-in analyzer list is unchanged.
* Reactive REST API endpoint at `/_admin/auth/reload`, as it is called by DC2DC.
* Fix an endless loop in FollowerInfo::persistInAgency which could trigger
a hanger if a collection was dropped at the wrong time.
* Updated LZ4 dependency to version 1.9.2.
* Fix cluster representation of the collection figures for RocksDB.
* Changed behavior for creating new collections in OneShard databases (i.e.
databases with "sharding" attribute set to "single"):
Previously it was allowed to override "distributeShardsLike" and
"numberOfShards" for each new collection. The default values were "_graphs"
and 1 and were not modified if the user did not alter them, but it was still
possible to alter them.
This is now (silently) ignored. Any attempt to set any value for
"distributeShardsLike" or "numberOfShards" for new collections in a OneShard
database will silently be ignored. The collection will automatically be
sharded like the sharding prototype and will have a single shard.
The behavior has been adjusted to match the behavior when creating collections
in a cluster started with `--cluster.force-one-shard true` option. Here any
user-supplied values for "distributeShardsLike" or "numberOfShards" were
always ignored.
Now the behavior is identical for OneShard databases and databases in cluster
started with `--cluster.force-one-shard true`.
The web interface now also hides the "Distribute shards like" settings in this
case, and makes the "Number of shards" input box read-only.
* Fix premature access to temporary path before a user-specified path was
read from the config options.
* Rebuild UI and update swagger.
* Added ability to store values in ArangoSearch views.
* Added LIKE operator/function support to SEARCH clause.
* Added NGRAM_SIMILARITY and NGRAM_POSITIONAL_SIMILARITY functions for
calculating Ngram similarity.
* Added ngram fuzzy search. Supported by NGRAM_MATCH filter function for SEARCH
clause.
* Allow to override the detected total amount of memory via an environment
variable ARANGODB_OVERRIDE_DETECTED_TOTAL_MEMORY.
* `splice-subqueries` optimization is not limited by any type of operation within the
subquery any more. It can now be applied on every subquery and will be by default.
However they may be a performance impact on some queries where splice-subqueries
are not as performant as non-spliced subqueries. This is due to internal memory
management right now and will be addressed in future versions. Spliced subqueries
can be less performant if the query around the subquery is complex and requires
lots of variables, or variables with large content, but the subquery itself
does not require a lot of variables and produces many intermediate results
s.t. good batching within the query does not pay off against memory overhead.
* Supervision to clean up zombie servers after 24h, if no
responsibility for shards.
* Fix SORT RAND() LIMIT 1 optimization for RocksDB when only a projection of the
attributes was used. When a projection was used and that projection was
covered by an index (e.g. `_key` via the primary index), then the access
pattern was transformed from random order collection seek to an index access,
which always resulted in the same index entry to be returned and not a random
one.
* Mark server startup options `--foxx.*`, `--frontend.*` and `--javascript.*`
as single server and Coordinator only for documentation (`--dump-options`).
* Supervision hot backup and supervision maintenance modes ttl fix.
* Fix a bug that leads to graph traversals yielding empty output when none of the
output variables (vertex, edge, path) are used. This is relevant when a query
is only interested in a COUNT of the outputs, for example.
* MoveShard to check, if target is in sync follower before promotion
to leader.
* Agency ttl bug fix
* Added the SNI feature for TLS. This means that one can configure multiple
server keys and certificate chains and the system dynamically uses
the right one depending on the value of the TLS servername extension.
This allows to use different TLS setups for the same server which is
reachable behind different DNS names, for example (Enterprise Edition only).
* Do not create a reboot tracker for empty serverId ubin sync repl.
* Fix the usage of the AQL functions `CALL` and `APPLY` for calling user-defined
AQL functions when invoking an AQL query from the arangosh or a client application.
Previously, invoking an AQL query and using the `CALL` or `APPLY` AQL functions
to call user-defined AQL function caused undefined behavior.
* Improved graph traversal performance via some internal code refactoring:
- Traversal cursors are reused instead of recreated from scratch, if possible.
This can save lots of calls to the memory management subsystem.
- Unnecessary checks have been removed from the cursors, by ensuring some
invariants.
- Each vertex lookup needs to perform slightly less work.
The traversal speedups observed by these changes alone were around 8 to 10% for
single-server traversals and traversals in OneShard setups. Cluster traversals
will also benefit from these changes, but to a lesser extent. This is because the
network roundtrips have a higher share of the total query execution times there.
* Traversal performance can also be improved by not fetching the visited vertices
from the storage engine in case the traversal query does not refer to them.
For example, in the query
FOR v, e, p IN 1..3 OUTBOUND 'collection/startVertex' edges
RETURN e
the vertex variable (`v`) is never accessed, making it unnecessary to fetch the
vertices from storage. If this optimization is applied, the traversal node will be
marked with `/* vertex optimized away */` in the query's execution plan output.
* The existing optimizer rule "move-calculations-down" is now able to also move
unrelated subqueries beyond SORT and LIMIT instructions, which can help avoid the
execution of subqueries for which the results are later discarded.
For example, in the query
FOR doc IN collection1
LET sub1 = FIRST(FOR sub IN collection2 FILTER sub.ref == doc._key RETURN sub)
LET sub2 = FIRST(FOR sub IN collection3 FILTER sub.ref == doc._key RETURN sub)
SORT sub1
LIMIT 10
RETURN { doc, sub1, sub2 }
the execution of the `sub2` subquery can be delayed to after the SORT and LIMIT,
turning it into
FOR doc IN collection1
LET sub1 = FIRST(FOR sub IN collection2 FILTER sub.ref == doc._key RETURN sub)
SORT sub1
LIMIT 10
LET sub2 = FIRST(FOR sub IN collection3 FILTER sub.ref == doc._key RETURN sub)
RETURN { doc, sub1, sub2 }
* Added JSON-Schema (draft-4) document validation. The validation can be
specified by providing the new `schema` collection property when creating a
new collection or when updating the properties of an existing collection:
db.mycollection.properties({
schema: {
rule : { nums : { type : "array", items : { type : "number", maximum : 6 }}},
message : "Json-Schema validation failed"
}
});
* Fix supervision mode detection when unlocking agency in hot backup.
* Foxx Response#throw will no longer generate incomplete error objects when
passed invalid arguments.
* Foxx will now always generate a numeric HTTP status code even if the response
status code has been set to a non-numeric value.
* Make arangobench return a proper error message when its initial attempt to
create the test collection fails.
* Added traversal options `vertexCollections` and `edgeCollections` to restrict
traversal to certain vertex or edge collections.
The use case for `vertexCollections` is to not follow any edges that will point
to other than the specified vertex collections, e.g.
FOR v, e, p IN 1..3 OUTBOUND 'products/123' components
OPTIONS { vertexCollections: [ "bolts", "screws" ] }
The traversal's start vertex is always considered valid, regardless of whether
it is present in the `vertexCollections` option.
The use case for `edgeCollections` is to not take into consideration any edges
from edge collections other than the specified ones, e.g.
FOR v, e, p IN 1..3 OUTBOUND 'products/123' GRAPH 'components'
OPTIONS { edgeCollections: [ "productsToBotls", "productsToScrews" ] }
This is mostly useful in the context of named graphs, when the named graph
contains many edge collections. Not restricting the edge collections for the
traversal will make the traversal search for edges in all edge collections of
the graph, which can be expensive. In case it is known that only certain edges
from the named graph are needed, the `edgeCollections` option can be a handy
performance optimization.
* Make arangobench return a proper error message when its initial attempt to
create the test collection fails.
* In some cases with a COLLECT LIMIT situation on a small limit the collect
does more calls to upstream than without a limit to provide the same
result. We improved this situation and made sure that LIMIT does
not cause the COLLECT to fetch too few input rows. There is a chance
that queries with a very small amount of data might suffer from this
modification, most queries will benefit however.
* Use OpenSSL's EVP interface for SHA256 instead of the deprecated low-level
message digest APIs.
* Prevent startup error "specifiy either '--server.jwt-secret-keyfile' or
'--server.jwt-secret-folder' but not both." from occurring when specifying the
JWT secret keyfile option.
* Honor `cacheEnabled` attribute when updating the properties of a collection
in case of the cluster. Previously, this attribute was ignored in the cluster,
but it shouldn't have been.
* Fixed issue #11137: http-proxy can't pass delete method req.body to foxx.
* Disable "collect-in-cluster" AQL optimizer rule in case a LIMIT node is
between the COLLECT and the source data. In this case it is not safe to apply
the distributed collect, as it may alter the results.
* Rebuild UI.
* Fix arangorestore:
If a smartGraphAttribute value was changed after document creation,
the _key of the document becomes inconsistent with the smartGraphAttribute
entry. However, the sharding of documents stays intact.
When restoring data from a dump that contains a document with inconsistent
_key and smartGraphAttribute this would cause an error and the document
would not be inserted. This is fixed by ignoring this inconsistency in
the case of restore.
* Updated rclone to 1.51.1.
* Fixed a memory leak in ModificationExecutors.
* Updated arangosync to 0.7.3.
* Updated ArangoDB Starter to 0.14.14.
* Fixed handling of `geometry` attributes in query editor of the web UI.
Previously, document attributes named `geometry` were treated in a special
way, assuming they were JSON objects. In case a document contained a
`geometry` attribute that had a non-object value (e.g. `null`) the web UI
threw a JavaScript exception and would not display AQL query results properly.
* Disable "collect-in-cluster" AQL optimizer rule in case a LIMIT node is between
the COLLECT and the source data. In this case it is not safe to apply the
distributed collect, as it may alter the results.
* Fixed internal issue #4932: COLLECT WITH COUNT together with FILTER yields
zero.
This bugfix fixes an issue when skipping over documents in an index scan
using a covering index and also at the same time using an early-pruning filter.
In this case wrong document data may have been injected into the filter
condition for filtering, with the filter wrongfully deciding which documents
to filter out.
* Added crash handler for Linux builds that taps into the following signals:
* SIGSEGV (segmentation violation)
* SIGBUS (bus error)
* SIGILL (illegal instruction)
* SIGFPE (floating point exception)
In case the arangod process catches one these signals, the crash handler
tries to log a message and a backtrace into the installation's logfile before
terminating the process.
* Removed dependency on libcurl library.
* Remove storage engine selection from package installers. The default storage
engine for new installations is now RocksDB. The MMFiles storage engine cannot
be selected for new installations.
* Remove misleading error messages about upgrade commands to carry out in
case of a database upgrade. These commands are highly platform-dependent and
also depend on whether ArangoDB is started manually, via the ArangoDB starter
or as a service.
In order to not confuse end users, remove the potentially misleading instructions.
* Clear backups from DB servers and agency, when plan unchanged not
met and not allowing for inconsistency.
* V8 version upgrade to 7.9.317; ICU version upgrade to 64.2.
- JSON parsing is roughly 60% faster than in V8 7.1; you should prefer
JSON.parse(string) over deeply nested javascript variable declarations.
- several new javascript language features
* Clean out server job checks preconditions plan version unchanged for start.
* Cluster collection creation preconditioned on involved db servers not
in process of being cleaned and Fixed.
* Fixed a bug, where a single host agency logged too early.