forked from arangodb/arangodb
/
CHANGELOG
16782 lines (11263 loc) 路 667 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
devel
-----
* Updated OpenSSL to 1.1.1i and OpenLDAP to 2.4.56.
* Make internal ClusterInfo::getPlan() wait for initial plan load from agency.
* Added AQL timezone functions `DATE_TIMEZONE` and `DATE_TIMEZONES`.
* Make DB servers report storage engine health to the agency, via a new "health"
attribute in requests sent to Sync/ServerStates/<id>.
The supervision can in the future check this attribute if it is posted,
and mark servers as BAD or FAILED in case an unhealthy status is reported.
DB server health is currently determined by whether or not the storage engine
(RocksDB) has reported a background error, and by whether or not the free disk
space has reached a critical low amount. The current threshold for free disk
space is set at 1% of the disk capacity (only the disk is considered that
contains the RocksDB database directory).
The minimum required free disk space percentage can be configured using the new
startup option `--rocksdb.minimum-disk-free-percent`, which needs to be between
0 and 1 (including). A value of 0 disables the check.
The minimum required free disk space can also be configured in bytes using the
new startup option `--rocksdb.minimum-disk-free-bytes`. A value of 0 disables
this check, too.
* Failed servers are now reported consistently in the web interface, at
approximately the same time in the navigation bar and in the nodes view.
Previously these two places had their own, independent poll mechanism for the
nodes' health, and they were updated independently, which could cause an
inconsistent view of the nodes' availability.
Using only one poll mechanism instead also saves some period background requests
for the second availability check.
* Updated arangosync to 1.2.1.
* Clean up callback bin and empty promises in single-host-agency.
* Stabilize a Foxx cleanup test.
* Drop a pair of braces {} in /_admin/metrics in case of empty labels, which
makes the API adhere better to the official Prometheus syntax.
* Add some more metrics to the ConnectionPool.
* Remove HTTP "Connection" header when forwarding requests in the cluster
from one coordinator to another, and let the internal network layer
handle closing of connections and keep-alive.
* Added new metric: "arangodb_collection_lock_sequential_mode" this will count
how many times we need to do a sequential locking of collections. If this
metric increases this indicates lock contention in transaction setup.
Most likely this is caused by exlcusive locks used on collections with
more than one shard.
* Fix for BTS-213
Changed the transaction locking mechanism in the cluster case.
For all installations that do not use "exclusive" collection locks this change
will not be noticable. In case of "exclusive" locks, and collections with more
than one shard, it is now less likely to get a LOCK_TIMEOUT (ErrorNum 18).
It is still possible to get into the LOCK_TIMEOUT case, especially if
the "exclusive" operation(s) are long-running.
* Reduce overhead of audit logging functionality if audit logging is turned
off.
* Add several more attributes to audit-logged queries, namely query execution
time and exit code (0 = no error, other values correspond to general ArangoDB
error codes).
* Added "startupTime", "computationTime" and "storageTime" to Pregel result
statistics.
* Fixed a bug in maintainer mode sorting followerinfo lists the wrong way.
* Limit value of `--rocksdb.block-cache-size` to 1 GB for agent instances to
reduce agency RAM usage, unless configured otherwise. In addition, limit the
value of `--rocksdb.total-write-buffer-size` to 512 MB on agent instances for
the same reason.
* Fixed issue #13238 Thread naming API on Windows are now used only if
available in KERNEL32.DLL
* When querying the list of currently running or slow AQL queries, ignore not-yet
created databases on other coordinators.
* Added support for fetching the list of currently running and slow AQL queries
from all databases at once, by adding an `all` parameter to the following
query APIs:
* `require("@arangodb/aql/queries").current({ all: true })`: will return the
currently running queries from all databases, not just the currently
selected database.
* HTTP GET `/_api/query/current?all=true`: same, but for the HTTP REST API.
* `require("@arangodb/aql/queries").slow({ all: true })`: will return the
slow query history from all databases, not just the currently selected
database.
* HTTP GET `/_api/query/slow?all=true`: same, but for the HTTP REST API.
* `require("@arangodb/aql/queries").clearSlow({ all: true })`: will clear
the slow query history for all databases, not just the currently selected
database.
* HTTP DELETE `/_api/query/slow?all=true`: same, but for the HTTP REST API.
Using the `all` parameter is only allowed when making the call inside the
`_system` database and with superuser privileges.
* Improve performance and memory efficiency of agency restart from persisted
database directory.
* Added the following agency-related metrics:
- `arangodb_agency_client_lookup_table_size`: current number of entries
in agency client id lookup table. This gauge is available only on
agent instances.
- `arangodb_agency_cache_callback_count`: current number of entries in
agency cache callbacks table. This gauge will be effective on coordinators
and DB servers.
- `arangodb_agency_callback_count`: current number of agency callbacks
registered. This gauge will be effective on coordinators and DB servers.
* Clean up agency change log, cluster info caches.
* Added new `rocksdb_write_stalls` and `rocksdb_write_stops` counter metrics,
which should be more accurate than existing metrics related to the underlying
conditions.
* Increased the default value of `--rocksdb.min-write-buffer-number-to-merge` in
some cases when we have allocated a sufficient amount of memory to the write
buffers for this to make sense. The increased value should help prevent
compaction-induced write stalls/stops, and should only be enabled when under
conditions such that it shouldn't greatly increase the chance of flush-induced
write stalls/stops.
* Changed the default values for `--rocksdb.cache-index-and-filter-blocks` and
`--rocksdb.cache-index-and-filter-blocks-with-high-priority` to true to
improve control over memory usage.
* Lowered the minimum allowed value for `--rocksdb.max-write-buffer-number` from
9 to 4 to allow more fine-grained memory usage control.
* Fix for issue #772: Optimized document counting for ArangoSearch views.
Added new ArangoSearch view option 'countApproximate' for customizing
view count strategy.
* Fix AR-113: Add a check in the AQL geo-index-optimizer rule that vars
referenced by a candidate index expression are actually available at the
candidate index node.
* Views on SmartGraph Edge collections do not contain some documents
twice.
* Fixed agency redirect in poll api.
* Fixed issue #12248: Web UI - Added missing HTML escaping in the setup script
section of a foxx app.
* The scheduler will now run a minimum of 4 threads at all times, and the
default and minimal value for `--server.maximal-threads` has been lowered from
64 to the greater of 32 and twice the number of detected cores.
* Throttle work coming from low priority queue, according to a constant
and to an estimate taking into account fanout for multi-shard operations.
* Move to 4 priority levels "low", "medium", "high" and "maintenance" in
scheduler to ensure that maintenance work and diagnostics is always
possible, even in the case of RocksDB throttles. Do not allow any
RocksDB work on "maintenance".
* Commit replications on high priority queue.
* Essentially get rid of timeout in replication to drop followers. This
is now entirely handled via reboot and failure tracking. The timeout
has now a default minimum of 15 minutes but can still be configured via
options.
* Additional metrics for all queue lengths and low prio ongoing work.
* New metric for number and total time of replication operations.
* New metrics for number of internal requests in flight, internal request
duration, and internal request timeouts
* Fix `Gauge` assignment operators.
* Fixed and extended LDAP log messages.
* Added LDAP_OFF if referrals and restart are false.
* If LDAP search fails, also retry (update to given number of retries).
* Add cluster support for collection.checksum() method to calculate CRC
checksums for collections.
* Fix cluster-internal replication of documents with special keys (percent
character, which has a special meaning when used inside URLs).
* Fix AQL cost estimate of spliced subqueries which could lead to overly large
numbers in the explain output of such queries.
* Make all Pregel HTTP and JavaScript APIs also accept stringified execution
number values, in addition to numeric ones.
This allows passing larger execution numbers as strings, so that any data
loss due to numeric data type conversion (uint32 => double) can be avoided.
The change also makes the Pregel HTTP and JavaScript APIs for starting a
run return a stringified execution number, e.g. "12345" instead of 12345.
* Updated ArangoDB Starter to 0.14.15-1.
* Fix Windows directory creation error handling.
* Add an AQL query kill check during early pruning. Fixes issue #13141.
* Remove a case in which followers were dropped unnecessarily in streaming
transactions that replicated to the same follower.
* Fixed ES-662 by introducing refactored thread pool to make more efficient
consolidation and commit routines for links of ArangoSearch views.
Added new command line options for fine-grained ArangoSearch maintenance
control:
- `--arangosearch.commit-threads` - max number of ArangoSearch commit
threads
- `--arangosearch.commit-threads-idle` - min number of ArangoSearch
commit threads
- `--arangosearch.consolidation-threads` - max number of ArangoSearch
consolidation threads
- `--arangosearch.consolidation-threads-idle` - min number of ArangoSearch
consolidation threads
Deprecated the following command line options:
- `--arangosearch.threads` - will be used in --arangosearch.commit-threads`
and `--arangosearch.consolidation-threads` as provided value divided by 2
if set to non-autodetect value > 0
- `--arangosearch.threads-limit`
* Turn off `StatisticsWorker` thread on DB servers.
This thread was previously only running queries on the local RocksDB
instance, but using the cluster-wide collection names. So effectively it
did nothing except use a bit of background CPU. In this case it is better
to turn off the background thread entirely on the DB servers.
* Avoid the usage of std::regex when constructing date/time string values
for log messages. This is a performance optimization only.
* Reimplement coordshort request handler. The new implementation only runs
two DB queries without any additional requests to other coordinators,
resulting in reduced load on the cluster. Previously this involved
requests to all coordinators, where each of them ran two DB queries.
* Fixed initial population of local AgencyCache values after a server restart.
Previously the local cache was populated from the agency using a commit index
value of 1, whereas it should have been 0 to get the full agency snapshot.
* Increase background garbage-collection interval for cluster transactions
from 1 second to 2 seconds. This change should reduce the amount of
background task activity a tiny bit (though hardly measurable on an
otherwise idle server).
* Make the audit log honor the configured logging date/time output format
(i.e. `--log.time-format` option). Previously the audit logging always
created a time value in the server's local time, and logged it in
format YYYY-MM-DDTHH:MM:SS.
From 3.8 onwards, the audit logger will honor the date/time format
specified via the `--log.time-format` option, which defaults to
`utc-datestring`. The means the audit logging will by default log all
dates/times in UTC time. To restore the pre-3.8 behavior, please set
the option `--log.time-format` to `local-datestring`, which will make
the audit logger (and all other server log messages) use the server's
local time.
* Fix the activation of the agency supervision maintenance via the REST API
`/_admin/cluster/maintenance`. This API stored a boolean value instead of
an (expected) maintenance period end date/time string.
* Make the cancel operation safe for asynchronoulsly started JavaScript
transactions (via HTTP POST to `/_api/transaction` with the `x-arango-async`
header set).
* Updated OpenSSL to 1.1.1h.
* Added new scheduler metrics:
- `arangodb_scheduler_jobs_done`: Total number of scheduler queue jobs done
- `arangodb_scheduler_jobs_submitted`: Total number of jobs submitted to the
scheduler queue
- `arangodb_scheduler_jobs_dequeued`: Total number of jobs dequeued from the
scheduler queue
- `arangodb_scheduler_num_working_threads`: Number of currently working
scheduler threads
* Added new metric `arangodb_replication_cluster_inventory_requests` to track
the number of requests received for cluster inventories. The cluster
inventory API is called at the beginning of a dump process or by arangosync.
* Added startup option `--server.unavailability-queue-fill-grade`. This option
has a consequence for the `/_admin/server/availability` API only, which is
often called by load-balancers and other availability probing systems.
The `/_admin/server/availability` API will now return HTTP 200 if the fill
grade of the scheduler's queue is below the configured value, or HTTP 503 if
the fill grade is above it. This can be used to flag a server as unavailable
in case it is already highly loaded.
The default value for this option is `0.75`, i.e. 75%. This is a change
compared to previous versions of ArangoDB, where the default value was `1`.
To prevent sending more traffic to an already overloaded server, it can be
sensible to reduce the default value to even `0.5`.
This would mean that instances with a queue longer than 50% of their
maximum queue capacity would return HTTP 503 instead of HTTP 200 when their
availability API is probed.
nb: the default value for the scheduler queue length is 4096.
* Added metrics for the system CPU usage:
- `arangodb_server_statistics_user_percent`: Percentage of time that the
system CPUs have spent in user mode
- `arangodb_server_statistics_system_percent`: Percentage of time that
the system CPUs have spent in kernel mode
- `arangodb_server_statistics_idle_percent`: Percentage of time that the
system CPUs have been idle
- `arangodb_server_statistics_iowait_percent`: Percentage of time that
the system CPUs have been waiting for I/O
These metrics resemble the overall CPU usage metrics in `top`.
They are available on Linux only.
* Fix internal issue #777: Fixed memory access while substituting
stored values for ArangoSearch view optimization
* Make the number of network I/O threads properly configurable via the
startup option `--network.io-threads`. This option existed before, but its
configured value was effectively clamped to a value of `1`.
* Improvements for the Pregel distributed graph processing feature:
- during the loading/startup phase, the in-memory edge cache is now
intentionally bypassed. The reason for this is that any edges are
looked up exactly once, so caching them is not beneficial, but would
only lead to cache pollution.
- the loading/startup phase can now load multiple collections in parallel,
whereas previously it was only loading multiple shards of the same
collection in parallel. This change helps to reduce load times in case
there are many collections with few shards, and on single server.
- the loading and result storage phases code has been overhauled so that
it runs slightly faster.
- for Pregel runs that are based on named graphs (in contrast to explicit
naming of the to-be-used vertex and edge collections), only those edge
collections are considered that, according to the graph definition, can
have connections with the vertex. This change can reduce the loading
time substantially in case the graph contains many edge definitions.
- the number of executed rounds for the underlying Pregel algorithm now
does not vary for different `parallelism` values.
* Fix HTTP/1.1 status response header in fuerte responses
This change makes fuerte return the full status header, including the
numeric status code and the status string in the `http/1.1` header of
fuerte responses.
Previously, the return header lacked the numeric status code, so it
looked like
```
"http/1.1" : "Ok"
```
Now, with the numeric status code, the response header will look like
```
"http/1.1" : "200 Ok"
```
This PR also adds a protocol() method for arango client connections in
order to check the protocol in use. The possible return values are
- "http" for HTTP/1.1 connections
- "http2" for HTTP/2 connections
- "vst" for VST connections
- "unknown" for everyhting else
This is needed during testing, but can also be used for other purposes.
* Updated arangosync to 0.7.12.
* Fix log topic of general shutdown message from "cluster" to general.
* Automatically add "www-authenticate" headers to server HTTP 401 responses,
as required by the HTTP specification.
* Fixed an AQL bug that ignored PRUNE statements in OneShard setups.
* Make the DOCUMENT AQL function eligible for running on DB servers in
OneShard deployment mode. This allows pushing more query parts to DB servers
for execution.
* Enable HTTP request statistics and provide metrics even in case
`--server.statistics-history` is set to `false` (this option will set
itself to off automatically on agency instances on startup if not
explicitly set).
This change provides more metrics on all server instances, without the
need to persist them in the instance's RocksDB storage engine.
* Fixed a deadlock between AQL write transactions and hotbackup, since
in AQL write transactions follower transactions did not know they are
follower transactions.
* Added metrics for collection locks:
- `arangodb_collection_lock_timeouts_exclusive`: Number of lock timeouts
when trying to acquire collection exclusive locks
- `arangodb_collection_lock_timeouts_write`: Number of lock timeouts when
trying to acquire collection write locks
- `arangodb_collection_lock_acquisition_micros`: Total amount of collection
lock acquisition time [渭s]
- `arangodb_collection_lock_acquisition_time`: Total collection lock
acquisition time histogram [s]
* Reduce lock timeout on followers to 15 seconds.
Rationale: we should not have any locking conflicts on followers, generally.
Any shard locking should be performed on leaders first, which will then,
eventually replicate changes to followers. replication to followers is only
done once the locks have been acquired on the leader(s).
* Fix a memory leak because cluster internal connections were not cleaned
up for agency communication.
* Added compile option USE_JEMALLOC_PROF to enable memory profiling.
* Remove extra CMake option `DEBUG_SYNC_REPLICATION` and use the already
existing `USE_FAILURE_TESTS` options for its purpose.
* Updated bundled version of Snappy compression/decompression library to 1.1.8.
* Fix REST API endpoint PUT `/_api/collection/<collection>/recalculateCount` on
coordinators. Coordinators sent a wrong message body to DB servers here, so
the request could not be handled properly.
* Fixed issue #12778: fails validation if additionalProperties: false.
* Fixed BTS-233 issue: Fixed invalid IndexId comparator.
* Fixed potential deadlock in cluster transactions if a transaction is
returned that was soft-aborted by transaction garbage collection before.
This deadlock should rarely ever occur in practice, as it can only be
triggered once during the server shutdown sequence.
* Added support of `GEO_DISTANCE`, `GEO_CONTAINS`, `GEO_INTERSECTS`,
`GEO_IN_RANGE` to ArangoSearch.
* Added new `GeoJSON` ArangoSearch analyzer.
* Added new `GeoPoint` ArangoSearch analyzer.
* Added new `GEO_IN_RANGE` AQL function.
* Fixed handling of failedLeaderJob. In case of a plan modification, that
removes a server from the plan, e.g. reduce replication factor. Directly
followed by a failure of the current shard leader, would reinsert
the just removed server in the plan, which is undesired, we first need
to have a full "desync" cycle on this server to be reusable in the plan
again.
* Fixed very spurious errors if the `holdReadLockCollection` replication API for
the getting-in-sync procedure of shards was called during server shutdown.
In this case that method could ask the transaction manager for a specific
transaction, but wasn't returning one due to the server shutdown.
* Added new 'aql' type for ArangoSearch analyzers.
* Obsoleted the startup options `--database.throw-collection-not-loaded-error`
and `--ttl.only-loaded-collection`.
These options were meaningful for the MMFiles storage engine only, but for
the RocksDB storage engine they did not make any difference. Using these startup
options is still possible, but will have no effect other than generating a
warning at server startup.
* Added CMake option `USE_MINIMAL_DEBUGINFO`.
This option is turned off by default. If turned on, the created binaries
will contain only a minimum amount of debug symbols, reducing the size
of the executables. If turned off (which is the default), the binaries
will contain full debug information, which will make them bigger in size
unless the debug information is later stripped again.
* Modified the returned error code for calling the `shards()` function on a
collection in single-server from "internal error" (error number 4) to "shards
API is only available in cluster" and error number 9, HTTP status code 501.
* Bug-fix: Allow to unlink a view created on a SmartGraphEdge collection.
* If a collection (or database) is dropped during the instantiation of an AQL query,
the setup code now aborts with an ERROR_QUERY_COLLECTION_LOCK_FAILED and earlier.
Before the setup code could abort with TRI_ERROR_INTERNAL in the same case.
* Added WINDOW keyword to AQL to allow aggregations on related rows.
* Added new graph method K_PATHS to AQL. This will enumerate all paths between a
source and a target vertex that match the given length.
For example, the query
```
FOR path IN 2..4 OUTBOUND K_PATHS "v/source" TO "v/target" GRAPH "g"
RETURN path
```
will yield all paths in format
{
vertices: [v/source, ... , v/target],
edges: [v/source -> v/1, ..., v/n -> v/target
}
that have length exactly 2 or 3 or 4, start at v/source and end at v/target.
The order of those paths in the result set is not guaranteed.
* Fixed issue BTS-195: AQL update queries using the `keepNull` option set to
false had an inconsistent behavior. For example, given a collection `test`
with an empty document with just key `testDoc`, the following query
would return different results when running for the first time or the second
time:
UPDATE 'testDoc'
WITH {test: {sub1: true, sub2: null}} IN test
OPTIONS { keepNull: false, mergeObjects: true }
For its first run, the query would return
{
"_key": "testDoc",
"test": {
"sub1": true,
"sub2": null
}
}
(with the `null` attribute value not being removed). For all subsequent runs,
the same query would return
{
"_key": "testDoc",
"test": {
"sub1": true,
}
}
(with the `null` value removed as requested).
This inconsistency was due to how the `keepNull` attribute was handled if
the attribute already existed in the to-be-updated document or not. The
behavior is now consistent, so `null` values are now properly removed from
sub-attributes even if in the to-be-updated document the target attribute
did not yet exist. This makes such updates idempotent again.
This a behavior change compared previous versions, but it will only have
effect when `keepNull` is set to `false` (the default value is `true` however),
and only when just-inserted object sub-attributes contained `null` values.
* Optimization of empty append entries.
* Bug-fix: Creating an additional index on the edge collection of a disjoint
SmartGraph could falsely result into an error:
`Could not find all smart collections ...`
This is now ruled out and indexes can be created as expected.
* Added startup option `--foxx.force-update-on-startup` to toggle waiting
for all Foxx services in all databases to be propagated to a coordinator
before it completes the boot sequence.
In case the option is set to `false` (i.e. no waiting), the coordinator
will complete the boot sequence faster, and the Foxx services will be
propagated lazily. Until the initialization procedure has completed for
the local Foxx apps, any request to a Foxx app will be responded to with
an HTTP 503 error and message
waiting for initialization of Foxx services in this database
This can cause an unavailability window for Foxx services on coordinator
startup for the initial requests to Foxx apps until the app propagation
has completed.
When not using Foxx, this option should be set to `false` (default) to
benefit from a faster coordinator startup.
Deployments relying on Foxx apps being available as soon as a coordinator
is integrated or responding should set this option to `true`.
The option only has an effect for cluster setups.
On single servers and in active failover mode, all Foxx apps will be
available from the very beginning.
Note: ArangoDB 3.6 and 3.7 also introduced this option, but with a default
value of `true`. ArangoDB 3.8 changes the default to `false`.
* Changed the server-side implementation of the following internal JavaScript
APIs to no-ops:
* `internal.reloadAqlFunctions()`: this is a no-op function now
* `@arangodb/actions.buildRouting()`: this is a no-op function now
* `@arangodb/actions.routingTree`: will return an empty object
* `@arangodb/actions.routingList`: will return an empty object
All the above APIs were intended to be used for internal means only. These
APIs are deprecated now and will be removed in ArangoDB v3.9.
* Instead of failing to connect to INADDR_ANY refuse it as a parameter, with a
descriptive error message for novice users (issue #12871).
* Remove any special handling for obsoleted collection attributes
`indexBuckets`, `journalSize`, `doCompact` and `isVolatile`. These
attributes were meaningful only with the MMFiles storage engine and have
no meaning with the RocksDB storage engine. Thus any special handling
for these attributes can be removed in the internal code.
Client applications and tests that rely on the behavior that setting
any of these attributes produces an error when using the RocksDB engine
may need adjustment now.
* Added a --continue options to arango restore. arangorestore now keeps track of the progress
and can continue the restore operation when some error occured.
* Added the following metrics for synchronous replication in the cluster:
- `arangodb_refused_followers_count`: Number of times a shard leader received
a refusal answer from a follower during synchronous replication.
- `arangodb_sync_wrong_checksum`: Number of times a mismatching shard
checksum was detected when syncing shards. In case this happens, a resync
will be triggered for the shard.
* Don't respond with misleading error in smart vertex collections.
When inserting a document with a non-conforming key pattern into
a smart vertex collection, the response error code and message are
1466 (ERROR_CLUSTER_MUST_NOT_SPECIFY_KEY) and "must not specify _key
for this collection".
This is misleading, because it is actually allowed to specify a key
value for documents in such collection. However, there are some
restrictions for valid key values (e.g. the key must be a string and
contain the smart graph attribute value at the front, followed by a
colon.
If any of these restrictions are not met, the server currently
responds with "must not specify key for this collection", which is
misleading. This change rectifies it so that the server responds with
error 4003 (ERROR_KEY_MUST_BE_PREFIXED_WITH_SMART_GRAPH_ATTRIBUTE)
and message "in smart vertex collections _key must be a string and
prefixed with the value of the smart graph attribute". This should
make it a lot easier to understand what the actual problem is.
* Fixed handling of failoverCandidates. Sometimes, a server can still be a
failoverCandidate even though it has been taken out of the Plan. With this
fix, such a server is quickly taken out of failoverCandidates and it can never
be re-added to the Plan before this has happened.
* Added configuration option `--query.tracking-slow-queries` to decide whether
slow queries are tracked extra.
* Added configuration option `--query.tracking-with-querystring` to decide
whether the query string is shown in the slow query log and the list of
currently running queries. The option is true by default.
When turned off, querystrings in the slow query log and the list of currently
running queries are just shown as "<hidden>".
* Added configuration option `--query.tracking-with-datasources` to toggle
whether the names of data sources used by queries are shown in the slow query
log and the list of currently running queries. The option is false by default.
When turned on, the names of data sources used by the query will be shown in
the slow query log and the list of currently running queries.
* Fix an issue in arangoimport improperly handling filenames with less than 3
characters. The specified input filename was checked for a potential ".gz"
ending, but the check required the filename to have at least 3 characters.
This is now fixed.
* Added optional verbose logging for agency write operations. This logging
is configurable by using the new log topic "agencystore".
The following log levels can be used for for the "agencystore" log topic
to log writes to the agency:
- DEBUG: will log all writes on the leader
- TRACE: will log all writes on both leaders and followers
The default log level for the "agencystore" log topic is WARN, meaning no
agency writes will be logged.
Turning on this logging can be used for auditing and debugging, but it is
not recommended in the general case, as it can lead to large amounts of
data being logged, which can have a performance impact and will lead to
higher disk space usage.
* Fix #12693: SORT inside a subquery could sometimes swallow part of its input
when it crossed boundaries of internal row batches.
* Added configuration option `--rocksdb.sync-delay-threshold`.
This option can be used to track if any RocksDB WAL sync operation is
delayed by more than the configured value (in milliseconds). The intention
is to get aware of severely delayed WAL sync operations.
* Fix for BTS-191: Made transaction API database-aware.
* Minor clean up of and less verbosity in agent callbacks.
* Speed up initial replication of collections/shards data by not wrapping
each document in a separate `{"type":2300,"data":...}` envelope. In
addition, the follower side of the replication will request data from
leaders in VelocyPack format if the leader is running at least version
3.8.
Stripping the envelopes and using VelocyPack for transfer allows for
smaller data sizes when exchanging the documents and faster processing,
and thus can lead to time savings in document packing and unpacking as
well as reduce the number of required HTTP requests.
* Add database, shard name and error information to several shard-related log
messages.
* Display shard names of a collection in the web interface when in the details
view of the collection.
* Added HTTP requests metrics for tracking the number of superuser and normal
user requests separately:
- `arangodb_http_request_statistics_superuser_requests`: Total number of HTTP
requests executed by superuser/JWT
- `arangodb_http_request_statistics_user_requests`: Total number of HTTP
requests executed by clients
* Added metric `arangodb_agency_callback_registered counter` for tracking the
total number of agency callbacks that were registered.
* Fixed a bug in handling of followers which refuse to replicate operations.
In the case that the follower has simply been dropped in the meantime, we now
avoid an error reported by the shard leader.
* Added weighted traversal. Use `mode: "weighted"` as option to enumerate
paths by increasing weights. The cost of an edge can be read from an
attribute which can be specified using `weightAttribute` option.
* Fix a performance regression when a LIMIT is combined with a COLLECT WITH
COUNT INTO. Reported in ES-692.
* Fixed issue ES-696: SEARCH vs FILTER lookup performance.
Consolidation functionality for ArangoSearch view links was able to hit non-
mergable enormous amount of segments due to improper scheduling logic.
* Data definition reconciliation in cluster has been modified
extensively to greatly accelerate the creation of 1000s of
databases through following means:
- AgencyCache offers change sets API based on Raft index.
- ClusterInfo caches are only updated using change sets.
- Maintenance uses local as well as agency change sets to limit
the scope of every runtime to these change sets.
* Make scheduler react and start new threads slightly faster in case a lot
of new work arrives.
* Make scheduler properly count down the number of working threads in case
an exception happens in a worker thread.
* Added startup option `--database.old-system-collections` to toggle automatic
creation of system collections `_modules` and `_fishbowl`, along with their
internal usage. These collections are useful only in very few cases, so it
is normally not worth to create them in all databases.
The `_modules` collection is only used to register custom JavaScript modules,
for which there exists no API, and `_fishbowl` is used to store the temporary
list of Foxx apps retrieved from the Github Foxx store.
If the option value is `false` (which is the default from v3.8 onwards), the
two collections will not be created for any new database. The `_fishbowl`
collection will still be created dynamically when needed. If the option value
is `true`, the collections will be created regularly as before.
The option will also be introduced to v3.7, where it will have a default
value of `true`, meaning the collections will still be created there.
Any functionality related to the `_modules` system collection is deprecated
and will be removed in ArangoDB v3.9.
Two side effects of turning this option off (which is the default) are:
* there will be no iteration over all databases at server startup just to check
the contents of all `_modules` collections.
* less collections/shards will be around for deployments that create a large
number of databases.
Already existing `_modules` and `_fishbowl` system collections will not be
modified by this change, even though they will likely be empty and unused.
* Don't iterate over all databases at server startup in order to initialize the
routing information. This is not necessary, as the routing information is
global and not tied to a specific database.
Any functionality related to the `_modules` system collection is deprecated
and will be removed in ArangoDB v3.9.
* Use rclone built from v1.51.0 source with go1.15.2 instead of prebuilt
v1.53.0 release.
* Fixed a possible crash during instantiation of an AQL graph traversal.
Reported in #12597.
* Added new ArangoSearch "pipeline" analyzer type
* Reduce the number of dropped followers when running larger (>= 128 MB)
write transactions.
* Fixed a bug in AQL COLLECT with OPTIONS { "hash" } that led to a quadratic
runtime in the number of output rows.
* Make the reboot tracker catch failed coordinators, too. Previously the
reboot tracker was invoked only when a DB server failed or was restarted,
and when a coordinator was restarted. Now it will also act if a coordinator
just fails (without restart).
* Added scheduler thread creation/destruction metrics:
- `arangodb_scheduler_threads_started`: Number of scheduler threads started
- `arangodb_scheduler_threads_stopped`: Number of scheduler threads stopped
* Added replication metrics `arangodb_replication_initial_sync_bytes_received`
for the number of bytes received during replication initial sync operations
and `arangodb_replication_tailing_bytes_received` for the number of bytes
received for replication tailing requests.
Also added `arangodb_replication_failed_connects` to track the number of
connection failures or non-OK response during replication.
* Added metrics `rocksdb_free_inodes` and `rocksdb_total_inodes` to track the
number of free inodes and the total/maximum number of inodes for the file
system the RocksDB database directory is located in. These metrics will
always be 0 on Windows.
* Fixed infinite reload of the login window after logout of an LDAP user.
* Added startup option `--query.max-runtime` to limit the maximum runtime of
all AQL queries to a specified threshold value (in seconds). By default,
the threshold is 0, meaning that the runtime of AQL queries is not limited.
Setting it to any positive value will restrict the runtime of all AQL
queries unless it is overwritten in the per-query "maxRuntime" query option.
Please note that setting this option will affect *all* queries in all
databases, and also queries issues for administration and database-internal
purposes.
If a query exceeds the configured runtime, it will be killed on the next
occasion when the query checks its own status. Killing is best effort,
so it is not guaranteed that a query will no longer than exactly the
configured amount of time.
* Updated rclone to 1.53.0.
* Fixed slightly wrong log level for authentication and also added login event
to the standard log.
* Ensure that the argument to an AQL OPTIONS clause is always an object
which does not contain any dynamic (run-time) values. Previously, this
was only enforced for traversal options and options for data-modification
queries. This change extends the check to all occurrences of OPTIONS.
* Added `details` option to figures command of a collection:
`collection.figures(details)`
Setting `details` to `true` will return extended storage engine-specific
details to the figures. The details are intended for debugging ArangoDB
itself and their format is subject to change. There is not much use in using
the details from a client application.
By default, `details` is set to `false`, so no details are returned and the
behavior is identical to previous versions of ArangoDB.
* Enforce a maximum result register usage limit in AQL queries. In an AQL
query, every user-defined or internal (unnamed) variable will need a
register to store results in.
AQL queries that use more result registers than allowed (currently 1000)
will now abort deterministically during the planning stage with error 32
(`resource limit exceeded`) and the error message
"too many registers (1000) needed for AQL query".
Before this fix, an AQL query that used more than 1000 result registers
crashed the server when assertions were turned on, and the behavior was
undefined when assertions were turned off.
* Implement RebootTracker usage for AQL queries in case of coordinator
restarts or failures. This will clean up the rest of an AQL query
on dbservers more quickly and in particular release locks faster.
* Serialize maintenance actions for each shard. This addresses lost document
problems found in chaos testing.
* Fixed an issue with audit logging misreporting some document requests as
internal instead of logging the proper request information
* Add option `--rocksdb.max-write-buffer-size-to-maintain` with default of 0.
This configures how much memory RocksDB is allowed to use for immutable
flushed memtables/write-buffers. The default of 0 will usually be good
for all purposes and restores the 3.6 memory usage for write-buffers.
* Updated arangosync to 0.7.10.
* Make followers in active failover run a compaction after they process a
truncate operation and the truncate removed more than 4k documents. This
can help to reclaim disk space on the follower earlier than without running
the truncate.
* Added REST API PUT `/_admin/compact` for compacting the entire database
data. This endpoint can be used to reclaim disk space after substantial data
deletions have taken place. The command is also exposed via the JavaScript
API as `db._compact();`.
This command can cause a full rewrite of all data in all databases, which
may take very long for large databases. It should thus only be used with care
and only when additional I/O load can be tolerated for a prolonged time.
This command requires superuser access.
* Added new metrics for the total and the free disk space for the mount
used for the RocksDB database directory:
* `arangodb_rocksdb_free_disk_space`: provides the free disk space for
the mount, in bytes
* `arangodb_rocksdb_total_disk_space`: provides the total disk space of
the mount, in bytes
* Fixed some cases where subqueries in PRUNE did not result in a parse error,
but either in an incomprehensible error (in 3.7), or undefined behavior
during execution (pre 3.7).
* Apply user-defined idle connection timeouts for HTTP/2 and VST connections.
The timeout value for idle HTTP/2 and VST connections can now be configured
via the configuration option `--http.keep-alive-timeout` in the same way
as for HTTP/1 connections.
HTTP/2 and VST connections that are sending data back to the client are now
closed after 300 seconds or the configured idle timeout (the higher of both
values is used here).
Before this change, the timeouts for HTTP/2 and VST connections were hard-
coded to 120 seconds, and even non-idle connections were closed after this
timeout.
* Added new metric `arangodb_network_forwarded_requests` to track the number
of requests forwarded from one coordinator to another in a load-balancing
context.
* Added new metrics for tracking AQL queries and slow queries:
* `arangodb_aql_query_time`: histogram with AQL query times distribution.
* `arangodb_aql_slow_query_time`: histogram with AQL slow query times
distribution.
* `arangodb_aql_all_query`: total number of all AQL queries.
* Added new metrics for replication:
* `arangodb_replication_dump_requests`: number of replication dump requests
made.
* `arangodb_replication_dump_bytes_received`: number of bytes received in
replication dump requests.
* `arangodb_replication_dump_documents`: number of documents received in
replication dump requests.
* `arangodb_replication_dump_request_time`: wait time for replication dump
requests.
* `arangodb_replication_dump_apply_time`: time required for applying data
from replication dump responses.
* `arangodb_replication_initial_sync_keys_requests`: number of replication
initial sync keys requests made.
* `arangodb_replication_initial_sync_docs_requests`: number of replication
initial sync docs requests made.
* `arangodb_replication_initial_sync_docs_requested`: number of documents
requested via replication initial sync requests.
* `arangodb_replication_initial_sync_docs_inserted`: number of documents
inserted by replication initial sync.
* `arangodb_replication_initial_sync_docs_removed`: number of documents
inserted by replication initial sync.
* `arangodb_replication_initial_chunks_requests_time`: wait time histogram
for replication key chunks determination requests.
* `arangodb_replication_initial_keys_requests_time`: wait time for replication
keys requests.
* `arangodb_replication_initial_docs_requests_time`: time needed to apply
replication docs data.
* `arangodb_replication_initial_insert_apply_time`: time needed to apply
replication initial sync insertions.
* `arangodb_replication_initial_remove_apply_time`: time needed to apply
replication initial sync removals.
* `arangodb_replication_initial_lookup_time`: time needed for replication
initial sync key lookups.
* `arangodb_replication_tailing_requests`: number of replication tailing
requests.
* `arangodb_replication_tailing_follow_tick_failures`: number of replication
tailing failures due to missing tick on leader.
* `arangodb_replication_tailing_markers`: number of replication tailing
markers processed.
* `arangodb_replication_tailing_documents`: number of replication tailing
document inserts/replaces processed.
* `arangodb_replication_tailing_removals`: number of replication tailing
document removals processed.
* `arangodb_replication_tailing_bytes_received`: number of bytes received
for replication tailing requests.
* `arangodb_replication_tailing_request_time`: wait time for replication
tailing requests.
* `arangodb_replication_tailing_apply_time`: time needed to apply replication
tailing markers.
* Allow calling of REST APIs `/_api/engine/stats`, GET `/_api/collection`,
GET `/_api/database/current` and GET `/_admin/metrics` on followers in active
failover deployments. This can help debugging and inspecting the follower.
* Added metrics for V8 contexts usage:
* `arangodb_v8_context_alive`: number of V8 contexts currently alive.
* `arangodb_v8_context_busy`: number of V8 contexts currently busy.
* `arangodb_v8_context_dirty`: number of V8 contexts currently dirty.
* `arangodb_v8_context_free`: number of V8 contexts currently free.
* `arangodb_v8_context_max`: maximum number of concurrent V8 contexts.
* `arangodb_v8_context_min`: minimum number of concurrent V8 contexts.
* Fix for issue BTS-183: added pending operations purging before ArangoSearch
index truncation
* Don't allow creation of smart satellite graphs or collections (i.e. using
`"isSmart":true` together with `"replicationFactor":"satellite"` when creating
graphs or collections. This combination of parameters makes no sense, so that
the server will now respond with "bad parameter" and an HTTP status code of
HTTP 400 ("Bad request").
* Fixed: More cases in AQL can now react to a query being killed, so reaction
time to query abortion is now shortened. This was a regression in comparison
to 3.6 series.
* Support projections on sub-attributes (e.g. `a.b.c`).
In previous versions of ArangoDB, projections were only supported on
top-level attributes. For example, in the query
FOR doc IN collection
RETURN doc.a.b