/
ChangeLog
2514 lines (2253 loc) · 105 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Meaning of stable tagnames
--------------------------
Example: mars0.1stable01:
0 = version of on-disk data structures
(only incremented when downgrades are impossible)
(not incremented on backwards-compatible upgrades)
1 = version of feature set
stable = feature set is never reduced during this series
01 = bugfix revision
Example: mars0.2beta2.3:
The general idea is as before.
"beta" means that new features are roughly tested
in the lab, but not in production, so there may be
some bugs.
Example: mars0.3alpha*:
Never use this for production. Only for (historic)
code inspection.
Release Conventions / Branches / Tagnames
-----------------------------------------
mars0.1 series (now EOL):
- Unstable tagnames: light0.1beta%d.%d (obsolete)
- Stable branch: mars0.1.y (obsolete)
- Stable tagnames: mars0.1stable%02d (obsolete)
mars0.1a series (stable):
New master branch. Now stable.
This branch is operational for several years on
several thousands of servers, and several petabytes
of data.
- Stable branch: mars0.1a.y
- Stable tagnames: mars0.1astable%02d
- Historic tagnames: light0.1abeta%d (obsolete)
mars0.5 series (planned interim series):
- The stable Linux kernel upstram series linux-5.10.y
and later will require new interfaces between MARS
and the kernel. For example, fs/aio.c is no
longer usable by kernelspace activities.
- I plan to take this opportunity for getting rid
of ~10 years old backward-compatibility parts,
interfacing with ancient kernels, e.g. from linux-2.*
to linux-3.x.y
- the branches mars0.1.y and mars.1a.y will become
unmaintained. Just use the standard branch "master"
as usual, since it will become the new alias for
the new mars0.5.y series.
- Newer MARS improvements like the newer .deleted format
are planned to become standard, and a few tons of
ancient code are planned to vanish over time.
Do not expect backwards compatibility over years of
development efforts. I won't test big jumps anymore.
- Consequence: you need a Linux kernel series
from linux-4.9.y onwards. The corresponding
historic MARS pre-patches will be removed from the
MARS git repo, starting with this branch.
The then-historic mars branches may be renamed to
*-historic in order to inform you.
mars1.0 series (planned future series):
- Replace the symlink tree by transactional status files
(future-proof)
This is required for upstream merging to the kernel.
It has further advantages, such as better scalability.
- Some migration strategy for the data format in /mars
will be certainly done, but details are not yet planned.
- Trying to additionally address public needs.
- Intended for Linux kernel upstream submission.
- See the FAQ: the out-of-tree MARS version in kernel/
shall become unmaintained after acceptance by
Linus Torvalds and the kernel community.
WIP-* branches are for development and may be rebased onto anything
at any time without notice. They will disappear eventually.
Never use them for production!
*stable* branches mean the following:
- Heavily tested. Has to obey an HA SLA of 99.98% end-to-end,
including network outages and HumanError(tm) at 1&1 Ionos
ShaHoLin. Thus the _component_ SLA of MARS must be much better.
- New features may be introduced, but are initially off by default.
They may become default after some operational experiences.
- There is always an upgrade path. Simply install the new
version, obeying the below compatibility rules.
- Rolling upgrades (temporarily different MARS kernel module
versions at primary vs secondary side) are supported, as
far as reasonable.
Typically, do "rmmod mars; modprobe mars" at the secondary
side first, then handover, then do the same at the former
primary side.
Or, of course, you may combine it with (typically security-
triggered) rolling kernel reboots.
I am putting high effort into maintaining rolling upgrades
of kernel modules. The network protocols are designed to
support this.
- COMPATIBILITY RULES:
Ensure that $marsadm_version >= $module_version.
This is the safe side of your update strategy.
Update marsadm first, before updating the kernel module.
This way, the controls for newer features are already in
place when the new kernel module is activated (no blind
flight).
Since marsadm is a plain Perl script with _no_ dependencies
from anything else, this is something I can reasonably expect
from users.
REASON: ensuring forever backwards compatibility to stone-aged
marsadm versions would make me ill. I cannot change old versions
anymore, but just provide new versions. I cannot ensure and
test all possible O(n^2) combinations of marsadm versions with
kernel module versions to work eternally for all times when
marsadm would be frozen, or even all O(n^3) combinations of
frozen marsadm with mixed-operations kernel modules.
The development of MARS would be hindered by too old marsadm
versions, since my effort would grow quadratically or
even worse.
Hint: nevertheless, many combinations of old marsadm with newer
kernel module version are working anyway, in particular when
the gap is a small $epsilon. But I cannot guarantee in general.
If you want to violate the above rule, you must test the
combination yourself.
- Best practice in bigger installations: first test your upgrade
or downgrade at some test clusters first.
If you have a separate pre-live stage, it definitely is
your friend.
- As long as $marsadm_version + $epsilon >= $module_version
remains true (at least "approximately") and has been tested
in pre-live, marsadm may be upgraded and downgraded
independently from kernel, and during operations
(best via your favorite package manager).
Of course, no magic will happen: newer features are only
available when newer versions of _both_ the userspace tool and
the kernel modules are installed.
- Please check this ChangeLog for any upgrade / downgrade
incompatibility problems. When bugs are detected, they will
be fixed. But I cannot retrospectivly change already released
versions and their bugs. Fixes are only possible in newer
versions.
- Downgrade is possible *inside* of the same stable branch
series, at least over about 20 minor releases. I cannot test
all possible O(n^2) downgrade combinations. Thus be careful when
downgrading over high version distances (which should not be
done anyway unless you have some very well-thought reasons).
- Downgrade to _prior_ *stable* branches, or over very big differences
in minor version number, may be restricted,
or may require some extaordinary actions.
Please read this ChangeLog for details.
Historic example: a new future-proof internal deletion format
had been introduced in mars0.1astable88. It was off by default.
But it will become default somewhen (e.g. during newer branches).
Only if you never had activated it, and if you had obeyed the
downgrade instructions as documented here, you might have
downgraded. I had invested a lot of time in testing, but
I don't know of anyone who actually did this in operations.
-----------------------------------
Changelog for series 0.1a:
This is the new master branch, starting January 2019.
The _historic_ stable branch mars0.1.y is EOL since a
long time, and had been forced to the current master.
INFO: upstream stable kernels v5.10.x and later will require
some new MARS-internal bricks and libraries, currently under
preparation.
Please stay patient; some old MARS bugs have been already found and
fixed during the preparation phase. I am releasing them
for the currently supported stable upstream kernels.
mars0.1astable174
* Minor fix: automatic restart hanging syncs.
* Minor fix: automatic restart hanging fetches.
* Minor improvement: adapt to newer pre-patch version.
mars0.1astable173
* Minor fix: do not spawn many server kthreads on
recpetion of masses of connections.
mars0.1astable172
* Major improvement: avoid some odd pointer addresses,
leading to unnecessary performance loss.
* Minor fix: give up the CPU more often during rmmod.
* Minor fixes: silence some compiler warnings.
* marsadm improvement: clarify warning on create-resource --force.
mars0.1astable171
* Minor fix: give up the CPU more often in a rare
corner case.
* Minor improvement: when the replication network is
interrupted at random places, the internal error propagation
is more finegrained. Typically not visible, but the reaction
should be smoother.
mars0.1astable170
(housekeeping release)
* marsadm update: also remove very old code, which may trigger
old-style symlink removals in rare corner cases (like
leave-resource --force). If you see many such symlinks
in /mars/defaults-* which are no longer processed by kernel
module version mars0.1astable169 or later, please update
marsadm to this version. It will force the new deletion
method everywhere.
mars0.1astable169
* Minor update: remove old code on obsolete deletion
method. When this was unnecessarily actived, it
consumed unnecessary resources.
mars0.1astable168
* Major fix: there was an unnecessary internal abort of
the copy brick. It was compenated by the higher layer,
but it could unnecessarily reduce the performance
of the replication.
mars0.1astable167
* Major fix: marsadm cron was not deleting versionlinks
in importnat cases. Regression from 70880946c.
Please update marsadm ASAP if you have a version after
mars0.1astable141.
Thanks to Ingvar Gilbert for reporting this bug!
mars0.1astable166
* Rework some parts of the copy brick.
This should make it more robust, and improve
performance.
mars0.1astable165
* Speedup client code.
* Minor safeguards.
* Minor improvements in debugging code.
mars0.1astable164
(bugfix release)
* Improved memory preallocation. This should reduce
the probability of order:5 allocation failures (if
you see them on very highly loaded servers).
* Improved detection of short-living sockets.
* Improved CPU consumption in an extremely rare
corner case.
* Minor code cleanups.
mars0.1astable163
(bugfix release)
* Improved robustness at log scanning.
* Some minor code cleanups.
* Typos fixed by Andrea Gelmini, many thanks!
mars0.1astable162
(bugfix release)
* Fix counting of server processes.
More to come.
mars0.1astable161
(bugfix release)
* Major fix: rmmod could hang when certain preconditions were met.
Rarely observed in thorough testing was an rmmod during sync.
Workaround was possible by anything closing the established state
of a leftover TCP conncetion, like rmmod at the other side.
Now socket shutdown is done by mars.ko.
* Major fix: modprobe could lead to an unfortunate initialization
order, potentially resulting in some kernel memory issues.
* Minor cleanups.
mars0.1astable160
* Minor non-functional improvements of client/server bricks
and their communicatiopn behaviour. Testing did not reveal
a functional difference. The only observation was that rmmod
could hang when the internal client/server communication got an
error-injected stuck at a certain internal phase, which AFAIK
should not occur in practice.
If you observe suchalike in operations, and you want to avoid
a reboot, then this release may be for you.
mars0.1astable159
* Major fix: the server part gives up the CPU usage much
more aggressively when it turns out that there is no clue
in burning CPU for performance.
This is extremely helpful for power saving, and particularly
in extreme situations like OOM.
* Minor compiler compatibilities.
mars0.1astable158
* Major fixes: rare races on copy brick internal state.
By the way, reduce internal state footprint and improve
performance.
* Improve trigger responsitivity.
* Minor safeguard: congestion control from various
upstream kernel versions.
* Minor compiler compatibilities.
mars0.1astable157
* Suppress annoying kernel stacktraces when RAM is short.
* Imvprove annoying KBuild failures.
* marsadm: hint on forgotten global-sync-limit-value settings.
mars0.1astable156
Critical fix for recovery after a crash, if it led to a certain
type of defective logfile.
Workaround was possible by manual creation of an empty next
logfile, such as "touch /mars/resource-$mydata/log-$next_nr-$host"
where $next_nr must have 9 digits like the other logfiles.
Please update to this version in place of manual workarounds.
mars0.1astable155
(housekeeping release)
Deprecate historic say logging.
It could have led to unnecessary overhead in bigger
distributed systems. Anyway, it was not necessary for
production.
So please do not activate CONFIG_MARS_DEBUG_DEVEL_VIA_SAY
unless you read the sourcecode.
mars0.1astable154
* Safeguard suspected SMP cache coherence problems.
Inspired by upstream commit d4252071b97d2027d246.
* marsadm: some minor improvements.
* Doc: new slides from FrOSCon2022.
mars0.1astable153
* Improved detection and mitigation of remote socket shutdown.
* Various other improvements.
mars0.1astable152
(bugfix release)
* Critical fix: under rare race conditions, the copy_thread
could stacktrace.
* Minor fix: marsadm resize could fail upon stray actsize
symlinks.
mars0.1astable151
(bugfix release)
* Critical fix: under rare OOM conditions, an allocation failure
in the kernel bio playground could spread to MARS, leading
to hanging IO and to hanging rmmod.
mars0.1astable150
(bugfix release)
* Critical fix: under OOM conditions, an allocation failure
in the kernel bio playground could spread to MARS, leading
to a variety of consecutive errors.
mars0.1astable149
(bugfix release)
* Critical improvement: reduce damages caused by OOM.
MARS will now refrain from high-frequency retries to
allocate vital memory in OOM situations, giving
servers with hundreds of CPUs a better chance to survive.
HIGHLY RECOMMENDED!
* Major improvement: improve latencies caused by RAM
shortages.
mars0.1astable148
* Major fix: under certain types of overloads during
shutdown of /dev/mars/mydata, a kernel stacktrace was
possible under further conditions.
* Reduce unnecessary network connection aborts, leading
to unnecessary retries.
* Serveral smaller fixes and improvements.
* Doc: describe new ERROR codes and new reports
like IncompleteLog or UnInitializedLogRecord.
mars0.1astable147
* Several fixes, detected during testing
a future replacement of the historic aio brick.
Some of these generic fixes and improvements may be also
beneficial for the current MARS release.
mars0.1astable146
(bugfix release)
* While testing with newer kernels and their better
POISONing variants (thanks to kernel upstream), extremely
rare trilemma-like races detected and hopefully fixed now.
mars0.1astable145
* Major fix: races between network data transport and local IO,
only observed during preparation of compatibility with
newer kernels. Depending on .config and on further rare
scenarios, it might also occur in combination with
currently supported kernels and/or some Frankenstein
kernels.
mars0.1astable144
(IMPORTANT bugfix release)
* CRITICAL FIX: network interruption or packet loss / scrambled
IP packets during certain transfers of strings could have led
to shortened in-kernel strings without trailing NUL bytes.
In turn, some internal error paths could have led to nasty
consecutive kernel crashes, right at the wrong moment from a
sysadmin's perspective.
Although this is extremely unlikely during ordinary operations,
and although the only "use case" known by me is around
{join,merge,split,leave}-cluster and {join,leave}-resource,
please UPDATE mars.ko for maximum safety.
- I am unsure whether there might be some more "use cases".
- Even if you think it won't hurt you: remember Murphy's law.
- Do not forget reboot || (rmmod mars && modprobe mars)
* Critical safeguard: natural races in copy_end_io() now compensated
in a better way. Probably also relevant for operations.
Another argument for upgrading mars.ko to this version ASAP.
mars0.1astable143
* Minor marsadm improvements.
* Doc: new section on Trilemma and CAP theorem.
mars0.1astable142
* Minor improvements, in particular for more than 2
replica per resource.
* Doc: explain active-active.
mars0.1astable141
* Major marsadm fix: network-wide race between
log-rotate / cron and invalidate / join-resource / etc.
* Major fixes, only relevant for people who need
changes in cluster setup, like {marge,split,join}-cluster after
leave-resource + leave-cluster & co.
More will hopefully follow (currently unclear and not
yet a raw timeline).
* Minor marsadm improvements: usability, reporting
improvements, etc.
* Minor doc improvements, clarifications. New FAQ item
on Business Value of OpenSource.
mars0.1astable140
* Critical doc improvement: try to make CRYSTAL CLEAR
why MISCONFIGURED hostnames can cause DATA LOSS.
In short: hostnames must not contain anything from FQDN.
* Major doc improvement: cite the NO WARRANTY clause
from the GPL several times, once again.
* Minor improvement: new primitive
marsadm view-total-space in units in GiB.
* More minor doc updates.
* New FAQ appendix, from the inventor of MARS.
mars0.1astable139
* Critical improvement:
Certain types of defective logfiles might have led to
false-positive or false-negative CRC checking
(under more or less obscure conditions).
For safety, update is recommended.
mars0.1astable138
* Critical fix: scarce list corruption could lead to
kernel stacktraces, requiring a reboot.
It was not reproducible, only observed at certain hardware
classes and after tenthousands of operating hours under
certain load conditions.
Please update when running mixed-role clusters.
* Various marsadm improvements and fixes.
* Corresponding doc updates.
* Architecture guide: more scalability examples.
mars0.1astable137
* Major fix: scarce race with network problems could
have led to a memory leak.
* Minor fix: an externally produced race between modprobe
and mount /mars could cause strange problems.
Improved reporting.
* Minor improvement: new option --autoclean removing
some historical remains via cron.
* Some more minor fixes and improvements, e.g. better
checking of illegal marsadm commandline arguments.
* Doc the new --autoclean feature.
* Improve architecture guide on "scalability".
mars0.1astable136
* Major fix: upon kernel-level fork() limits, or OOM etc,
it was possible that aio went into an endless loop
consuming much CPU.
Probably, the system survival rate of OOM-like problems is now
also better, although miracles cannot be generally expected
in suchalike operational states (independently from MARS).
Simply try to prevent OOM via ordinary sysadmin measures /
best practices like reasonable RLIMITS & co in userspace.
mars0.1astable135
* Minor fix: extremely scarce race on rmmod, was
practically impossible to reproduce.
* Minor improvement: speedup rmmod.
* Doc: updates of architecture-guide-geo-redundancy.pdf
* Doc: update user manual appendix B.
mars0.1astable134
* Minor fix: the marsadm --force argument was not fully independent
from other arguments.
* Minor usability: more clear warning when the cluster IP config
looks suspicious.
* Doc: examples of actual geo disasters.
mars0.1astable133
* Major fix: certain types of primary crashes could lead to
unnecessary replication hangs at the secondaries (but no problems
at the primary side), when just waiting for reboot and running
_no_ failover.
As always, "marsadm invalidate" could workaround it.
Now there is some automatic self-repair of versionlinks,
correcting both primary and secondary sides.
This bug was extremely hard to reproduce; ordinary "standard"
crashes could not trigger it.
* Further doc update: the architecture guide is now called
architecture-guide-geo-redundancy.{lyx,pdf} (but the old name
remains accessible via symlink).
mars0.1astable132
* Major doc update of architecture guide.
More improvements like splits for various interests are certainly
needed, but currently not in focus due to limited time.
mars0.1astable131
* Major fix: long-standing race condition on aspect {de,}allocation.
Was triggering extremely rarely and was very hard to
reproduce, but it could lead to rare stacktraces and
to rare kernel hangs.
I am not 100% sure to have fixed it fully, but massive testing
over a long time tells me that it has _at least_ improved.
Further stresstest improvements after safeguarding potentially
misleading callbacks from aio & co over shared kernel files,
where their shared pagecache cannot distinguish different callers.
Possibly more improvements might appear in future releases.
This will take a lot of time for extreme stress-testing.
* Minor safeguard: better safeguard of indirect calls via mb().
Theoretically, this should be unnecessary. But I saw some
extremely rare effects (only at a certain hardware class),
so I prefer stability over maximum performance.
mars0.1astable130
* Minor improvement: marsadm now compensates race between
emergency mode removal and invalidate.
mars0.1astable129
* Painful regression from mars0.1astable128: rmmod mars (and possibly
some other situations) could hang forever, requiring reboot.
Please upgrade ASAP in case you already have deployed
exactly this version. Do not use it anymore.
* Minor fix: when the new prepatch series v2 was used
(LTS kernels 4.19 and 5.4), "modinfo mars" was incorrectly
reporting "no_prepatch". Now the report tells you the
prepatch series (v1 versus v2).
mars0.1astable128 (do not use anymore)
* Critical fix: inherent races between automatic network re-submission
and completion are compensated now.
Likely, these races could have been the reason for very rare
stack traces.
mars0.1astable127
* Major fixes / safeguards: very rare stacktraces in copy_endio()
and client_io() are hopefully better addressed now.
I have no reproduced at the moment.
* Various minor fixes and improvements.
mars0.1astable126
* Critical regression from mars0.1astable125: uninitialized
pointer could lead to (rare) kernel oops. Not observed in
practice until now. Please upgrade for safety.
* Major fix, only relevant when logfile compression was enabled:
In certain corner cases, compressed logfiles were not always
decompressible, as indicated by DefectiveLog error messages.
Workaround was possible by switching compression off, cron,
and invalidate.
* Minor compat fix: very old kernels missing WRITE_ONCE.
* Minor cosmetics: benchmark results now priority KERN_INFO.
mars0.1astable125
* Critical fixes: over a very long time, internal int counters
could wrap around into negative numbers and cause kernel Oops.
* Critical safeguard: about once per 1 million of operation hours,
a stacktrace was observed in copy_endio().
At the moment, I have no reproducer for the very spurious
bug. Hopefully it is fixed now.
* Minor improvement: alternation between sync and replay
now avoids unnecessary waiting.
mars0.1astable124
* Major improvement: support for LTS kernels 4.19 and 5.4.
A new pre-patch generation obeying the new ksys_* conventions
has been added. This will help further porting in the future.
IMPORTANT: do NOT OMIT the fix for upstream bug
0001-sched-wait-fix-endless-kthread-loop-at-timeout.patch
from directoy pre-patches/vanilla-*/ .
Leaving out this fix may SERIOUSLY HARM your experience,
due to kernel soft lockups happening when the network
is interrupted, i.e. exactly during certain types of incidents.
Anyway, please also use 0001-mars-v2-minimum-pre-patch-for-mars.patch
because IO performance is MUCH WORSE without this pre-patch.
mars0.1astable123
* Major fix: under very unlikely conditions, deadlock
of the logger IO scheduling was possible.
Never observed during millions of total operation hours.
Please update for maximum HA safety.
* Minor fix: scarce divide by zero in IO or network throughput
limiters (normally not used) was possible under very
special cirmumstances.
mars0.1astable122
* Minor improvement (possibly a regression from 0.1astable116),
observed in a very tricky situation where _both_ the primary
and secondary RAIDs were heavily degraded at the same time:
The replay could take too much preference over sync, leading
to quasi-starvation of sync.
Workaround was possible by temporarily setting
/proc/sys/mars/sync_flip_interval_sec to 0, and manually
switching between replay and sync.
Now awful RAID degradation should be handled more gracefully.
* Minor improvement, stimulated by Gabriel Franciso:
Before marsadm asks DNS, first /etc/hosts is consulted
via /usr/bin/getent.
mars0.1astable121
* Fix scarce use-after-free, only observed at rmmod operations
under KASAN test kernel 4.14.
For maximum safety, please update to this version.
mars0.1astable120
* Fix build with LTS kernel 4.14.
IMPORTANT: you need to patch your 4.14 kernel sources with
pre-patches/vanilla-4.14/0001-sched-wait-fix-endless-kthread-loop-at-timeout.patch
Otherwise you will encounter _massive_ problems!
mars0.1astable119
* Major systemd update, only relevant when you are using the
systemd template generator:
- Now works completely lockless. This should improve
the parallelism degree and reduce the risk of deadlocks.
- Now uses per-resource triggers for parallel incremental updates
after {create,join,leave}-resource etc.
Attention: checkout the new templates in systemd-testing/
(the old ones will no longer work).
- New pseudo unit type .script : in place of being interpreted
by systemd, you may now write some (wrapper) scripts for
more complex operations, and/or for achieving idempotence.
Although I would like to prefer native systemd units, I
added this feature after I became desperate when trying to
achieve true idempotence via native systemd units. After several
months of fruitless attempts, I gave up and added .script .
For details, please read the new docs.
An example of .script can be found in the new systemd-icpu/ .
Notice that "nodeagent" is a third-party tool which in turn
may call systemctl for startup of LXC containers (after
contacting a database, and a plethora of other things).
Any attempts to call nodeagent via ExecStart= and co did
not really work as it should.
- Some new template engine features, like markers DEFAULT_START
and much more.
* Updated docs on systemd.
* No other changes outside of the systemd area.
mars0.1astable118
* Critical fix, only relevant for trial builds without
pre-patch: kernel NULL deref, fixed by Gabriel Francisco.
I am releasing this alone because the next release will
take some more time.
mars0.1astable117
* Minor fix, only relevant for k > 2 replicas:
invalidate and log-purge-all could abort unnecessarily due to races.
Workaround was possible by retrying the command.
* Minor usability: new commands {de,}activate-guest for
enabling or getting rid of temporary guests. Only relevant for
clusters with > 2 members.
* Some minor fixes and improvements.
* Minor doc update (new commands).
* Further dkms improvements / tuning from Gabriel Francisco.
mars0.1astable116
* Critical fix, only relevant for cluster naming schemes where hostname A
may be a _prefix_ of hostname B: such naming schemes could have led to
a multitude of bizarre and unexplainable confusions and problems.
Example: hostnames icpu-bs6 and icpu-bs60 .
Please UPDATE when such hostnames may occur in the _same_ cluster.
I was unable to find the bug via the test suite because such hostnames
were not deployed at the test machines. Thanks to Stephan Christiany
who pointed me at the problem.
* Fix annoying bug: during long-lasting sync (several TB), the automatic
flipping between replay and sync could sometimes get stuck in sync mode,
and then /mars could fill up because replay was starving unnecessarily
(waiting that sync would finish, which could take a long time).
Workaround was possible by "pause-sync"; wait until replay has
caught up; "resume-sync".
mars0.1astable115
* Critical regression from mars0.1astable113 / 114, only relevant
when the new ssh-less peer operations are actually used:
Race on peer thread creation could lead to kernel memory corruption.
For maximum safety, please avoid the affected kernel module versions.
* dkms improvements from Gabriel Francisco.
mars0.1astable114
* Major usability: ssh-less {merge,split}-cluster.
Now all cluster operations should work without ssh and
its agent forwarding.
Of course, you will need to update mars.ko and marsadm
on all of your machines first.
* Doc update: describe new options and behaviour.
* Some smaller fixes / safeguards / improvements.
mars0.1astable113
* Critical fix: deadlock was possible after receiving _corrupted_
data over the network. Very unlikely to trigger, since there
is a lot of other magic checking, but anyway.
Now treated like any other communication error.
* Major improvement: "marsadm primary" (also with --force) now does
the equivalent of "up", after the operation has succeeded.
This should be useful for people who forget to do the "up" manually
after a manual unplanned failover.
* Minor fix: race at {wait,update}-cluster leading to unnecessary
abort.
* Minor fix: update-cluster did not always transfer directories.
* Minor fix: new join-cluster method could sometimes fail
at the first try. Workaround by repetition.
* Minor fix: primitive macros wait-todo-primary-{on,off} were
documented, but not implemented.
* Minor improvement: by the way, all missing combinations from
{is,nr,todo}-secondary and
wait-{is,todo}-{primary,secondary}-{on,off} are also implemented.
* Minor improvement: try to automatically fetch any unknown
peer info. May help after failed join-cluster & co.
* Minor improvement: speedup new join-cluster method.
* Minor doc update: describe new primitives.
* Some smaller fixes and improvements.
mars0.1astable112
* Critical fix: generic mars_readlink() did not work with an
extremely low probability, so it slipped through years
of testing. My reproducer indicates that it "fixed" itself
after a while, just leading to some unnecessary delays.
Nevertheless, I mark it "critical" under a HA viewpoint,
although most people likely might have never noticed it.
Recommendation: please update.
* Major fix, only relevant for k > 2 replica:
fetch could get stuck in cyclic dependencies for some
time, making only slow progress.
* Major fix: join-resource could loop when
old method is selected and ssh was not working.
* Minor fix: do not produce alive-timestamp & co on a fresh
/mars, before {create,join}-cluster has been executed.
* Several smaller fixes and improvements.
mars0.1astable111
* Minor fix, only relevant for new deletions:
Split-brain cleanup was sometimes stumbling over
deleted logfiles. Workaround by cron.
mars0.1astable110
* Minor improvement: new disk-error for better diagnosing
any problems with disk setup / LVM etc.
* Doc update (new macros etc).
mars0.1astable109
* Regression from mars0.1astable106: when the old
deletions were active, logfiles could be unlinked
unnecessarily (displayed as Orphan).
It did not really harm due to automatic re-fetching, but
caused unnecessary network traffic.
mars0.1astable108
* Improved metadata scalability.
* Some smaller fixes and improvements.
mars0.1astable107
* Critical regression from mars0.1astable106: use-after-free.
* Fix use-after-free at rmmmod.
mars0.1astable106
* Major regression from mars0.1astable97: marsadm primitive
disk-present erronously reported the disk name in place of
boolean value 0 or 1.
* Minor fix for new deletions (beta):
invalidate / re- join-resource were sometimes hanging
in Orphan due to a conflict with the new deletions.
* Minor improvements: somewhat more improved scalability both
in #resources and in #hosts.
mars0.1astable105
* Minor marsadm regression from mars0.1astable104: race on
_old_ deletions could lead to lost deletions. Workaround
by repeating any affected commands, e.g. leave-resource.
mars0.1astable104
* Major fix: marsadm did not obey an abort of certain phased
commands when a single resource argument was given. As a result,
a wrong exit code could be returned in such a case.
* Minor fix: when beta feature logfile digests were disabled
_during_ operations, already existing old logfiles were
not always checked correctly at the secondary,
reporting DefectiveLog (although they were healthy).
Workaround by just enabling again and invalidate.
With the fix, you may now replay the old logfiles :)
* Minor fix: inherent race between join-resource and log-rotate
(unavoidable in the Distributed System) could lead to split brain,
or to hanging replay. Now compensated.
* Minor fix: join-cluster without ssh was sometimes not
updating the local link tree immediately.
* Usability (BETA feature): improved scalability in #hosts.
The below BETA feature warnings apply.
Do not exceed the "officially documented" limits too much.
* Usability: join-resource avoids unnecessary fallback
to ssh / rsync.
IMPORTANT: please update marsadm first, before updating the
kernel module. See the above compatibility rules.
This time the compatibility rules are important. I know that
marsadm < 0.1astable85 does no reliable join-resource anymore,
while combinations with old 0.1astable95 appear to work. There is
no merit in bisecting old marsadm releases, instead of just
fucking update the old userspace script in a controlled manner.
* Usability: more accurate IOPS and friends.
* Several smaller fixes and improvements.
mars0.1astable103
* Major regression from mars0.1astable99: secondary replay could
hang unnecessarily due to a cascade of race conditions.
AFAICS consistency was not affected (thanks to md5 checksumming).
Observed with a specific load pattern at less than 1% of resources,
or in average after ~ 120 operation hours when logrotate
was 12 times per hour. Unfortunately, it slipped through all my
release tests due to relatively low trigger probability.
Workaround by "invalidate". Which is however no good solution.
Please avoid kernel module versions between *99 and *102
for production.
mars0.1astable102
* Major usability (BETA): scalability in number of hosts.
It should have no visible side effect in functionality,
but better non-functional properties.
Tested in the _lab_ with 1000 additional dummy hosts
and additionally 8000 dummy resources in total.
BETA WARNING: at the moment, there are no practical experiences.
There might be problems which might not show up during lab tests.
Do not blindly rollout or merge-cluster big masses in production.
I will tell you when practical experiences allow for rising
the "official" limits as documented in the user manual.
mars0.1astable101
* Major usability: join-cluster now works without ssh.
Of course, you need to rollout the new marsadm and
the new mars.ko first, and to modprobe it at any
pre-existing cluster.
The new feature is automatically activated when you
modprobe _before_ doing join-cluster. By running
join-cluster first (without modprobe), you can fallback
to the old ssh + rsync based method.
Important: now you can modprobe before /mars/uuid is
created or retrieved. Previously, you could accidentally
try the wrong sequence "modprobe mars; mount /mars"
without harm because it was denied by missing uuid, but now
such illegal attempts would result in a big fuckup.
Suchalike fuckup is now prevented by always insisting on
/mars being a mountpoint.
This might break old ill-behaved scripts or buggy /etc/fstab
or racy systemd dependencies, which need to be fixed.
Always ensure that no modprobe is attempted before /mars
has been mounted in a race-free and reboot-safe manner.
Notice: merge-cluster and split-cluster are not yet
ssh-free zones. This will be addressed in a later release.
* Minor usability: show age of any hanging /dev/mars/
IO requests. This is useful for diagnosing faulty RAID
controllers etc.
* Lots of further minor fixes and improvements.
mars0.1astable100
* Minor fix: UpToDate was not reported in a very weird
corner case.
* Minor fix, only relevant when the new deletion method
is enabled: leave-resource did sometimes not delete
all superfluous logfiles at the other peers, sometimes
not clearing a split brain situation immediately.
Workaround by cron which did the cleanup later.
* Minor usability: reduced speakiness of "marsadm view all"
with respect to the new compression / digest features.
Full info can be obtained with --verbose.
* Minor fix, only observed at join-cluster without ssh:
Not all symlink infos were transferred in a corner case.
* Further minor fixes and improvements.
mars0.1astable99
* Minor fixes: some more corner cases of unnecessary
split brain rarely occuring after fatal primary
crashes.
mars0.1astable98
* Minor regression from mars0.1astable97: when old
kernel modules < mars0.1astable97 were combined with
exactly that marsadm version, the presence of
/dev/mars/$resource was detected incorrectly.
Do not use exactly that combination. Simply skip
the marsadm version mars0.1astable97.
Other version combinations are still possible for independent
and rolling updates of kernel and marsadm.
Best practice: first update marsadm to mars0.1astable98
or newer, so this bug is fixed, and then your rolling
kernel updates will work again for updating or even
downgrading old kernels.
* Minor fix: in a hardly reachable corner case, detach
was hanging. Workaround by rmmod was possible.
* Minor fix: spurious races at join-resource without ssh could
occur, so it sometimes did not notice that a new resource
was added in the meantime. Usage of ssh, or just retrying
was helpful. Thus hardly relevant in practice.
* Various minor fixes and improvements. Some masked bugs,
not visible, only triggerable by a future version of MARS.
mars0.1astable97
* Critical fix: when logfile is damaged (e.g. after a
primary crash), some corner cases of primary recovery
could hang. Workaround by "detach ; attach" seemed
possible (as far as observed during testing).
* Critical fix for BETA feature network compression only:
Memory deallocation could fail under certain circumstances,
resulting in a memory leak, or potentially memory corruption.
Only relevant when network transport compression is enabled.
* Major fix: when a primary crash was occuring exactly during
a very short log-rotate time window, a race condition could
sometimes lead to unnecessary split brain (secondaries could
bypass the primary).
* Several minor fixes and improvements.
mars0.1astable96
* Minor improvement: auto-correct defective symlink
timestamps which are too far in the future.
This can happen when running with a defective CMOS
hardware clock, e.g. after a fatal hardware failure, and
before ntpd has corrected the local clock.
* Minor usability: more pretty formatting of compression
and digest flags in "marsadm view".
mars0.1astable95
* Minor fix: sometimes, in a hardly relevant corner case,
join-resource could abort unnecessarily.
* Minor improvement: marsadm view now distinguishes role ForcedPrimary
from plain Primary. This could help a larger team of sysadmins
earlier noticing potentially upcoming SplitBrain even while the
network is interrupted, so any actual SplitBrain cannot be
detected, although it is suspectible.
* Reduce footprint of some deprecated marsadm functions
and macros.
mars0.1astable94
* Major regression from mars0.1astable86:
Memory leak in remote communication.
This could accumulate over a longer time. Please update when
affected.
mars0.1astable93
* Minor improvement: in some special cases, secondaries
may now follow primaries having a damaged logfile.
mars0.1astable92
* Major improvement from an operational perspective:
"marsadm view all" now reports the current status of
/dev/mars/mydata in human-readable form, including
the Open status, the current IOPS, the number of currently
flying IO requests = IO queue length = indicator for IO problems
or overload, and any error information.
mars0.1astable91
* Major features, disabled by default:
- Network transport compression.
May improve network bottlenecks.
- Transaction logfile payload compression.
May improve the filling speed of /mars.
* Major feature, enabled by default:
- More logfile checksumming digests, some
consuming less CPU.
* Rough benchmarks, supporting you activation decisions.
Please read mars-user-manual.pdf for instructions.
Rolling updates with mixed versions are supported.
mars0.1astable90
* Minor improvement: more reactiveness. This release
is meant as an anchor point in case you would need
a downgrade.
mars0.1astable89
* Minor improvement: better kernel module reactiveness.
More on scalability is in the dev pipeline.
For now, use marsadm --timeout=300 or similar when
stretching the official limits (but don't stretch too
much until I have improved all relevant parts).
mars0.1astable88
* New experimental scalability feature, deactivated
by default:
New deletion method, uses the special symlink value