-
Notifications
You must be signed in to change notification settings - Fork 0
/
docker-ozones3-ozones3-s3-scm.log
7833 lines (7833 loc) · 862 KB
/
docker-ozones3-ozones3-s3-scm.log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Attaching to ozones3_s3g_1, ozones3_datanode_3, ozones3_datanode_2, ozones3_datanode_1, ozones3_om_1, ozones3_scm_1
s3g_1 | 2019-11-16 00:36:43,640 INFO hdfs.DFSUtil: Starting Web-server for s3gateway at: http://0.0.0.0:9878
s3g_1 | 2019-11-16 00:36:43,665 INFO util.log: Logging initialized @1298ms
s3g_1 | 2019-11-16 00:36:43,765 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
s3g_1 | 2019-11-16 00:36:43,801 INFO http.HttpRequestLog: Http request log for http.requests.s3gateway is not defined
s3g_1 | 2019-11-16 00:36:43,811 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
s3g_1 | 2019-11-16 00:36:43,815 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context s3gateway
s3g_1 | 2019-11-16 00:36:43,815 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
s3g_1 | 2019-11-16 00:36:43,816 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
s3g_1 | 2019-11-16 00:36:43,837 INFO s3.Gateway: Starting Ozone S3 gateway
s3g_1 | 2019-11-16 00:36:43,846 INFO http.HttpServer2: Jetty bound to port 9878
datanode_2 | 2019-11-16 00:36:42,463 INFO ozone.HddsDatanodeService: STARTUP_MSG:
datanode_3 | 2019-11-16 00:36:42,293 INFO ozone.HddsDatanodeService: STARTUP_MSG:
datanode_2 | /************************************************************
s3g_1 | 2019-11-16 00:36:43,848 INFO server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732
datanode_3 | /************************************************************
datanode_2 | STARTUP_MSG: Starting HddsDatanodeService
om_1 | 2019-11-16 00:36:41,999 INFO om.OzoneManagerStarter: STARTUP_MSG:
s3g_1 | 2019-11-16 00:36:43,882 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@1b919693{/logs,file:///var/log/hadoop/,AVAILABLE}
datanode_1 | 2019-11-16 00:36:41,912 INFO ozone.HddsDatanodeService: STARTUP_MSG:
datanode_3 | STARTUP_MSG: Starting HddsDatanodeService
datanode_2 | STARTUP_MSG: host = ab54c8aa3abc/172.19.0.6
om_1 | /************************************************************
s3g_1 | 2019-11-16 00:36:43,883 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@6933b6c6{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-ozone-s3gateway-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
datanode_1 | /************************************************************
datanode_3 | STARTUP_MSG: host = 70c5f52b4175/172.19.0.5
datanode_2 | STARTUP_MSG: args = []
s3g_1 | ERROR StatusLogger No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 internal initialization logging. See https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions on how to configure Log4j 2
om_1 | STARTUP_MSG: Starting OzoneManager
datanode_3 | STARTUP_MSG: args = []
datanode_1 | STARTUP_MSG: Starting HddsDatanodeService
datanode_2 | STARTUP_MSG: version = 3.2.0
scm_1 | 2019-11-16 00:36:41,923 INFO server.StorageContainerManagerStarter: STARTUP_MSG:
s3g_1 | WARNING: An illegal reflective access operation has occurred
om_1 | STARTUP_MSG: host = d45a264bff6a/172.19.0.3
datanode_3 | STARTUP_MSG: version = 3.2.0
datanode_1 | STARTUP_MSG: host = c29ed315f282/172.19.0.4
datanode_2 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/jaxb-impl-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/jaxb-api-2.3.0.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/activation-1.1.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jaxb-core-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-ozone-datanode-0.5.0-SNAPSHOT.jar
scm_1 | /************************************************************
s3g_1 | WARNING: Illegal reflective access by org.jboss.classfilewriter.ClassFile$1 (file:/opt/hadoop/share/ozone/lib/weld-servlet-2.4.7.Final.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
om_1 | STARTUP_MSG: args = [--init]
datanode_3 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/jaxb-impl-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/jaxb-api-2.3.0.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/activation-1.1.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jaxb-core-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-ozone-datanode-0.5.0-SNAPSHOT.jar
datanode_1 | STARTUP_MSG: args = []
datanode_2 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z
scm_1 | STARTUP_MSG: Starting StorageContainerManager
s3g_1 | WARNING: Please consider reporting this to the maintainers of org.jboss.classfilewriter.ClassFile$1
om_1 | STARTUP_MSG: version = 3.2.0
datanode_3 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z
datanode_1 | STARTUP_MSG: version = 3.2.0
datanode_2 | STARTUP_MSG: java = 11.0.3
scm_1 | STARTUP_MSG: host = 897320affd68/172.19.0.2
s3g_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
om_1 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/sqlite-jdbc-3.25.2.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/jaxb-impl-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/hadoop-ozone-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-tools-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/hadoop-ozone-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-docs-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-ozone-ozone-manager-0.5.0-SNAPSHOT.jar
datanode_3 | STARTUP_MSG: java = 11.0.3
datanode_2 | ************************************************************/
datanode_1 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/jaxb-impl-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/jaxb-api-2.3.0.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/activation-1.1.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jaxb-core-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-ozone-datanode-0.5.0-SNAPSHOT.jar
s3g_1 | WARNING: All illegal access operations will be denied in a future release
scm_1 | STARTUP_MSG: args = [--init]
om_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z
datanode_2 | 2019-11-16 00:36:42,474 INFO ozone.HddsDatanodeService: registered UNIX signal handlers for [TERM, HUP, INT]
datanode_3 | ************************************************************/
datanode_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z
s3g_1 | Nov 16, 2019 12:36:46 AM org.glassfish.jersey.internal.Errors logErrors
scm_1 | STARTUP_MSG: version = 3.2.0
om_1 | STARTUP_MSG: java = 11.0.3
datanode_3 | 2019-11-16 00:36:42,299 INFO ozone.HddsDatanodeService: registered UNIX signal handlers for [TERM, HUP, INT]
datanode_1 | STARTUP_MSG: java = 11.0.3
s3g_1 | WARNING: The following warnings have been detected: WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.get(java.lang.String,java.lang.String,java.lang.String,int,java.lang.String,java.io.InputStream) throws java.io.IOException,org.apache.hadoop.ozone.s3.exception.OS3Exception, should not consume any entity.
scm_1 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/hamcrest-all-1.3.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-docs-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-scm-0.5.0-SNAPSHOT.jar
om_1 | ************************************************************/
datanode_2 | 2019-11-16 00:36:42,777 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
datanode_1 | ************************************************************/
datanode_3 | 2019-11-16 00:36:42,613 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
s3g_1 |
scm_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z
om_1 | 2019-11-16 00:36:42,005 INFO om.OzoneManagerStarter: registered UNIX signal handlers for [TERM, HUP, INT]
datanode_1 | 2019-11-16 00:36:41,928 INFO ozone.HddsDatanodeService: registered UNIX signal handlers for [TERM, HUP, INT]
datanode_2 | 2019-11-16 00:36:42,978 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
s3g_1 | 2019-11-16 00:36:46,568 INFO handler.ContextHandler: Started o.e.j.w.WebAppContext@5382184b{/,file:///tmp/jetty-0.0.0.0-9878-s3gateway-_-any-3695870932811324332.dir/webapp/,AVAILABLE}{/s3gateway}
scm_1 | STARTUP_MSG: java = 11.0.3
datanode_3 | 2019-11-16 00:36:42,803 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
om_1 | 2019-11-16 00:36:42,986 INFO ha.OMHANodeDetails: Configuration either no ozone.om.address set. Falling back to the default OM address om/172.19.0.3:9862
datanode_2 | 2019-11-16 00:36:42,983 INFO impl.MetricsSystemImpl: HddsDatanode metrics system started
datanode_1 | 2019-11-16 00:36:42,222 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
s3g_1 | 2019-11-16 00:36:46,574 INFO server.AbstractConnector: Started ServerConnector@5d7911d5{HTTP/1.1,[http/1.1]}{0.0.0.0:9878}
scm_1 | ************************************************************/
datanode_3 | 2019-11-16 00:36:42,803 INFO impl.MetricsSystemImpl: HddsDatanode metrics system started
om_1 | 2019-11-16 00:36:42,986 INFO ha.OMHANodeDetails: OM Service ID is not set. Setting it to the default ID: omServiceIdDefault
datanode_2 | 2019-11-16 00:36:43,270 INFO ozone.HddsDatanodeService: HddsDatanodeService host:ab54c8aa3abc ip:172.19.0.6
datanode_1 | 2019-11-16 00:36:42,393 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
s3g_1 | 2019-11-16 00:36:46,574 INFO server.Server: Started @4209ms
scm_1 | 2019-11-16 00:36:41,932 INFO server.StorageContainerManagerStarter: registered UNIX signal handlers for [TERM, HUP, INT]
datanode_3 | 2019-11-16 00:36:43,005 INFO ozone.HddsDatanodeService: HddsDatanodeService host:70c5f52b4175 ip:172.19.0.5
datanode_2 | 2019-11-16 00:36:43,320 INFO volume.HddsVolume: Creating Volume: /data/hdds/hdds of storage type : DISK and capacity : 10908285698048
om_1 | 2019-11-16 00:36:42,993 WARN server.ServerUtils: ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead.
datanode_1 | 2019-11-16 00:36:42,394 INFO impl.MetricsSystemImpl: HddsDatanode metrics system started
s3g_1 | 2019-11-16 00:36:46,577 INFO server.BaseHttpServer: HTTP server of S3GATEWAY is listening at http://0.0.0.0:9878
scm_1 | 2019-11-16 00:36:41,980 WARN server.ServerUtils: ozone.scm.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead.
datanode_3 | 2019-11-16 00:36:43,063 INFO volume.HddsVolume: Creating Volume: /data/hdds/hdds of storage type : DISK and capacity : 10908285698048
datanode_2 | 2019-11-16 00:36:43,322 INFO volume.VolumeSet: Added Volume : /data/hdds/hdds to VolumeSet
datanode_1 | 2019-11-16 00:36:42,570 INFO ozone.HddsDatanodeService: HddsDatanodeService host:c29ed315f282 ip:172.19.0.4
om_1 | 2019-11-16 00:36:44,323 INFO ipc.Client: Retrying connect to server: scm/172.19.0.2:9863. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
scm_1 | SCM initialization succeeded.Current cluster id for sd=/data/metadata/scm;cid=CID-e0e3fa81-dff6-4fc7-aeb4-80e03811a63e
s3g_1 | 2019-11-16 00:38:09,400 INFO endpoint.BucketEndpoint: Location is /bucket-87438
datanode_3 | 2019-11-16 00:36:43,065 INFO volume.VolumeSet: Added Volume : /data/hdds/hdds to VolumeSet
datanode_2 | 2019-11-16 00:36:43,332 INFO volume.ThrottledAsyncChecker: Scheduling a check for /data/hdds/hdds
datanode_1 | 2019-11-16 00:36:42,636 INFO volume.HddsVolume: Creating Volume: /data/hdds/hdds of storage type : DISK and capacity : 10908285698048
om_1 | 2019-11-16 00:36:45,325 INFO ipc.Client: Retrying connect to server: scm/172.19.0.2:9863. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
scm_1 | 2019-11-16 00:36:42,055 INFO server.StorageContainerManagerStarter: SHUTDOWN_MSG:
s3g_1 | 2019-11-16 00:38:14,071 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
datanode_3 | 2019-11-16 00:36:43,072 INFO volume.ThrottledAsyncChecker: Scheduling a check for /data/hdds/hdds
datanode_2 | 2019-11-16 00:36:43,355 INFO volume.HddsVolumeChecker: Scheduled health check for volume /data/hdds/hdds
datanode_1 | 2019-11-16 00:36:42,638 INFO volume.VolumeSet: Added Volume : /data/hdds/hdds to VolumeSet
om_1 | OM initialization succeeded.Current cluster id for sd=/data/metadata/om;cid=CID-e0e3fa81-dff6-4fc7-aeb4-80e03811a63e
scm_1 | /************************************************************
s3g_1 | 2019-11-16 00:38:14,120 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
datanode_3 | 2019-11-16 00:36:43,093 INFO volume.HddsVolumeChecker: Scheduled health check for volume /data/hdds/hdds
datanode_2 | 2019-11-16 00:36:44,265 WARN scm.HddsServerUtil: Storage directory for Ratis is not configured. It is a good idea to map this to an SSD disk. Falling back to ozone.metadata.dirs
om_1 | 2019-11-16 00:36:46,001 INFO om.OzoneManagerStarter: SHUTDOWN_MSG:
scm_1 | SHUTDOWN_MSG: Shutting down StorageContainerManager at 897320affd68/172.19.0.2
s3g_1 | 2019-11-16 00:38:14,120 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics system started
datanode_3 | 2019-11-16 00:36:43,984 WARN scm.HddsServerUtil: Storage directory for Ratis is not configured. It is a good idea to map this to an SSD disk. Falling back to ozone.metadata.dirs
datanode_1 | 2019-11-16 00:36:42,649 INFO volume.ThrottledAsyncChecker: Scheduling a check for /data/hdds/hdds
datanode_2 | 2019-11-16 00:36:44,294 INFO impl.RaftServerProxy: raft.rpc.type = GRPC (default)
om_1 | /************************************************************
scm_1 | ************************************************************/
s3g_1 | 2019-11-16 00:38:14,123 WARN impl.MetricsSystemImpl: Sink prometheus already exists!
datanode_3 | 2019-11-16 00:36:44,011 INFO impl.RaftServerProxy: raft.rpc.type = GRPC (default)
datanode_1 | 2019-11-16 00:36:42,671 INFO volume.HddsVolumeChecker: Scheduled health check for volume /data/hdds/hdds
datanode_2 | 2019-11-16 00:36:44,588 INFO grpc.GrpcConfigKeys$Server: raft.grpc.server.port = 9858 (custom)
om_1 | SHUTDOWN_MSG: Shutting down OzoneManager at d45a264bff6a/172.19.0.3
scm_1 | 2019-11-16 00:36:43,330 INFO server.StorageContainerManagerStarter: STARTUP_MSG:
s3g_1 | 2019-11-16 00:38:21,312 INFO endpoint.BucketEndpoint: Location is /bucket-09761
datanode_3 | 2019-11-16 00:36:44,227 INFO grpc.GrpcConfigKeys$Server: raft.grpc.server.port = 9858 (custom)
datanode_1 | 2019-11-16 00:36:43,701 WARN scm.HddsServerUtil: Storage directory for Ratis is not configured. It is a good idea to map this to an SSD disk. Falling back to ozone.metadata.dirs
datanode_2 | 2019-11-16 00:36:44,589 INFO server.GrpcService: raft.grpc.message.size.max = 33570816 (custom)
om_1 | ************************************************************/
scm_1 | /************************************************************
s3g_1 | 2019-11-16 00:38:21,999 INFO endpoint.BucketEndpoint: Location is /bucket-09761
datanode_3 | 2019-11-16 00:36:44,228 INFO server.GrpcService: raft.grpc.message.size.max = 33570816 (custom)
datanode_1 | 2019-11-16 00:36:43,725 INFO impl.RaftServerProxy: raft.rpc.type = GRPC (default)
datanode_2 | 2019-11-16 00:36:44,592 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
om_1 | 2019-11-16 00:36:47,001 INFO om.OzoneManagerStarter: STARTUP_MSG:
scm_1 | STARTUP_MSG: Starting StorageContainerManager
s3g_1 | 2019-11-16 00:38:25,685 INFO endpoint.BucketEndpoint: Location is /bucket-09976
datanode_3 | 2019-11-16 00:36:44,230 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
datanode_1 | 2019-11-16 00:36:43,916 INFO grpc.GrpcConfigKeys$Server: raft.grpc.server.port = 9858 (custom)
datanode_2 | 2019-11-16 00:36:44,593 INFO server.GrpcService: raft.grpc.flow.control.window = 1MB (=1048576) (default)
om_1 | /************************************************************
datanode_3 | 2019-11-16 00:36:44,231 INFO server.GrpcService: raft.grpc.flow.control.window = 1MB (=1048576) (default)
s3g_1 | 2019-11-16 00:38:26,966 ERROR endpoint.BucketEndpoint: Exception occurred in headBucket
scm_1 | STARTUP_MSG: host = 897320affd68/172.19.0.2
datanode_1 | 2019-11-16 00:36:43,917 INFO server.GrpcService: raft.grpc.message.size.max = 33570816 (custom)
datanode_2 | 2019-11-16 00:36:44,594 INFO server.RaftServerConfigKeys: raft.server.rpc.request.timeout = 3000ms (default)
datanode_3 | 2019-11-16 00:36:44,232 INFO server.RaftServerConfigKeys: raft.server.rpc.request.timeout = 3000ms (default)
s3g_1 | org.apache.hadoop.ozone.s3.exception.OS3Exception
scm_1 | STARTUP_MSG: args = []
om_1 | STARTUP_MSG: Starting OzoneManager
datanode_1 | 2019-11-16 00:36:43,919 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
datanode_2 | 2019-11-16 00:36:44,771 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom)
datanode_3 | 2019-11-16 00:36:44,391 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom)
s3g_1 | at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:103)
scm_1 | STARTUP_MSG: version = 3.2.0
om_1 | STARTUP_MSG: host = d45a264bff6a/172.19.0.3
datanode_1 | 2019-11-16 00:36:43,920 INFO server.GrpcService: raft.grpc.flow.control.window = 1MB (=1048576) (default)
datanode_2 | 2019-11-16 00:36:44,986 INFO hdfs.DFSUtil: Starting Web-server for hddsDatanode at: http://0.0.0.0:9882
datanode_3 | 2019-11-16 00:36:44,556 INFO hdfs.DFSUtil: Starting Web-server for hddsDatanode at: http://0.0.0.0:9882
s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.EndpointBase.getBucket(EndpointBase.java:81)
scm_1 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/hamcrest-all-1.3.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-docs-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-scm-0.5.0-SNAPSHOT.jar
om_1 | STARTUP_MSG: args = []
datanode_2 | 2019-11-16 00:36:45,015 INFO util.log: Logging initialized @3333ms
datanode_1 | 2019-11-16 00:36:43,920 INFO server.RaftServerConfigKeys: raft.server.rpc.request.timeout = 3000ms (default)
datanode_3 | 2019-11-16 00:36:44,582 INFO util.log: Logging initialized @3051ms
s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.head(BucketEndpoint.java:250)
scm_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z
om_1 | STARTUP_MSG: version = 3.2.0
datanode_2 | 2019-11-16 00:36:45,124 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
datanode_1 | 2019-11-16 00:36:44,066 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom)
s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
scm_1 | STARTUP_MSG: java = 11.0.3
datanode_3 | 2019-11-16 00:36:44,673 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
datanode_2 | 2019-11-16 00:36:45,127 INFO http.HttpRequestLog: Http request log for http.requests.hddsDatanode is not defined
om_1 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/sqlite-jdbc-3.25.2.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/jaxb-impl-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/hadoop-ozone-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-tools-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/hadoop-ozone-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-docs-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-d6d58d0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-ozone-ozone-manager-0.5.0-SNAPSHOT.jar
datanode_1 | 2019-11-16 00:36:44,230 INFO hdfs.DFSUtil: Starting Web-server for hddsDatanode at: http://0.0.0.0:9882
s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
datanode_3 | 2019-11-16 00:36:44,675 INFO http.HttpRequestLog: Http request log for http.requests.hddsDatanode is not defined
datanode_2 | 2019-11-16 00:36:45,134 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
datanode_1 | 2019-11-16 00:36:44,250 INFO util.log: Logging initialized @3106ms
om_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z
scm_1 | ************************************************************/
s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
datanode_3 | 2019-11-16 00:36:44,682 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
datanode_1 | 2019-11-16 00:36:44,332 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
om_1 | STARTUP_MSG: java = 11.0.3
s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566)
scm_1 | 2019-11-16 00:36:43,337 INFO server.StorageContainerManagerStarter: registered UNIX signal handlers for [TERM, HUP, INT]
datanode_3 | 2019-11-16 00:36:44,683 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hddsDatanode
datanode_2 | 2019-11-16 00:36:45,136 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hddsDatanode
datanode_1 | 2019-11-16 00:36:44,335 INFO http.HttpRequestLog: Http request log for http.requests.hddsDatanode is not defined
om_1 | ************************************************************/
s3g_1 | at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
scm_1 | 2019-11-16 00:36:43,425 WARN server.ServerUtils: ozone.scm.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead.
datanode_3 | 2019-11-16 00:36:44,683 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
datanode_2 | 2019-11-16 00:36:45,136 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
datanode_1 | 2019-11-16 00:36:44,342 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
om_1 | 2019-11-16 00:36:47,008 INFO om.OzoneManagerStarter: registered UNIX signal handlers for [TERM, HUP, INT]
s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
scm_1 | 2019-11-16 00:36:43,438 WARN server.ServerUtils: ozone.scm.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead.
datanode_2 | 2019-11-16 00:36:45,136 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
datanode_3 | 2019-11-16 00:36:44,683 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
datanode_1 | 2019-11-16 00:36:44,352 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hddsDatanode
om_1 | 2019-11-16 00:36:47,754 INFO ha.OMHANodeDetails: Configuration either no ozone.om.address set. Falling back to the default OM address om/172.19.0.3:9862
s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
scm_1 | 2019-11-16 00:36:43,468 INFO util.log: Logging initialized @1233ms
datanode_2 | 2019-11-16 00:36:45,163 INFO http.HttpServer2: Jetty bound to port 9882
datanode_3 | 2019-11-16 00:36:44,711 INFO http.HttpServer2: Jetty bound to port 9882
datanode_1 | 2019-11-16 00:36:44,352 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
om_1 | 2019-11-16 00:36:47,755 INFO ha.OMHANodeDetails: OM Service ID is not set. Setting it to the default ID: omServiceIdDefault
s3g_1 | at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
scm_1 | 2019-11-16 00:36:43,572 INFO db.DBStoreBuilder: using custom profile for table: deletedBlocks
datanode_2 | 2019-11-16 00:36:45,165 INFO server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732
datanode_3 | 2019-11-16 00:36:44,712 INFO server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732
datanode_1 | 2019-11-16 00:36:44,352 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
om_1 | 2019-11-16 00:36:47,761 WARN server.ServerUtils: ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead.
scm_1 | 2019-11-16 00:36:43,572 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:deletedBlocks
datanode_2 | 2019-11-16 00:36:45,194 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@317a118b{/logs,file:///var/log/hadoop/,AVAILABLE}
datanode_3 | 2019-11-16 00:36:44,741 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@518bfd90{/logs,file:///var/log/hadoop/,AVAILABLE}
datanode_1 | 2019-11-16 00:36:44,384 INFO http.HttpServer2: Jetty bound to port 9882
s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
om_1 | 2019-11-16 00:36:47,774 WARN server.ServerUtils: ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead.
scm_1 | 2019-11-16 00:36:43,572 INFO db.DBStoreBuilder: using custom profile for table: validCerts
datanode_2 | 2019-11-16 00:36:45,195 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@73844119{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
datanode_3 | 2019-11-16 00:36:44,741 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@18b8d173{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
datanode_1 | 2019-11-16 00:36:44,385 INFO server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732
s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
om_1 | 2019-11-16 00:36:48,529 WARN server.ServerUtils: ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead.
scm_1 | 2019-11-16 00:36:43,572 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:validCerts
datanode_3 | 2019-11-16 00:36:44,810 INFO handler.ContextHandler: Started o.e.j.w.WebAppContext@862f408{/,file:///tmp/jetty-0.0.0.0-9882-hddsDatanode-_-any-2749332621796837032.dir/webapp/,AVAILABLE}{/hddsDatanode}
datanode_1 | 2019-11-16 00:36:44,415 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@317a118b{/logs,file:///var/log/hadoop/,AVAILABLE}
datanode_2 | 2019-11-16 00:36:45,250 INFO handler.ContextHandler: Started o.e.j.w.WebAppContext@178f268a{/,file:///tmp/jetty-0.0.0.0-9882-hddsDatanode-_-any-8353602704762447812.dir/webapp/,AVAILABLE}{/hddsDatanode}
om_1 | 2019-11-16 00:36:48,553 INFO util.log: Logging initialized @2377ms
scm_1 | 2019-11-16 00:36:43,572 INFO db.DBStoreBuilder: using custom profile for table: revokedCerts
s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
datanode_3 | 2019-11-16 00:36:44,816 INFO server.AbstractConnector: Started ServerConnector@3c78e551{HTTP/1.1,[http/1.1]}{0.0.0.0:9882}
datanode_1 | 2019-11-16 00:36:44,416 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@73844119{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
datanode_2 | 2019-11-16 00:36:45,255 INFO server.AbstractConnector: Started ServerConnector@7f1ef916{HTTP/1.1,[http/1.1]}{0.0.0.0:9882}
om_1 | 2019-11-16 00:36:48,644 INFO db.DBStoreBuilder: using custom profile for table: userTable
scm_1 | 2019-11-16 00:36:43,572 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:revokedCerts
s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
datanode_3 | 2019-11-16 00:36:44,819 INFO server.Server: Started @3288ms
datanode_1 | 2019-11-16 00:36:44,551 INFO handler.ContextHandler: Started o.e.j.w.WebAppContext@178f268a{/,file:///tmp/jetty-0.0.0.0-9882-hddsDatanode-_-any-15223405031894776508.dir/webapp/,AVAILABLE}{/hddsDatanode}
datanode_2 | 2019-11-16 00:36:45,256 INFO server.Server: Started @3574ms
om_1 | 2019-11-16 00:36:48,644 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:userTable
scm_1 | 2019-11-16 00:36:43,585 INFO db.DBStoreBuilder: using custom profile for table: default
s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
datanode_1 | 2019-11-16 00:36:44,562 INFO server.AbstractConnector: Started ServerConnector@69097eb0{HTTP/1.1,[http/1.1]}{0.0.0.0:9882}
datanode_3 | 2019-11-16 00:36:44,822 INFO impl.MetricsSinkAdapter: Sink prometheus started
datanode_2 | 2019-11-16 00:36:45,258 INFO impl.MetricsSinkAdapter: Sink prometheus started
om_1 | 2019-11-16 00:36:48,645 INFO db.DBStoreBuilder: using custom profile for table: volumeTable
scm_1 | 2019-11-16 00:36:43,586 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:default
s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
datanode_1 | 2019-11-16 00:36:44,563 INFO server.Server: Started @3419ms
datanode_3 | 2019-11-16 00:36:44,822 INFO impl.MetricsSystemImpl: Registered sink prometheus
datanode_2 | 2019-11-16 00:36:45,258 INFO impl.MetricsSystemImpl: Registered sink prometheus
om_1 | 2019-11-16 00:36:48,645 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:volumeTable
s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
scm_1 | 2019-11-16 00:36:43,587 INFO db.DBStoreBuilder: Using default options. DBProfile.DISK
datanode_1 | 2019-11-16 00:36:44,566 INFO impl.MetricsSinkAdapter: Sink prometheus started
datanode_3 | 2019-11-16 00:36:44,824 INFO server.BaseHttpServer: HTTP server of HDDSDATANODE is listening at http://0.0.0.0:9882
datanode_2 | 2019-11-16 00:36:45,260 INFO server.BaseHttpServer: HTTP server of HDDSDATANODE is listening at http://0.0.0.0:9882
om_1 | 2019-11-16 00:36:48,645 INFO db.DBStoreBuilder: using custom profile for table: bucketTable
s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
scm_1 | 2019-11-16 00:36:43,844 INFO net.NodeSchemaLoader: Loading file from java.lang.CompoundEnumeration@39d76cb5
datanode_3 | 2019-11-16 00:36:44,838 INFO util.JvmPauseMonitor: Starting JVM pause monitor
datanode_1 | 2019-11-16 00:36:44,566 INFO impl.MetricsSystemImpl: Registered sink prometheus
datanode_2 | 2019-11-16 00:36:45,272 INFO util.JvmPauseMonitor: Starting JVM pause monitor
om_1 | 2019-11-16 00:36:48,645 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:bucketTable
s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
scm_1 | 2019-11-16 00:36:43,845 INFO net.NodeSchemaLoader: Loading network topology layer schema file
datanode_3 | 2019-11-16 00:36:45,008 INFO datanode.InitDatanodeState: DatanodeDetails is persisted to /data/datanode.id
datanode_1 | 2019-11-16 00:36:44,568 INFO server.BaseHttpServer: HTTP server of HDDSDATANODE is listening at http://0.0.0.0:9882
datanode_2 | 2019-11-16 00:36:45,418 INFO datanode.InitDatanodeState: DatanodeDetails is persisted to /data/datanode.id
om_1 | 2019-11-16 00:36:48,645 INFO db.DBStoreBuilder: using custom profile for table: keyTable
s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
scm_1 | 2019-11-16 00:36:43,943 INFO node.SCMNodeManager: Entering startup safe mode.
datanode_3 | 2019-11-16 00:36:46,946 INFO ozoneimpl.OzoneContainer: Attempting to start container services.
datanode_1 | 2019-11-16 00:36:44,582 INFO util.JvmPauseMonitor: Starting JVM pause monitor
datanode_2 | 2019-11-16 00:36:47,400 INFO ozoneimpl.OzoneContainer: Attempting to start container services.
om_1 | 2019-11-16 00:36:48,646 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:keyTable
s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
scm_1 | 2019-11-16 00:36:44,049 INFO algorithms.ContainerPlacementPolicyFactory: Create container placement policy of type org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRandom
datanode_1 | 2019-11-16 00:36:44,731 INFO datanode.InitDatanodeState: DatanodeDetails is persisted to /data/datanode.id
datanode_3 | 2019-11-16 00:36:46,948 INFO ozoneimpl.OzoneContainer: Background container scanner has been disabled.
datanode_2 | 2019-11-16 00:36:47,402 INFO ozoneimpl.OzoneContainer: Background container scanner has been disabled.
om_1 | 2019-11-16 00:36:48,646 INFO db.DBStoreBuilder: using custom profile for table: deletedTable
scm_1 | 2019-11-16 00:36:44,065 WARN server.ServerUtils: ozone.scm.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead.
datanode_1 | 2019-11-16 00:36:46,775 INFO ozoneimpl.OzoneContainer: Attempting to start container services.
s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
datanode_3 | 2019-11-16 00:36:46,948 INFO ratis.XceiverServerRatis: Starting XceiverServerRatis 6f2abcfb-ae75-487f-9927-306aeedbcad3 at port 9858
datanode_2 | 2019-11-16 00:36:47,402 INFO ratis.XceiverServerRatis: Starting XceiverServerRatis 0e873e00-679f-460c-b6e9-cef3ff1ecd70 at port 9858
om_1 | 2019-11-16 00:36:48,646 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:deletedTable
scm_1 | 2019-11-16 00:36:44,144 INFO pipeline.SCMPipelineManager: No pipeline exists in current db
datanode_1 | 2019-11-16 00:36:46,776 INFO ozoneimpl.OzoneContainer: Background container scanner has been disabled.
s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
datanode_2 | 2019-11-16 00:36:47,422 INFO impl.RaftServerProxy: 0e873e00-679f-460c-b6e9-cef3ff1ecd70: start RPC server
om_1 | 2019-11-16 00:36:48,646 INFO db.DBStoreBuilder: using custom profile for table: openKeyTable
scm_1 | 2019-11-16 00:36:44,146 WARN server.ServerUtils: ozone.scm.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead.
datanode_1 | 2019-11-16 00:36:46,776 INFO ratis.XceiverServerRatis: Starting XceiverServerRatis d57cb209-6a43-4e94-bc69-740186336f55 at port 9858
datanode_3 | 2019-11-16 00:36:46,966 INFO impl.RaftServerProxy: 6f2abcfb-ae75-487f-9927-306aeedbcad3: start RPC server
s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
datanode_2 | 2019-11-16 00:36:47,548 INFO server.GrpcService: 0e873e00-679f-460c-b6e9-cef3ff1ecd70: GrpcService started, listening on 0.0.0.0/0.0.0.0:9858
om_1 | 2019-11-16 00:36:48,646 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:openKeyTable
scm_1 | 2019-11-16 00:36:44,275 WARN events.EventQueue: No event handler registered for event TypedEvent{payloadType=SafeModeStatus, name='SafeModeStatus'}
datanode_1 | 2019-11-16 00:36:46,800 INFO impl.RaftServerProxy: d57cb209-6a43-4e94-bc69-740186336f55: start RPC server
s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
datanode_3 | 2019-11-16 00:36:47,070 INFO server.GrpcService: 6f2abcfb-ae75-487f-9927-306aeedbcad3: GrpcService started, listening on 0.0.0.0/0.0.0.0:9858
datanode_2 | WARNING: An illegal reflective access operation has occurred
om_1 | 2019-11-16 00:36:48,646 INFO db.DBStoreBuilder: using custom profile for table: s3Table
scm_1 | 2019-11-16 00:36:44,952 INFO ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
datanode_1 | 2019-11-16 00:36:46,913 INFO server.GrpcService: d57cb209-6a43-4e94-bc69-740186336f55: GrpcService started, listening on 0.0.0.0/0.0.0.0:9858
s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
datanode_3 | WARNING: An illegal reflective access operation has occurred
datanode_2 | WARNING: Illegal reflective access by org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil (file:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar) to field java.nio.Buffer.address
om_1 | 2019-11-16 00:36:48,646 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:s3Table
scm_1 | 2019-11-16 00:36:44,972 INFO ipc.Server: Starting Socket Reader #1 for port 9861
datanode_1 | WARNING: An illegal reflective access operation has occurred
s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
datanode_3 | WARNING: Illegal reflective access by org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil (file:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar) to field java.nio.Buffer.address
om_1 | 2019-11-16 00:36:48,646 INFO db.DBStoreBuilder: using custom profile for table: multipartInfoTable
datanode_2 | WARNING: Please consider reporting this to the maintainers of org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil
datanode_1 | WARNING: Illegal reflective access by org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil (file:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar) to field java.nio.Buffer.address
s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840)
datanode_3 | WARNING: Please consider reporting this to the maintainers of org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil
om_1 | 2019-11-16 00:36:48,647 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:multipartInfoTable
scm_1 | 2019-11-16 00:36:45,006 INFO ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
datanode_2 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
datanode_1 | WARNING: Please consider reporting this to the maintainers of org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780)
datanode_3 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
om_1 | 2019-11-16 00:36:48,647 INFO db.DBStoreBuilder: using custom profile for table: dTokenTable
scm_1 | 2019-11-16 00:36:45,007 INFO ipc.Server: Starting Socket Reader #1 for port 9863
datanode_2 | WARNING: All illegal access operations will be denied in a future release
datanode_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
datanode_3 | WARNING: All illegal access operations will be denied in a future release
om_1 | 2019-11-16 00:36:48,647 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:dTokenTable
s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609)
scm_1 | 2019-11-16 00:36:45,023 INFO ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
datanode_2 | 2019-11-16 00:36:50,041 INFO impl.RaftServerProxy: 0e873e00-679f-460c-b6e9-cef3ff1ecd70: addNew group-D11041DA73D9:[0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858] returns group-D11041DA73D9:java.util.concurrent.CompletableFuture@757a1737[Not completed]
datanode_1 | WARNING: All illegal access operations will be denied in a future release
datanode_3 | 2019-11-16 00:36:50,558 INFO impl.RaftServerProxy: 6f2abcfb-ae75-487f-9927-306aeedbcad3: addNew group-82BA2C1717AB:[6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858] returns group-82BA2C1717AB:java.util.concurrent.CompletableFuture@3fdabd9e[Not completed]
om_1 | 2019-11-16 00:36:48,647 INFO db.DBStoreBuilder: using custom profile for table: s3SecretTable
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
scm_1 | 2019-11-16 00:36:45,024 INFO ipc.Server: Starting Socket Reader #1 for port 9860
datanode_2 | 2019-11-16 00:36:50,054 INFO impl.RaftServerImpl: 0e873e00-679f-460c-b6e9-cef3ff1ecd70: new RaftServerImpl for group-D11041DA73D9:[0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858] with ContainerStateMachine:uninitialized
datanode_1 | 2019-11-16 00:36:49,573 INFO impl.RaftServerProxy: d57cb209-6a43-4e94-bc69-740186336f55: addNew group-AD1B671D20B0:[d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858] returns group-AD1B671D20B0:java.util.concurrent.CompletableFuture@d12fe24[Not completed]
datanode_3 | 2019-11-16 00:36:50,581 INFO impl.RaftServerImpl: 6f2abcfb-ae75-487f-9927-306aeedbcad3: new RaftServerImpl for group-82BA2C1717AB:[6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858] with ContainerStateMachine:uninitialized
om_1 | 2019-11-16 00:36:48,647 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:s3SecretTable
s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
scm_1 | 2019-11-16 00:36:45,057 INFO hdfs.DFSUtil: Starting Web-server for scm at: http://0.0.0.0:9876
datanode_2 | 2019-11-16 00:36:50,055 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom)
datanode_1 | 2019-11-16 00:36:49,590 INFO impl.RaftServerImpl: d57cb209-6a43-4e94-bc69-740186336f55: new RaftServerImpl for group-AD1B671D20B0:[d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858] with ContainerStateMachine:uninitialized
datanode_3 | 2019-11-16 00:36:50,583 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom)
om_1 | 2019-11-16 00:36:48,647 INFO db.DBStoreBuilder: using custom profile for table: prefixTable
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
datanode_2 | 2019-11-16 00:36:50,056 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom)
datanode_3 | 2019-11-16 00:36:50,584 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom)
datanode_1 | 2019-11-16 00:36:49,591 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom)
scm_1 | 2019-11-16 00:36:45,257 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
datanode_2 | 2019-11-16 00:36:50,056 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom)
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
datanode_3 | 2019-11-16 00:36:50,584 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom)
om_1 | 2019-11-16 00:36:48,647 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:prefixTable
datanode_1 | 2019-11-16 00:36:49,591 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom)
scm_1 | 2019-11-16 00:36:45,270 INFO http.HttpRequestLog: Http request log for http.requests.scm is not defined
datanode_2 | 2019-11-16 00:36:50,056 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default)
s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
datanode_3 | 2019-11-16 00:36:50,584 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default)
om_1 | 2019-11-16 00:36:48,664 INFO db.DBStoreBuilder: using custom profile for table: default
scm_1 | 2019-11-16 00:36:45,277 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
datanode_1 | 2019-11-16 00:36:49,592 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom)
datanode_2 | 2019-11-16 00:36:50,057 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom)
s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
datanode_3 | 2019-11-16 00:36:50,585 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom)
scm_1 | 2019-11-16 00:36:45,280 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context scm
datanode_1 | 2019-11-16 00:36:49,592 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default)
datanode_2 | 2019-11-16 00:36:50,061 INFO impl.RaftServerImpl: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-D11041DA73D9: ConfigurationManager, init=-1: [0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858], old=null, confs=<EMPTY_MAP>
om_1 | 2019-11-16 00:36:48,664 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:default
s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
datanode_3 | 2019-11-16 00:36:50,593 INFO impl.RaftServerImpl: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-82BA2C1717AB: ConfigurationManager, init=-1: [6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858], old=null, confs=<EMPTY_MAP>
scm_1 | 2019-11-16 00:36:45,280 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
datanode_1 | 2019-11-16 00:36:49,593 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom)
datanode_2 | 2019-11-16 00:36:50,061 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom)
om_1 | 2019-11-16 00:36:48,665 INFO db.DBStoreBuilder: Using default options. DBProfile.DISK
s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
datanode_3 | 2019-11-16 00:36:50,593 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom)
scm_1 | 2019-11-16 00:36:45,280 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
datanode_1 | 2019-11-16 00:36:49,598 INFO impl.RaftServerImpl: d57cb209-6a43-4e94-bc69-740186336f55@group-AD1B671D20B0: ConfigurationManager, init=-1: [d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858], old=null, confs=<EMPTY_MAP>
datanode_2 | 2019-11-16 00:36:50,065 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default)
om_1 | 2019-11-16 00:36:49,346 INFO ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 2000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
datanode_3 | 2019-11-16 00:36:50,599 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default)
scm_1 | 2019-11-16 00:36:45,303 INFO server.StorageContainerManager: StorageContainerLocationProtocol RPC server is listening at /0.0.0.0:9860
datanode_1 | 2019-11-16 00:36:49,599 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom)
datanode_2 | 2019-11-16 00:36:50,066 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/464a36a4-4722-4a03-b9b3-d11041da73d9 does not exist. Creating ...
om_1 | 2019-11-16 00:36:49,357 INFO ipc.Server: Starting Socket Reader #1 for port 9862
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)
datanode_3 | 2019-11-16 00:36:50,601 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/c2c4f6bf-e8c7-4e1b-a2cb-82ba2c1717ab does not exist. Creating ...
scm_1 | 2019-11-16 00:36:45,362 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
datanode_1 | 2019-11-16 00:36:49,603 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default)
datanode_2 | 2019-11-16 00:36:50,083 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/464a36a4-4722-4a03-b9b3-d11041da73d9/in_use.lock acquired by nodename 7@ab54c8aa3abc
om_1 | 2019-11-16 00:36:49,398 INFO om.OzoneManager: OzoneManager RPC server is listening at om/172.19.0.3:9862
s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
datanode_3 | 2019-11-16 00:36:50,620 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/c2c4f6bf-e8c7-4e1b-a2cb-82ba2c1717ab/in_use.lock acquired by nodename 7@70c5f52b4175
datanode_1 | 2019-11-16 00:36:49,605 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/71b493d2-59e5-44fa-82f3-ad1b671d20b0 does not exist. Creating ...
scm_1 | 2019-11-16 00:36:45,402 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
datanode_2 | 2019-11-16 00:36:50,097 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/464a36a4-4722-4a03-b9b3-d11041da73d9 has been successfully formatted.
om_1 | 2019-11-16 00:36:49,488 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
datanode_3 | 2019-11-16 00:36:50,635 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/c2c4f6bf-e8c7-4e1b-a2cb-82ba2c1717ab has been successfully formatted.
datanode_1 | 2019-11-16 00:36:49,622 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/71b493d2-59e5-44fa-82f3-ad1b671d20b0/in_use.lock acquired by nodename 7@c29ed315f282
scm_1 | 2019-11-16 00:36:45,402 INFO impl.MetricsSystemImpl: StorageContainerManager metrics system started
om_1 | 2019-11-16 00:36:49,535 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
datanode_3 | 2019-11-16 00:36:50,639 INFO ratis.ContainerStateMachine: group-82BA2C1717AB: The snapshot info is null. Setting the last applied indexto:(t:0, i:~)
datanode_2 | 2019-11-16 00:36:50,100 INFO ratis.ContainerStateMachine: group-D11041DA73D9: The snapshot info is null. Setting the last applied indexto:(t:0, i:~)
datanode_1 | 2019-11-16 00:36:49,637 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/71b493d2-59e5-44fa-82f3-ad1b671d20b0 has been successfully formatted.
om_1 | 2019-11-16 00:36:49,535 INFO impl.MetricsSystemImpl: OzoneManager metrics system started
scm_1 | 2019-11-16 00:36:45,615 INFO server.SCMClientProtocolServer: RPC server for Client is listening at /0.0.0.0:9860
s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
datanode_3 | 2019-11-16 00:36:50,639 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom)
datanode_2 | 2019-11-16 00:36:50,100 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom)
datanode_1 | 2019-11-16 00:36:49,641 INFO ratis.ContainerStateMachine: group-AD1B671D20B0: The snapshot info is null. Setting the last applied indexto:(t:0, i:~)
om_1 | 2019-11-16 00:36:49,574 INFO ipc.Server: IPC Server listener on 9862: starting
scm_1 | 2019-11-16 00:36:45,615 INFO ipc.Server: IPC Server Responder: starting
s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
datanode_3 | 2019-11-16 00:36:50,642 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default)
datanode_2 | 2019-11-16 00:36:50,102 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default)
datanode_1 | 2019-11-16 00:36:49,641 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom)
om_1 | 2019-11-16 00:36:49,575 INFO ipc.Server: IPC Server Responder: starting
scm_1 | 2019-11-16 00:36:45,615 INFO ipc.Server: IPC Server listener on 9860: starting
s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539)
datanode_3 | 2019-11-16 00:36:50,646 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom)
datanode_2 | 2019-11-16 00:36:50,106 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom)
datanode_1 | 2019-11-16 00:36:49,643 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default)
scm_1 | 2019-11-16 00:36:45,619 INFO server.StorageContainerManager: ScmBlockLocationProtocol RPC server is listening at /0.0.0.0:9863
s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
datanode_3 | 2019-11-16 00:36:50,646 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
datanode_2 | 2019-11-16 00:36:50,106 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
datanode_1 | 2019-11-16 00:36:49,647 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom)
scm_1 | 2019-11-16 00:36:45,620 INFO server.SCMBlockProtocolServer: RPC server for Block Protocol is listening at /0.0.0.0:9863
s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
datanode_3 | 2019-11-16 00:36:50,648 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom)
datanode_2 | 2019-11-16 00:36:50,107 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom)
datanode_1 | 2019-11-16 00:36:49,647 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
scm_1 | 2019-11-16 00:36:45,621 INFO ipc.Server: IPC Server Responder: starting
om_1 | 2019-11-16 00:36:49,620 INFO hdfs.DFSUtil: Starting Web-server for ozoneManager at: http://0.0.0.0:9874
s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
datanode_3 | 2019-11-16 00:36:50,654 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl
datanode_2 | 2019-11-16 00:36:50,111 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl
datanode_1 | 2019-11-16 00:36:49,649 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom)
scm_1 | 2019-11-16 00:36:45,621 INFO ipc.Server: IPC Server listener on 9863: starting
om_1 | 2019-11-16 00:36:49,743 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
datanode_3 | 2019-11-16 00:36:50,657 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
datanode_2 | 2019-11-16 00:36:50,114 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
datanode_1 | 2019-11-16 00:36:49,655 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl
scm_1 | 2019-11-16 00:36:45,625 INFO server.StorageContainerManager: ScmDatanodeProtocl RPC server is listening at /0.0.0.0:9861
om_1 | 2019-11-16 00:36:49,746 INFO http.HttpRequestLog: Http request log for http.requests.ozoneManager is not defined
s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
datanode_3 | 2019-11-16 00:36:50,662 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom)
datanode_2 | 2019-11-16 00:36:50,117 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom)
scm_1 | 2019-11-16 00:36:45,626 INFO server.SCMDatanodeProtocolServer: RPC server for DataNodes is listening at /0.0.0.0:9861
s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
datanode_3 | 2019-11-16 00:36:50,666 INFO segmented.SegmentedRaftLogWorker: new 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-82BA2C1717AB-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/c2c4f6bf-e8c7-4e1b-a2cb-82ba2c1717ab
datanode_1 | 2019-11-16 00:36:49,658 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
om_1 | 2019-11-16 00:36:49,753 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
datanode_2 | 2019-11-16 00:36:50,121 INFO segmented.SegmentedRaftLogWorker: new 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-D11041DA73D9-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/464a36a4-4722-4a03-b9b3-d11041da73d9
scm_1 | 2019-11-16 00:36:45,627 INFO ipc.Server: IPC Server Responder: starting
s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
datanode_1 | 2019-11-16 00:36:49,663 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom)
om_1 | 2019-11-16 00:36:49,755 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context ozoneManager
datanode_2 | 2019-11-16 00:36:50,121 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom)
scm_1 | 2019-11-16 00:36:45,627 INFO ipc.Server: IPC Server listener on 9861: starting
datanode_3 | 2019-11-16 00:36:50,667 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom)
s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
datanode_1 | 2019-11-16 00:36:49,667 INFO segmented.SegmentedRaftLogWorker: new d57cb209-6a43-4e94-bc69-740186336f55@group-AD1B671D20B0-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/71b493d2-59e5-44fa-82f3-ad1b671d20b0
om_1 | 2019-11-16 00:36:49,755 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
datanode_2 | 2019-11-16 00:36:50,121 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom)
scm_1 | 2019-11-16 00:36:45,631 INFO http.HttpServer2: Jetty bound to port 9876
datanode_3 | 2019-11-16 00:36:50,667 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom)
s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
datanode_1 | 2019-11-16 00:36:49,667 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom)
om_1 | 2019-11-16 00:36:49,755 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
datanode_2 | 2019-11-16 00:36:50,122 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom)
scm_1 | 2019-11-16 00:36:45,633 INFO server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732
datanode_3 | 2019-11-16 00:36:50,668 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom)
s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
datanode_1 | 2019-11-16 00:36:49,668 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom)
om_1 | 2019-11-16 00:36:49,775 INFO http.HttpServer2: Jetty bound to port 9874
datanode_2 | 2019-11-16 00:36:50,123 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom)
scm_1 | 2019-11-16 00:36:45,688 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@2ba33e2c{/logs,file:///var/log/hadoop/,AVAILABLE}
datanode_3 | 2019-11-16 00:36:50,668 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom)
s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834)
datanode_1 | 2019-11-16 00:36:49,669 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom)
om_1 | 2019-11-16 00:36:49,776 INFO server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732
datanode_2 | 2019-11-16 00:36:50,123 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom)
scm_1 | 2019-11-16 00:36:45,688 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@629ae7e{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-scm-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
datanode_3 | 2019-11-16 00:36:50,669 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom)
s3g_1 | 2019-11-16 00:38:30,216 INFO endpoint.BucketEndpoint: Location is /bucket-84278
datanode_1 | 2019-11-16 00:36:49,669 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom)
om_1 | 2019-11-16 00:36:49,805 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@3b5c665c{/logs,file:///var/log/hadoop/,AVAILABLE}
datanode_2 | 2019-11-16 00:36:50,123 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default)
scm_1 | 2019-11-16 00:36:45,796 INFO handler.ContextHandler: Started o.e.j.w.WebAppContext@5300cac{/,file:///tmp/jetty-0.0.0.0-9876-scm-_-any-17493975762482183383.dir/webapp/,AVAILABLE}{/scm}
datanode_3 | 2019-11-16 00:36:50,669 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default)
s3g_1 | 2019-11-16 00:38:34,281 INFO endpoint.BucketEndpoint: Location is /bucket-11455
datanode_1 | 2019-11-16 00:36:49,669 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom)
om_1 | 2019-11-16 00:36:49,805 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@60d40ff4{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-ozone-ozone-manager-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
datanode_2 | 2019-11-16 00:36:50,124 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default)
scm_1 | 2019-11-16 00:36:45,805 INFO server.AbstractConnector: Started ServerConnector@6f49d153{HTTP/1.1,[http/1.1]}{0.0.0.0:9876}
datanode_3 | 2019-11-16 00:36:50,670 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default)
s3g_1 | 2019-11-16 00:38:36,445 INFO io.BlockOutputStreamEntryPool: Allocating block with ExcludeList {datanodes = [], containerIds = [], pipelineIds = []}
datanode_1 | 2019-11-16 00:36:49,670 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default)
om_1 | 2019-11-16 00:36:49,879 INFO handler.ContextHandler: Started o.e.j.w.WebAppContext@44af588b{/,file:///tmp/jetty-0.0.0.0-9874-ozoneManager-_-any-4767582424262264414.dir/webapp/,AVAILABLE}{/ozoneManager}
datanode_2 | 2019-11-16 00:36:50,124 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default)
scm_1 | 2019-11-16 00:36:45,806 INFO server.Server: Started @3571ms
datanode_3 | 2019-11-16 00:36:50,670 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default)
s3g_1 | 2019-11-16 00:38:36,962 WARN servlet.ServletHandler:
datanode_1 | 2019-11-16 00:36:49,670 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default)
om_1 | 2019-11-16 00:36:49,884 INFO server.AbstractConnector: Started ServerConnector@332bcab0{HTTP/1.1,[http/1.1]}{0.0.0.0:9874}
datanode_2 | 2019-11-16 00:36:50,124 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default)
scm_1 | 2019-11-16 00:36:45,809 INFO impl.MetricsSinkAdapter: Sink prometheus started
datanode_3 | 2019-11-16 00:36:50,670 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default)
s3g_1 | javax.servlet.ServletException: java.lang.NullPointerException
datanode_1 | 2019-11-16 00:36:49,670 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default)
om_1 | 2019-11-16 00:36:49,884 INFO server.Server: Started @3708ms
datanode_2 | 2019-11-16 00:36:50,131 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom)
scm_1 | 2019-11-16 00:36:45,809 INFO impl.MetricsSystemImpl: Registered sink prometheus
s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432)
datanode_3 | 2019-11-16 00:36:50,678 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom)
datanode_1 | 2019-11-16 00:36:49,671 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default)
om_1 | 2019-11-16 00:36:49,887 INFO impl.MetricsSinkAdapter: Sink prometheus started
datanode_2 | 2019-11-16 00:36:50,134 INFO segmented.SegmentedRaftLogWorker: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-D11041DA73D9-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1
s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
datanode_3 | 2019-11-16 00:36:50,682 INFO segmented.SegmentedRaftLogWorker: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-82BA2C1717AB-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1
om_1 | 2019-11-16 00:36:49,887 INFO impl.MetricsSystemImpl: Registered sink prometheus
datanode_1 | 2019-11-16 00:36:49,678 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom)
scm_1 | 2019-11-16 00:36:45,812 INFO server.BaseHttpServer: HTTP server of SCM is listening at http://0.0.0.0:9876
datanode_2 | 2019-11-16 00:36:50,137 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom)
s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
datanode_3 | 2019-11-16 00:36:50,688 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom)
om_1 | 2019-11-16 00:36:49,888 INFO server.BaseHttpServer: HTTP server of OZONEMANAGER is listening at http://0.0.0.0:9874
datanode_1 | 2019-11-16 00:36:49,682 INFO segmented.SegmentedRaftLogWorker: d57cb209-6a43-4e94-bc69-740186336f55@group-AD1B671D20B0-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1
scm_1 | 2019-11-16 00:36:45,821 INFO util.JvmPauseMonitor: Starting JVM pause monitor
datanode_2 | 2019-11-16 00:36:50,137 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom)
s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
datanode_3 | 2019-11-16 00:36:50,689 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom)
om_1 | 2019-11-16 00:37:04,267 INFO volume.OMVolumeCreateRequest: created volume:vol-0-78125 for user:hadoop
datanode_1 | 2019-11-16 00:36:49,685 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom)
scm_1 | 2019-11-16 00:36:48,607 INFO net.NetworkTopology: Added a new node: /default-rack/d57cb209-6a43-4e94-bc69-740186336f55
datanode_2 | 2019-11-16 00:36:50,138 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom)
s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
datanode_3 | 2019-11-16 00:36:50,690 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom)
om_1 | 2019-11-16 00:37:04,298 INFO volume.OMVolumeCreateRequest: created volume:vol-1-98072 for user:hadoop
datanode_1 | 2019-11-16 00:36:49,686 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom)
datanode_2 | 2019-11-16 00:36:50,138 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom)
s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840)
datanode_1 | 2019-11-16 00:36:49,686 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom)
om_1 | 2019-11-16 00:37:04,328 INFO volume.OMVolumeCreateRequest: created volume:vol-2-27408 for user:hadoop
datanode_3 | 2019-11-16 00:36:50,691 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom)
datanode_1 | 2019-11-16 00:36:49,687 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom)
datanode_2 | 2019-11-16 00:36:50,150 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780)
datanode_1 | 2019-11-16 00:36:49,702 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
om_1 | 2019-11-16 00:37:04,355 INFO volume.OMVolumeCreateRequest: created volume:vol-3-51360 for user:hadoop
datanode_3 | 2019-11-16 00:36:50,715 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
scm_1 | 2019-11-16 00:36:48,607 INFO node.SCMNodeManager: Registered Data node : d57cb209-6a43-4e94-bc69-740186336f55{ip: 172.19.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}
datanode_2 | 2019-11-16 00:36:50,152 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609)
datanode_1 | 2019-11-16 00:36:49,704 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
om_1 | 2019-11-16 00:37:04,382 INFO volume.OMVolumeCreateRequest: created volume:vol-4-49764 for user:hadoop
datanode_3 | 2019-11-16 00:36:50,717 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
scm_1 | 2019-11-16 00:36:48,610 INFO safemode.SCMSafeModeManager: SCM in safe mode. 1 DataNodes registered, 1 required.
datanode_2 | 2019-11-16 00:36:50,153 INFO impl.RaftServerImpl: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-D11041DA73D9: start as a follower, conf=-1: [0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858], old=null
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
datanode_1 | 2019-11-16 00:36:49,705 INFO impl.RaftServerImpl: d57cb209-6a43-4e94-bc69-740186336f55@group-AD1B671D20B0: start as a follower, conf=-1: [d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858], old=null
datanode_3 | 2019-11-16 00:36:50,719 INFO impl.RaftServerImpl: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-82BA2C1717AB: start as a follower, conf=-1: [6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858], old=null
scm_1 | 2019-11-16 00:36:48,610 INFO safemode.SCMSafeModeManager: ScmSafeModeManager, all rules are successfully validated
s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
datanode_2 | 2019-11-16 00:36:50,154 INFO impl.RaftServerImpl: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-D11041DA73D9: changes role from null to FOLLOWER at term 0 for startAsFollower
om_1 | 2019-11-16 00:38:21,956 ERROR proto.OzoneManagerProtocolProtos$S3CreateBucketRequest: S3Bucket Creation Failed for userName: 7c35ba70126b04fca007e4dbfab9c47c, s3BucketName bucket-09761, VolumeName s37c35ba70126b04fca007e4dbfab9c47c
datanode_1 | 2019-11-16 00:36:49,706 INFO impl.RaftServerImpl: d57cb209-6a43-4e94-bc69-740186336f55@group-AD1B671D20B0: changes role from null to FOLLOWER at term 0 for startAsFollower
datanode_3 | 2019-11-16 00:36:50,720 INFO impl.RaftServerImpl: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-82BA2C1717AB: changes role from null to FOLLOWER at term 0 for startAsFollower
scm_1 | 2019-11-16 00:36:48,612 INFO safemode.SCMSafeModeManager: SCM exiting safe mode.
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
datanode_2 | 2019-11-16 00:36:50,154 INFO impl.RoleInfo: 0e873e00-679f-460c-b6e9-cef3ff1ecd70: start FollowerState
om_1 | 2019-11-16 00:39:23,147 ERROR multipart.S3MultipartUploadCompleteRequest: Complete MultipartUpload failed for key /s37c35ba70126b04fca007e4dbfab9c47c/bucket-11455/multipartKey3 , MPU Key has no parts in OM, parts given to upload are [partNumber: 1
datanode_3 | 2019-11-16 00:36:50,721 INFO impl.RoleInfo: 6f2abcfb-ae75-487f-9927-306aeedbcad3: start FollowerState
scm_1 | 2019-11-16 00:36:48,857 INFO net.NetworkTopology: Added a new node: /default-rack/6f2abcfb-ae75-487f-9927-306aeedbcad3
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
datanode_2 | 2019-11-16 00:36:50,158 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-D11041DA73D9,id=0e873e00-679f-460c-b6e9-cef3ff1ecd70
om_1 | partName: "etag1"
datanode_1 | 2019-11-16 00:36:49,707 INFO impl.RoleInfo: d57cb209-6a43-4e94-bc69-740186336f55: start FollowerState
datanode_3 | 2019-11-16 00:36:50,726 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-82BA2C1717AB,id=6f2abcfb-ae75-487f-9927-306aeedbcad3
scm_1 | 2019-11-16 00:36:48,858 INFO node.SCMNodeManager: Registered Data node : 6f2abcfb-ae75-487f-9927-306aeedbcad3{ip: 172.19.0.5, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}
datanode_2 | 2019-11-16 00:36:50,159 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
om_1 | , partNumber: 2
datanode_1 | 2019-11-16 00:36:49,711 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-AD1B671D20B0,id=d57cb209-6a43-4e94-bc69-740186336f55
datanode_3 | 2019-11-16 00:36:50,728 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
scm_1 | 2019-11-16 00:36:49,299 INFO net.NetworkTopology: Added a new node: /default-rack/0e873e00-679f-460c-b6e9-cef3ff1ecd70
datanode_2 | 2019-11-16 00:36:50,861 INFO impl.RaftServerProxy: 0e873e00-679f-460c-b6e9-cef3ff1ecd70: addNew group-2EBDCA662DA4:[6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858, d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858, 0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858] returns group-2EBDCA662DA4:java.util.concurrent.CompletableFuture@5ae5cebf[Not completed]
om_1 | partName: "etag2"
datanode_1 | 2019-11-16 00:36:49,712 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
datanode_3 | 2019-11-16 00:36:50,846 INFO impl.RaftServerProxy: 6f2abcfb-ae75-487f-9927-306aeedbcad3: addNew group-2EBDCA662DA4:[6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858, d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858, 0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858] returns group-2EBDCA662DA4:java.util.concurrent.CompletableFuture@2dea7d2a[Not completed]
s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
scm_1 | 2019-11-16 00:36:49,299 INFO node.SCMNodeManager: Registered Data node : 0e873e00-679f-460c-b6e9-cef3ff1ecd70{ip: 172.19.0.6, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}
datanode_2 | 2019-11-16 00:36:50,864 INFO impl.RaftServerImpl: 0e873e00-679f-460c-b6e9-cef3ff1ecd70: new RaftServerImpl for group-2EBDCA662DA4:[6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858, d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858, 0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858] with ContainerStateMachine:uninitialized
om_1 | ]
datanode_1 | 2019-11-16 00:36:50,831 INFO impl.RaftServerProxy: d57cb209-6a43-4e94-bc69-740186336f55: addNew group-2EBDCA662DA4:[6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858, d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858, 0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858] returns group-2EBDCA662DA4:java.util.concurrent.CompletableFuture@452ca1ac[Not completed]
s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
datanode_3 | 2019-11-16 00:36:50,850 INFO impl.RaftServerImpl: 6f2abcfb-ae75-487f-9927-306aeedbcad3: new RaftServerImpl for group-2EBDCA662DA4:[6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858, d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858, 0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858] with ContainerStateMachine:uninitialized
scm_1 | WARNING: An illegal reflective access operation has occurred
datanode_2 | 2019-11-16 00:36:50,864 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom)
om_1 | 2019-11-16 00:39:23,149 ERROR multipart.S3MultipartUploadCompleteRequest: MultipartUpload Complete request failed for Key: multipartKey3 in Volume/Bucket s37c35ba70126b04fca007e4dbfab9c47c/bucket-11455
datanode_1 | 2019-11-16 00:36:50,835 INFO impl.RaftServerImpl: d57cb209-6a43-4e94-bc69-740186336f55: new RaftServerImpl for group-2EBDCA662DA4:[6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858, d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858, 0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858] with ContainerStateMachine:uninitialized
datanode_3 | 2019-11-16 00:36:50,851 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom)
s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
datanode_2 | 2019-11-16 00:36:50,864 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom)
scm_1 | WARNING: Illegal reflective access by org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil (file:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar) to field java.nio.Buffer.address
om_1 | INVALID_PART org.apache.hadoop.ozone.om.exceptions.OMException: Complete Multipart Upload Failed: volume: s37c35ba70126b04fca007e4dbfab9c47cbucket: bucket-11455key: multipartKey3
datanode_1 | 2019-11-16 00:36:50,835 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom)
datanode_3 | 2019-11-16 00:36:50,851 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom)
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)
datanode_2 | 2019-11-16 00:36:50,865 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom)
scm_1 | WARNING: Please consider reporting this to the maintainers of org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil
datanode_1 | 2019-11-16 00:36:50,836 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom)
om_1 | at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequest.validateAndUpdateCache(S3MultipartUploadCompleteRequest.java:146)
datanode_3 | 2019-11-16 00:36:50,851 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom)
datanode_2 | 2019-11-16 00:36:50,865 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default)
scm_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
datanode_1 | 2019-11-16 00:36:50,836 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom)
s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:217)
datanode_3 | 2019-11-16 00:36:50,851 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default)
datanode_2 | 2019-11-16 00:36:50,865 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom)
datanode_1 | 2019-11-16 00:36:50,836 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default)
scm_1 | WARNING: All illegal access operations will be denied in a future release
s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:132)
datanode_3 | 2019-11-16 00:36:50,851 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom)
datanode_2 | 2019-11-16 00:36:50,865 INFO impl.RaftServerImpl: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-2EBDCA662DA4: ConfigurationManager, init=-1: [6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858, d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858, 0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858], old=null, confs=<EMPTY_MAP>
datanode_1 | 2019-11-16 00:36:50,836 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom)
scm_1 | 2019-11-16 00:36:49,790 INFO pipeline.PipelineStateManager: Created pipeline Pipeline[ Id: 71b493d2-59e5-44fa-82f3-ad1b671d20b0, Nodes: d57cb209-6a43-4e94-bc69-740186336f55{ip: 172.19.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:ALLOCATED]
s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
om_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72)
datanode_2 | 2019-11-16 00:36:50,865 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom)
datanode_3 | 2019-11-16 00:36:50,851 INFO impl.RaftServerImpl: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-2EBDCA662DA4: ConfigurationManager, init=-1: [6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858, d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858, 0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858], old=null, confs=<EMPTY_MAP>
datanode_1 | 2019-11-16 00:36:50,836 INFO impl.RaftServerImpl: d57cb209-6a43-4e94-bc69-740186336f55@group-2EBDCA662DA4: ConfigurationManager, init=-1: [6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858, d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858, 0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858], old=null, confs=<EMPTY_MAP>
scm_1 | 2019-11-16 00:36:50,220 INFO pipeline.PipelineStateManager: Created pipeline Pipeline[ Id: 464a36a4-4722-4a03-b9b3-d11041da73d9, Nodes: 0e873e00-679f-460c-b6e9-cef3ff1ecd70{ip: 172.19.0.6, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:ALLOCATED]
s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100)
datanode_3 | 2019-11-16 00:36:50,852 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom)
datanode_2 | 2019-11-16 00:36:50,865 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default)
datanode_1 | 2019-11-16 00:36:50,837 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom)
scm_1 | 2019-11-16 00:36:50,793 INFO pipeline.PipelineStateManager: Created pipeline Pipeline[ Id: c2c4f6bf-e8c7-4e1b-a2cb-82ba2c1717ab, Nodes: 6f2abcfb-ae75-487f-9927-306aeedbcad3{ip: 172.19.0.5, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:ALLOCATED]
s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
om_1 | at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
datanode_3 | 2019-11-16 00:36:50,868 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default)
datanode_2 | 2019-11-16 00:36:50,865 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/a271516b-3e3f-4393-b225-2ebdca662da4 does not exist. Creating ...
datanode_1 | 2019-11-16 00:36:50,837 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default)
s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539)
om_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
datanode_3 | 2019-11-16 00:36:50,868 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/a271516b-3e3f-4393-b225-2ebdca662da4 does not exist. Creating ...
datanode_2 | 2019-11-16 00:36:50,889 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/a271516b-3e3f-4393-b225-2ebdca662da4/in_use.lock acquired by nodename 7@ab54c8aa3abc
datanode_1 | 2019-11-16 00:36:50,837 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/a271516b-3e3f-4393-b225-2ebdca662da4 does not exist. Creating ...
s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
scm_1 | 2019-11-16 00:36:50,927 INFO pipeline.PipelineStateManager: Created pipeline Pipeline[ Id: a271516b-3e3f-4393-b225-2ebdca662da4, Nodes: 0e873e00-679f-460c-b6e9-cef3ff1ecd70{ip: 172.19.0.6, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}d57cb209-6a43-4e94-bc69-740186336f55{ip: 172.19.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}6f2abcfb-ae75-487f-9927-306aeedbcad3{ip: 172.19.0.5, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:ALLOCATED]
om_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
datanode_3 | 2019-11-16 00:36:50,889 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/a271516b-3e3f-4393-b225-2ebdca662da4/in_use.lock acquired by nodename 7@70c5f52b4175
datanode_2 | 2019-11-16 00:36:50,912 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/a271516b-3e3f-4393-b225-2ebdca662da4 has been successfully formatted.
datanode_1 | 2019-11-16 00:36:50,869 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/a271516b-3e3f-4393-b225-2ebdca662da4/in_use.lock acquired by nodename 7@c29ed315f282
s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
scm_1 | 2019-11-16 00:36:55,068 INFO pipeline.PipelineStateManager: Pipeline Pipeline[ Id: 71b493d2-59e5-44fa-82f3-ad1b671d20b0, Nodes: d57cb209-6a43-4e94-bc69-740186336f55{ip: 172.19.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN] moved to OPEN state
datanode_3 | 2019-11-16 00:36:50,912 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/a271516b-3e3f-4393-b225-2ebdca662da4 has been successfully formatted.
om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
datanode_2 | 2019-11-16 00:36:50,913 INFO ratis.ContainerStateMachine: group-2EBDCA662DA4: The snapshot info is null. Setting the last applied indexto:(t:0, i:~)
datanode_1 | 2019-11-16 00:36:50,890 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/a271516b-3e3f-4393-b225-2ebdca662da4 has been successfully formatted.
s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
scm_1 | 2019-11-16 00:36:55,446 INFO pipeline.PipelineStateManager: Pipeline Pipeline[ Id: 464a36a4-4722-4a03-b9b3-d11041da73d9, Nodes: 0e873e00-679f-460c-b6e9-cef3ff1ecd70{ip: 172.19.0.6, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN] moved to OPEN state
datanode_3 | 2019-11-16 00:36:50,913 INFO ratis.ContainerStateMachine: group-2EBDCA662DA4: The snapshot info is null. Setting the last applied indexto:(t:0, i:~)
om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
datanode_2 | 2019-11-16 00:36:50,913 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom)
datanode_1 | 2019-11-16 00:36:50,891 INFO ratis.ContainerStateMachine: group-2EBDCA662DA4: The snapshot info is null. Setting the last applied indexto:(t:0, i:~)
s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
scm_1 | 2019-11-16 00:36:55,875 INFO pipeline.PipelineStateManager: Pipeline Pipeline[ Id: c2c4f6bf-e8c7-4e1b-a2cb-82ba2c1717ab, Nodes: 6f2abcfb-ae75-487f-9927-306aeedbcad3{ip: 172.19.0.5, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN] moved to OPEN state
datanode_3 | 2019-11-16 00:36:50,913 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom)
om_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
datanode_2 | 2019-11-16 00:36:50,913 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default)
datanode_1 | 2019-11-16 00:36:50,891 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom)
s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
scm_1 | 2019-11-16 00:37:01,223 INFO pipeline.PipelineStateManager: Pipeline Pipeline[ Id: a271516b-3e3f-4393-b225-2ebdca662da4, Nodes: 0e873e00-679f-460c-b6e9-cef3ff1ecd70{ip: 172.19.0.6, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}d57cb209-6a43-4e94-bc69-740186336f55{ip: 172.19.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}6f2abcfb-ae75-487f-9927-306aeedbcad3{ip: 172.19.0.5, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN] moved to OPEN state
datanode_3 | 2019-11-16 00:36:50,913 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default)
om_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
datanode_2 | 2019-11-16 00:36:50,913 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom)
datanode_1 | 2019-11-16 00:36:50,891 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default)
s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
datanode_3 | 2019-11-16 00:36:50,913 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom)
scm_1 | 2019-11-16 00:38:49,612 INFO server.SCMBlockProtocolServer: SCM is informed by OM to delete 3 blocks
om_1 | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
datanode_2 | 2019-11-16 00:36:50,914 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
datanode_1 | 2019-11-16 00:36:50,891 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom)
s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
datanode_3 | 2019-11-16 00:36:50,913 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
scm_1 | 2019-11-16 00:38:49,613 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144798058250367 bcsId: 0
om_1 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
datanode_2 | 2019-11-16 00:36:50,914 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom)
datanode_1 | 2019-11-16 00:36:50,891 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
datanode_3 | 2019-11-16 00:36:50,913 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom)
scm_1 | 2019-11-16 00:38:49,620 INFO block.BlockManagerImpl: Deleting blocks conID: 3 locID: 103144798146134144 bcsId: 0
om_1 | 2019-11-16 00:39:23,885 ERROR multipart.S3MultipartUploadCompleteRequest: Complete MultipartUpload failed for key /s37c35ba70126b04fca007e4dbfab9c47c/bucket-11455/multipartKey3 , MPU Key has no parts in OM, parts given to upload are [partNumber: 2
datanode_1 | 2019-11-16 00:36:50,891 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom)
datanode_2 | 2019-11-16 00:36:50,914 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom)
s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
datanode_3 | 2019-11-16 00:36:50,914 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom)
scm_1 | 2019-11-16 00:38:49,620 INFO block.BlockManagerImpl: Deleting blocks conID: 1 locID: 103144798271438977 bcsId: 0
datanode_1 | 2019-11-16 00:36:50,891 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom)
om_1 | partName: "etag1"
s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
datanode_2 | 2019-11-16 00:36:50,914 INFO segmented.SegmentedRaftLogWorker: new 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-2EBDCA662DA4-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/a271516b-3e3f-4393-b225-2ebdca662da4
scm_1 | 2019-11-16 00:39:49,679 INFO server.SCMBlockProtocolServer: SCM is informed by OM to delete 13 blocks
datanode_1 | 2019-11-16 00:36:50,891 INFO segmented.SegmentedRaftLogWorker: new d57cb209-6a43-4e94-bc69-740186336f55@group-2EBDCA662DA4-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/a271516b-3e3f-4393-b225-2ebdca662da4
datanode_3 | 2019-11-16 00:36:50,914 INFO segmented.SegmentedRaftLogWorker: new 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-2EBDCA662DA4-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/a271516b-3e3f-4393-b225-2ebdca662da4
om_1 | , partNumber: 1
s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834)
scm_1 | 2019-11-16 00:39:49,681 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144798497603714 bcsId: 0
datanode_2 | 2019-11-16 00:36:50,915 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom)
datanode_1 | 2019-11-16 00:36:50,892 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom)
om_1 | partName: "etag2"
s3g_1 | Caused by: java.lang.NullPointerException
scm_1 | 2019-11-16 00:39:49,682 INFO block.BlockManagerImpl: Deleting blocks conID: 1 locID: 103144799483265156 bcsId: 0
datanode_3 | 2019-11-16 00:36:50,914 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom)
datanode_2 | 2019-11-16 00:36:50,915 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom)
datanode_1 | 2019-11-16 00:36:50,892 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom)
om_1 | ]
scm_1 | 2019-11-16 00:39:49,682 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144799514394757 bcsId: 0
s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.createMultipartKey(ObjectEndpoint.java:576)
datanode_3 | 2019-11-16 00:36:50,914 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom)
datanode_2 | 2019-11-16 00:36:50,915 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom)
om_1 | 2019-11-16 00:39:23,886 ERROR multipart.S3MultipartUploadCompleteRequest: MultipartUpload Complete request failed for Key: multipartKey3 in Volume/Bucket s37c35ba70126b04fca007e4dbfab9c47c/bucket-11455
datanode_1 | 2019-11-16 00:36:50,892 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom)
scm_1 | 2019-11-16 00:39:49,682 INFO block.BlockManagerImpl: Deleting blocks conID: 3 locID: 103144799655297158 bcsId: 0
s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:136)
om_1 | INVALID_PART org.apache.hadoop.ozone.om.exceptions.OMException: Complete Multipart Upload Failed: volume: s37c35ba70126b04fca007e4dbfab9c47cbucket: bucket-11455key: multipartKey3
datanode_2 | 2019-11-16 00:36:50,915 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom)
datanode_3 | 2019-11-16 00:36:50,914 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom)
scm_1 | 2019-11-16 00:39:49,683 INFO block.BlockManagerImpl: Deleting blocks conID: 1 locID: 103144799739510919 bcsId: 0
s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
om_1 | at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequest.validateAndUpdateCache(S3MultipartUploadCompleteRequest.java:146)
datanode_3 | 2019-11-16 00:36:50,914 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom)
datanode_2 | 2019-11-16 00:36:50,915 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom)
datanode_1 | 2019-11-16 00:36:50,892 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom)
scm_1 | 2019-11-16 00:39:49,683 INFO block.BlockManagerImpl: Deleting blocks conID: 3 locID: 103144800467878025 bcsId: 0
om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:217)
datanode_3 | 2019-11-16 00:36:50,914 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom)
s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
datanode_2 | 2019-11-16 00:36:50,915 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default)
datanode_1 | 2019-11-16 00:36:50,892 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom)
scm_1 | 2019-11-16 00:39:49,684 INFO block.BlockManagerImpl: Deleting blocks conID: 1 locID: 103144800488784010 bcsId: 0
om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:132)
datanode_3 | 2019-11-16 00:36:50,914 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default)
s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
datanode_2 | 2019-11-16 00:36:50,915 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default)
datanode_1 | 2019-11-16 00:36:50,892 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default)
scm_1 | 2019-11-16 00:39:49,684 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144800625557643 bcsId: 0
om_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72)
datanode_3 | 2019-11-16 00:36:50,915 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default)
s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566)
datanode_2 | 2019-11-16 00:36:50,916 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default)
datanode_1 | 2019-11-16 00:36:50,892 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default)
scm_1 | 2019-11-16 00:39:49,684 INFO block.BlockManagerImpl: Deleting blocks conID: 3 locID: 103144800762200204 bcsId: 0
om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100)
datanode_3 | 2019-11-16 00:36:50,915 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default)
s3g_1 | at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
datanode_1 | 2019-11-16 00:36:50,892 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default)
datanode_2 | 2019-11-16 00:36:50,916 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default)
scm_1 | 2019-11-16 00:39:49,684 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144801225474190 bcsId: 0
om_1 | at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
datanode_3 | 2019-11-16 00:36:50,915 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default)
s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
datanode_1 | 2019-11-16 00:36:50,892 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default)
datanode_2 | 2019-11-16 00:36:50,916 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom)
scm_1 | 2019-11-16 00:39:49,685 INFO block.BlockManagerImpl: Deleting blocks conID: 3 locID: 103144801258897551 bcsId: 0
om_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
datanode_3 | 2019-11-16 00:36:50,915 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom)
s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
datanode_1 | 2019-11-16 00:36:50,892 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom)
datanode_2 | 2019-11-16 00:36:50,916 INFO segmented.SegmentedRaftLogWorker: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-2EBDCA662DA4-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1
scm_1 | 2019-11-16 00:39:49,685 INFO block.BlockManagerImpl: Deleting blocks conID: 1 locID: 103144801411072144 bcsId: 0
om_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
datanode_3 | 2019-11-16 00:36:50,915 INFO segmented.SegmentedRaftLogWorker: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-2EBDCA662DA4-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1
s3g_1 | at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
datanode_1 | 2019-11-16 00:36:50,893 INFO segmented.SegmentedRaftLogWorker: d57cb209-6a43-4e94-bc69-740186336f55@group-2EBDCA662DA4-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1
datanode_2 | 2019-11-16 00:36:50,917 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom)
scm_1 | 2019-11-16 00:39:49,685 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144801638416529 bcsId: 0
om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
datanode_3 | 2019-11-16 00:36:50,916 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom)
s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
datanode_1 | 2019-11-16 00:36:50,893 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom)
datanode_2 | 2019-11-16 00:36:50,917 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom)
om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
scm_1 | 2019-11-16 00:40:49,718 INFO server.SCMBlockProtocolServer: SCM is informed by OM to delete 25 blocks
datanode_3 | 2019-11-16 00:36:50,916 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom)
s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
datanode_1 | 2019-11-16 00:36:50,894 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom)
datanode_2 | 2019-11-16 00:36:50,917 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom)
om_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
scm_1 | 2019-11-16 00:40:49,718 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144805741166764 bcsId: 0
datanode_3 | 2019-11-16 00:36:50,916 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom)
s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
datanode_1 | 2019-11-16 00:36:50,894 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom)
datanode_2 | 2019-11-16 00:36:50,917 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom)
om_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
scm_1 | 2019-11-16 00:40:49,718 INFO block.BlockManagerImpl: Deleting blocks conID: 3 locID: 103144805822169261 bcsId: 0
datanode_3 | 2019-11-16 00:36:50,916 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom)
s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
datanode_1 | 2019-11-16 00:36:50,894 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom)
datanode_2 | 2019-11-16 00:36:50,917 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
om_1 | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
scm_1 | 2019-11-16 00:40:49,719 INFO block.BlockManagerImpl: Deleting blocks conID: 1 locID: 103144805995249838 bcsId: 0
datanode_3 | 2019-11-16 00:36:50,916 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
datanode_1 | 2019-11-16 00:36:50,894 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
datanode_2 | 2019-11-16 00:36:50,917 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
om_1 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
scm_1 | 2019-11-16 00:40:49,719 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144806326206639 bcsId: 0
datanode_3 | 2019-11-16 00:36:50,916 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
datanode_1 | 2019-11-16 00:36:50,894 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
datanode_2 | 2019-11-16 00:36:50,918 INFO impl.RaftServerImpl: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-2EBDCA662DA4: start as a follower, conf=-1: [6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858, d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858, 0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858], old=null
om_1 | 2019-11-16 00:39:41,684 ERROR multipart.S3MultipartUploadAbortRequest: Abort Multipart request is failed for KeyName multipartKey5 in VolumeName/Bucket s37c35ba70126b04fca007e4dbfab9c47c/bucket-11455
datanode_3 | 2019-11-16 00:36:50,917 INFO impl.RaftServerImpl: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-2EBDCA662DA4: start as a follower, conf=-1: [6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858, d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858, 0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858], old=null
datanode_1 | 2019-11-16 00:36:50,895 INFO impl.RaftServerImpl: d57cb209-6a43-4e94-bc69-740186336f55@group-2EBDCA662DA4: start as a follower, conf=-1: [6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858, d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858, 0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858], old=null
s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
scm_1 | 2019-11-16 00:40:49,719 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144804646846630 bcsId: 0
datanode_2 | 2019-11-16 00:36:50,918 INFO impl.RaftServerImpl: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-2EBDCA662DA4: changes role from null to FOLLOWER at term 0 for startAsFollower
om_1 | NO_SUCH_MULTIPART_UPLOAD_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Abort Multipart Upload Failed: volume: s37c35ba70126b04fca007e4dbfab9c47cbucket: bucket-11455key: multipartKey5
datanode_3 | 2019-11-16 00:36:50,917 INFO impl.RaftServerImpl: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-2EBDCA662DA4: changes role from null to FOLLOWER at term 0 for startAsFollower
datanode_1 | 2019-11-16 00:36:50,895 INFO impl.RaftServerImpl: d57cb209-6a43-4e94-bc69-740186336f55@group-2EBDCA662DA4: changes role from null to FOLLOWER at term 0 for startAsFollower
scm_1 | 2019-11-16 00:40:49,719 INFO block.BlockManagerImpl: Deleting blocks conID: 3 locID: 103144804696391847 bcsId: 0
s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
datanode_2 | 2019-11-16 00:36:50,918 INFO impl.RoleInfo: 0e873e00-679f-460c-b6e9-cef3ff1ecd70: start FollowerState
om_1 | at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadAbortRequest.validateAndUpdateCache(S3MultipartUploadAbortRequest.java:116)
datanode_3 | 2019-11-16 00:36:50,917 INFO impl.RoleInfo: 6f2abcfb-ae75-487f-9927-306aeedbcad3: start FollowerState
datanode_1 | 2019-11-16 00:36:50,895 INFO impl.RoleInfo: d57cb209-6a43-4e94-bc69-740186336f55: start FollowerState
s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
datanode_2 | 2019-11-16 00:36:50,919 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-2EBDCA662DA4,id=0e873e00-679f-460c-b6e9-cef3ff1ecd70
om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:217)
scm_1 | 2019-11-16 00:40:49,719 INFO block.BlockManagerImpl: Deleting blocks conID: 1 locID: 103144804787617960 bcsId: 0
datanode_3 | 2019-11-16 00:36:50,917 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-2EBDCA662DA4,id=6f2abcfb-ae75-487f-9927-306aeedbcad3
datanode_1 | 2019-11-16 00:36:50,896 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-2EBDCA662DA4,id=d57cb209-6a43-4e94-bc69-740186336f55
s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
datanode_2 | 2019-11-16 00:36:50,919 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:132)
scm_1 | 2019-11-16 00:40:49,720 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144804850270377 bcsId: 0
datanode_3 | 2019-11-16 00:36:50,917 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
datanode_1 | 2019-11-16 00:36:50,896 WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
datanode_2 | 2019-11-16 00:36:55,339 INFO impl.FollowerState: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-D11041DA73D9-FollowerState: change to CANDIDATE, lastRpcTime:5184ms, electionTimeout:5184ms
om_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72)
scm_1 | 2019-11-16 00:40:49,720 INFO block.BlockManagerImpl: Deleting blocks conID: 3 locID: 103144803467788440 bcsId: 0
datanode_3 | 2019-11-16 00:36:55,750 INFO impl.FollowerState: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-82BA2C1717AB-FollowerState: change to CANDIDATE, lastRpcTime:5029ms, electionTimeout:5028ms
s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
datanode_2 | 2019-11-16 00:36:55,341 INFO impl.RoleInfo: 0e873e00-679f-460c-b6e9-cef3ff1ecd70: shutdown FollowerState
scm_1 | 2019-11-16 00:40:49,720 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144803469295770 bcsId: 0
om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100)
datanode_1 | 2019-11-16 00:36:54,886 INFO impl.FollowerState: d57cb209-6a43-4e94-bc69-740186336f55@group-AD1B671D20B0-FollowerState: change to CANDIDATE, lastRpcTime:5178ms, electionTimeout:5178ms
datanode_3 | 2019-11-16 00:36:55,752 INFO impl.RoleInfo: 6f2abcfb-ae75-487f-9927-306aeedbcad3: shutdown FollowerState
s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
datanode_2 | 2019-11-16 00:36:55,341 INFO impl.RaftServerImpl: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-D11041DA73D9: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate
om_1 | at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
scm_1 | 2019-11-16 00:40:49,721 INFO block.BlockManagerImpl: Deleting blocks conID: 1 locID: 103144803469295769 bcsId: 0
datanode_1 | 2019-11-16 00:36:54,888 INFO impl.RoleInfo: d57cb209-6a43-4e94-bc69-740186336f55: shutdown FollowerState
datanode_3 | 2019-11-16 00:36:55,753 INFO impl.RaftServerImpl: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-82BA2C1717AB: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate
s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
datanode_2 | 2019-11-16 00:36:55,344 INFO impl.RoleInfo: 0e873e00-679f-460c-b6e9-cef3ff1ecd70: start LeaderElection
om_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
scm_1 | 2019-11-16 00:40:49,721 INFO block.BlockManagerImpl: Deleting blocks conID: 3 locID: 103144803519299739 bcsId: 0
datanode_1 | 2019-11-16 00:36:54,889 INFO impl.RaftServerImpl: d57cb209-6a43-4e94-bc69-740186336f55@group-AD1B671D20B0: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate
datanode_3 | 2019-11-16 00:36:55,757 INFO impl.RoleInfo: 6f2abcfb-ae75-487f-9927-306aeedbcad3: start LeaderElection
s3g_1 | ... 33 more
om_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
datanode_2 | 2019-11-16 00:36:55,366 INFO impl.LeaderElection: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-D11041DA73D9-LeaderElection1: begin an election at term 1 for -1: [0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858], old=null
scm_1 | 2019-11-16 00:40:49,721 INFO block.BlockManagerImpl: Deleting blocks conID: 1 locID: 103144803558817948 bcsId: 0
datanode_1 | 2019-11-16 00:36:54,893 INFO impl.RoleInfo: d57cb209-6a43-4e94-bc69-740186336f55: start LeaderElection
datanode_3 | 2019-11-16 00:36:55,773 INFO impl.LeaderElection: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-82BA2C1717AB-LeaderElection1: begin an election at term 1 for -1: [6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858], old=null
s3g_1 | 2019-11-16 00:38:36,965 WARN server.HttpChannel: //s3g:9878/bucket-11455/multipartKey?partNumber=1&uploadId=7dfac5db-859b-4410-8f20-06bfde1a8a77-103144798004707456
om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
datanode_2 | 2019-11-16 00:36:55,367 INFO impl.RoleInfo: 0e873e00-679f-460c-b6e9-cef3ff1ecd70: shutdown LeaderElection
scm_1 | 2019-11-16 00:40:49,722 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144803562619037 bcsId: 0
datanode_1 | 2019-11-16 00:36:54,955 INFO impl.LeaderElection: d57cb209-6a43-4e94-bc69-740186336f55@group-AD1B671D20B0-LeaderElection1: begin an election at term 1 for -1: [d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858], old=null
datanode_3 | 2019-11-16 00:36:55,775 INFO impl.RoleInfo: 6f2abcfb-ae75-487f-9927-306aeedbcad3: shutdown LeaderElection
s3g_1 | javax.servlet.ServletException: javax.servlet.ServletException: java.lang.NullPointerException
om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
datanode_2 | 2019-11-16 00:36:55,367 INFO impl.RaftServerImpl: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-D11041DA73D9: changes role from CANDIDATE to LEADER at term 1 for changeToLeader
scm_1 | 2019-11-16 00:40:49,722 INFO block.BlockManagerImpl: Deleting blocks conID: 3 locID: 103144803568582814 bcsId: 0
datanode_1 | 2019-11-16 00:36:54,957 INFO impl.RoleInfo: d57cb209-6a43-4e94-bc69-740186336f55: shutdown LeaderElection
datanode_3 | 2019-11-16 00:36:55,776 INFO impl.RaftServerImpl: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-82BA2C1717AB: changes role from CANDIDATE to LEADER at term 1 for changeToLeader
s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:139)
om_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
datanode_2 | 2019-11-16 00:36:55,368 INFO ratis.XceiverServerRatis: Leader change notification received for group: group-D11041DA73D9 with new leaderId: 0e873e00-679f-460c-b6e9-cef3ff1ecd70
scm_1 | 2019-11-16 00:40:49,722 INFO block.BlockManagerImpl: Deleting blocks conID: 1 locID: 103144803606200479 bcsId: 0
datanode_1 | 2019-11-16 00:36:54,958 INFO impl.RaftServerImpl: d57cb209-6a43-4e94-bc69-740186336f55@group-AD1B671D20B0: changes role from CANDIDATE to LEADER at term 1 for changeToLeader
datanode_3 | 2019-11-16 00:36:55,776 INFO ratis.XceiverServerRatis: Leader change notification received for group: group-82BA2C1717AB with new leaderId: 6f2abcfb-ae75-487f-9927-306aeedbcad3
s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
om_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
datanode_2 | 2019-11-16 00:36:55,372 INFO impl.RaftServerImpl: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-D11041DA73D9: change Leader from null to 0e873e00-679f-460c-b6e9-cef3ff1ecd70 at term 1 for becomeLeader, leader elected after 5267ms
scm_1 | 2019-11-16 00:40:49,723 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144803665576096 bcsId: 0
datanode_1 | 2019-11-16 00:36:54,959 INFO ratis.XceiverServerRatis: Leader change notification received for group: group-AD1B671D20B0 with new leaderId: d57cb209-6a43-4e94-bc69-740186336f55
datanode_3 | 2019-11-16 00:36:55,783 INFO impl.RaftServerImpl: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-82BA2C1717AB: change Leader from null to 6f2abcfb-ae75-487f-9927-306aeedbcad3 at term 1 for becomeLeader, leader elected after 5136ms
s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539)
om_1 | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
datanode_2 | 2019-11-16 00:36:55,376 INFO server.RaftServerConfigKeys: raft.server.staging.catchup.gap = 1000 (default)
scm_1 | 2019-11-16 00:40:49,723 INFO block.BlockManagerImpl: Deleting blocks conID: 3 locID: 103144803686875297 bcsId: 0
datanode_1 | 2019-11-16 00:36:54,967 INFO impl.RaftServerImpl: d57cb209-6a43-4e94-bc69-740186336f55@group-AD1B671D20B0: change Leader from null to d57cb209-6a43-4e94-bc69-740186336f55 at term 1 for becomeLeader, leader elected after 5317ms
datanode_3 | 2019-11-16 00:36:55,788 INFO server.RaftServerConfigKeys: raft.server.staging.catchup.gap = 1000 (default)
s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
om_1 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
datanode_2 | 2019-11-16 00:36:55,376 INFO server.RaftServerConfigKeys: raft.server.rpc.sleep.time = 25ms (default)
scm_1 | 2019-11-16 00:40:49,723 INFO block.BlockManagerImpl: Deleting blocks conID: 1 locID: 103144803753394338 bcsId: 0
datanode_1 | 2019-11-16 00:36:54,974 INFO server.RaftServerConfigKeys: raft.server.staging.catchup.gap = 1000 (default)
datanode_3 | 2019-11-16 00:36:55,788 INFO server.RaftServerConfigKeys: raft.server.rpc.sleep.time = 25ms (default)
s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
om_1 | 2019-11-16 00:39:45,319 ERROR key.OMKeyRequest: ALLOCATE_KEY failed for Key: multipartKey in volume/bucket:s37c35ba70126b04fca007e4dbfab9c47c/bucket-11455
datanode_2 | 2019-11-16 00:36:55,379 INFO server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default)
scm_1 | 2019-11-16 00:40:49,724 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144803849601187 bcsId: 0
datanode_1 | 2019-11-16 00:36:54,974 INFO server.RaftServerConfigKeys: raft.server.rpc.sleep.time = 25ms (default)
datanode_3 | 2019-11-16 00:36:55,792 INFO server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default)
s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
om_1 | NO_SUCH_MULTIPART_UPLOAD_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: No such Multipart upload is with specified uploadId random
datanode_2 | 2019-11-16 00:36:55,382 INFO server.RaftServerConfigKeys: raft.server.watch.timeout = 10s (default)
scm_1 | 2019-11-16 00:40:49,724 INFO block.BlockManagerImpl: Deleting blocks conID: 3 locID: 103144803861069988 bcsId: 0
datanode_1 | 2019-11-16 00:36:54,978 INFO server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default)
datanode_3 | 2019-11-16 00:36:55,797 INFO server.RaftServerConfigKeys: raft.server.watch.timeout = 10s (default)
s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
om_1 | at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.prepareMultipartKeyInfo(OMKeyRequest.java:471)
datanode_2 | 2019-11-16 00:36:55,383 INFO server.RaftServerConfigKeys: raft.server.watch.timeout.denomination = 1s (default)
scm_1 | 2019-11-16 00:40:49,724 INFO block.BlockManagerImpl: Deleting blocks conID: 1 locID: 103144802865053843 bcsId: 0
datanode_1 | 2019-11-16 00:36:54,985 INFO server.RaftServerConfigKeys: raft.server.watch.timeout = 10s (default)
datanode_3 | 2019-11-16 00:36:55,798 INFO server.RaftServerConfigKeys: raft.server.watch.timeout.denomination = 1s (default)
s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
om_1 | at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.prepareKeyInfo(OMKeyRequest.java:423)
datanode_2 | 2019-11-16 00:36:55,383 INFO server.RaftServerConfigKeys: raft.server.watch.element-limit = 65536 (default)
scm_1 | 2019-11-16 00:40:49,725 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144802895134868 bcsId: 0
datanode_1 | 2019-11-16 00:36:54,985 INFO server.RaftServerConfigKeys: raft.server.watch.timeout.denomination = 1s (default)
datanode_3 | 2019-11-16 00:36:55,799 INFO server.RaftServerConfigKeys: raft.server.watch.element-limit = 65536 (default)
s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
om_1 | at org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:182)
datanode_2 | 2019-11-16 00:36:55,389 INFO impl.RoleInfo: 0e873e00-679f-460c-b6e9-cef3ff1ecd70: start LeaderState
scm_1 | 2019-11-16 00:40:49,725 INFO block.BlockManagerImpl: Deleting blocks conID: 3 locID: 103144802946449557 bcsId: 0
datanode_1 | 2019-11-16 00:36:54,986 INFO server.RaftServerConfigKeys: raft.server.watch.element-limit = 65536 (default)
datanode_3 | 2019-11-16 00:36:55,805 INFO impl.RoleInfo: 6f2abcfb-ae75-487f-9927-306aeedbcad3: start LeaderState
s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:217)
datanode_2 | 2019-11-16 00:36:55,405 INFO segmented.SegmentedRaftLogWorker: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-D11041DA73D9-SegmentedRaftLogWorker: Starting segment from index:0
scm_1 | 2019-11-16 00:40:49,726 INFO block.BlockManagerImpl: Deleting blocks conID: 1 locID: 103144803127722134 bcsId: 0
datanode_1 | 2019-11-16 00:36:54,996 INFO impl.RoleInfo: d57cb209-6a43-4e94-bc69-740186336f55: start LeaderState
datanode_3 | 2019-11-16 00:36:55,827 INFO segmented.SegmentedRaftLogWorker: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-82BA2C1717AB-SegmentedRaftLogWorker: Starting segment from index:0
s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:132)
datanode_2 | 2019-11-16 00:36:55,414 INFO impl.RaftServerImpl: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-D11041DA73D9: set configuration 0: [0e873e00-679f-460c-b6e9-cef3ff1ecd70:172.19.0.6:9858], old=null at 0
scm_1 | 2019-11-16 00:41:48,622 INFO container.ReplicationManager: Starting Replication Monitor Thread.
datanode_1 | 2019-11-16 00:36:55,019 INFO segmented.SegmentedRaftLogWorker: d57cb209-6a43-4e94-bc69-740186336f55@group-AD1B671D20B0-SegmentedRaftLogWorker: Starting segment from index:0
datanode_3 | 2019-11-16 00:36:55,837 INFO impl.RaftServerImpl: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-82BA2C1717AB: set configuration 0: [6f2abcfb-ae75-487f-9927-306aeedbcad3:172.19.0.5:9858], old=null at 0
s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
datanode_2 | 2019-11-16 00:36:55,518 INFO segmented.SegmentedRaftLogWorker: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-D11041DA73D9-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/464a36a4-4722-4a03-b9b3-d11041da73d9/current/log_inprogress_0
scm_1 | 2019-11-16 00:41:48,634 INFO container.ReplicationManager: Replication Monitor Thread took 8 milliseconds for processing 3 containers.
datanode_3 | 2019-11-16 00:36:55,938 INFO segmented.SegmentedRaftLogWorker: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-82BA2C1717AB-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/c2c4f6bf-e8c7-4e1b-a2cb-82ba2c1717ab/current/log_inprogress_0
datanode_1 | 2019-11-16 00:36:55,032 INFO impl.RaftServerImpl: d57cb209-6a43-4e94-bc69-740186336f55@group-AD1B671D20B0: set configuration 0: [d57cb209-6a43-4e94-bc69-740186336f55:172.19.0.4:9858], old=null at 0
s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
om_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72)
datanode_2 | 2019-11-16 00:36:55,942 INFO impl.FollowerState: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-2EBDCA662DA4-FollowerState: change to CANDIDATE, lastRpcTime:5023ms, electionTimeout:5023ms
scm_1 | 2019-11-16 00:41:49,768 INFO server.SCMBlockProtocolServer: SCM is informed by OM to delete 3 blocks
datanode_1 | 2019-11-16 00:36:55,137 INFO segmented.SegmentedRaftLogWorker: d57cb209-6a43-4e94-bc69-740186336f55@group-AD1B671D20B0-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/71b493d2-59e5-44fa-82f3-ad1b671d20b0/current/log_inprogress_0
s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834)
om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100)
datanode_3 | 2019-11-16 00:36:56,083 INFO impl.FollowerState: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-2EBDCA662DA4-FollowerState: change to CANDIDATE, lastRpcTime:5166ms, electionTimeout:5166ms
datanode_2 | 2019-11-16 00:36:55,942 INFO impl.RoleInfo: 0e873e00-679f-460c-b6e9-cef3ff1ecd70: shutdown FollowerState
scm_1 | 2019-11-16 00:41:49,768 INFO block.BlockManagerImpl: Deleting blocks conID: 1 locID: 103144808866185396 bcsId: 0
datanode_1 | 2019-11-16 00:36:56,057 INFO impl.FollowerState: d57cb209-6a43-4e94-bc69-740186336f55@group-2EBDCA662DA4-FollowerState: change to CANDIDATE, lastRpcTime:5161ms, electionTimeout:5161ms
s3g_1 | Caused by: javax.servlet.ServletException: java.lang.NullPointerException
om_1 | at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
datanode_3 | 2019-11-16 00:36:56,084 INFO impl.RoleInfo: 6f2abcfb-ae75-487f-9927-306aeedbcad3: shutdown FollowerState
scm_1 | 2019-11-16 00:41:49,768 INFO block.BlockManagerImpl: Deleting blocks conID: 2 locID: 103144809578234037 bcsId: 0
datanode_1 | 2019-11-16 00:36:56,057 INFO impl.RoleInfo: d57cb209-6a43-4e94-bc69-740186336f55: shutdown FollowerState
s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432)
datanode_2 | 2019-11-16 00:36:55,943 INFO impl.RaftServerImpl: 0e873e00-679f-460c-b6e9-cef3ff1ecd70@group-2EBDCA662DA4: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate
datanode_3 | 2019-11-16 00:36:56,084 INFO impl.RaftServerImpl: 6f2abcfb-ae75-487f-9927-306aeedbcad3@group-2EBDCA662DA4: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate
scm_1 | 2019-11-16 00:41:49,769 INFO block.BlockManagerImpl: Deleting blocks conID: 3 locID: 103144810003693750 bcsId: 0
om_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)