Release 2.8+ remote storage doesn't work on ext4 bare metal, running RH7 #5424

aned · Apr 1, 2019

Remote write doesn't work on fs type ext4 on bare metal running RH7 after “ [ENHANCEMENT] Use the WAL for remote_write API. #4588”.

Getting this in logs:

Mar 30 00:44:15 serverA prometheus[16760]: ts=2019-03-30T00:44:14.355978792Z caller=dedupe.go:111 component=remote level=info queue=0:http://localhost:7201/api/v1/prom/remote/write msg="dropped sample for series that was not explicitly dropped via relabelling" ref=479450336

Remote read works fine on 2.8+

Downgraded to 2.7.2 - no issues with remote write.

Relevant config:

remote_write:
  - url: "http://localhost:7201/api/v1/prom/remote/write"
    write_relabel_configs:
     - source_labels: [__name__]
       regex: 'job:.*'
       action: keep
     - source_labels: [__name__]
       regex: '.*Garbage.*'
       action: drop
    queue_config:
      capacity: 30000

100% drop rate:

tomwilkie · Apr 1, 2019

Hi @aned, sorry you're having issues with this.

Can you post the logs from the startup of your Prometheus please?

Also, can you post us screen shots of the following queries?

rate(prometheus_tsdb_head_samples_appended_total[1m])
rate(prometheus_remote_storage_samples_in_total[1m])
rate(prometheus_remote_storage_dropped_samples_total[1m])
prometheus_wal_watcher_current_segment
rate(prometheus_wal_watcher_records_read_total[1m])

I suspect this might be the same issue as #5389.

aned · Apr 1, 2019

Mar 29 23:49:19 server_A systemd[1]: Stopped Prometheus Server.
Mar 29 23:50:41 server_A systemd[1]: Started Prometheus Server.
Mar 29 23:50:41 server_A systemd[1]: Starting Prometheus Server...
Mar 29 23:50:41 server_A prometheus[16760]: level=warn ts=2019-03-29T23:50:41.765322949Z caller=main.go:274 deprecation_notice="'storage.tsdb.retention' flag is deprecated use 'storage.tsdb.retention.time' instead."
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.76540597Z caller=main.go:321 msg="Starting Prometheus" version="(version=2.8.1, branch=HEAD, revision=4d60eb36dcbed725fcac5b27018574118f12fffb)"
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.765444129Z caller=main.go:322 build_context="(go=go1.11.6, user=root@bfdd6a22a683, date=20190328-18:04:08)"
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.765480394Z caller=main.go:323 host_details="(Linux 3.10.0-514.36.5.el7.x86_64 #1 SMP Thu Dec 28 21:42:18 EST 2017 x86_64 server_A (none))"
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.765525292Z caller=main.go:324 fd_limits="(soft=1000000, hard=1000000)"
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.76555986Z caller=main.go:325 vm_limits="(soft=unlimited, hard=unlimited)"
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.766561706Z caller=main.go:640 msg="Starting TSDB ..."
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.766612912Z caller=web.go:418 component=web msg="Start listening for connections" address=:7320
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.766943871Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1549368000000 maxt=1549562400000 ulid=01D34XA3M71Y1E5DPQWN0YVA6X
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767034946Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1549562400000 maxt=1549756800000 ulid=01D3APPMXJ3270QP939STYDQYH
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767098098Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1549756800000 maxt=1549951200000 ulid=01D3GG3ZBWF3TN1JJTFXKP48FC
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767157488Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1549951200000 maxt=1550145600000 ulid=01D3P9GZH3S2GYX3CSW9M1JZQH
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767213116Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1550145600000 maxt=1550340000000 ulid=01D3W2XKCYVQGTED149CZTMC5J
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767267051Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1550340000000 maxt=1550534400000 ulid=01D41WAFGFC3HJCZDCEZ1YJDFY
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767318439Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1550534400000 maxt=1550728800000 ulid=01D47NQ01T2S552WPA10Q6KSK7
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767375546Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1550728800000 maxt=1550923200000 ulid=01D4DF4R5JKS0H3GT2YSF3WZA6
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767427709Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1550923200000 maxt=1551117600000 ulid=01D4K8H6K5JA31RG7QG5PAW4G7
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767482572Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1551117600000 maxt=1551312000000 ulid=01D4S1XF5VYE8W390NZK3HA54C
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767542511Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1551312000000 maxt=1551506400000 ulid=01D4YVA74MD82442FRPF3J97FN
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767604612Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1551506400000 maxt=1551700800000 ulid=01D54MP42HKQ4HCPCCFQXKPEQA
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767656551Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1551700800000 maxt=1551895200000 ulid=01D5AE4XKAMSD26N573QAZNQE9
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767715094Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1551895200000 maxt=1552089600000 ulid=01D5G7HGA0PB61X7838FJC5GYX
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767768036Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1552089600000 maxt=1552284000000 ulid=01D5P0XFGMRWGEXGYFP8M4RHV2
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767826357Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1552284000000 maxt=1552478400000 ulid=01D5VT8VQMX6C3BBD4B49A4RV2
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767878051Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1552478400000 maxt=1552672800000 ulid=01D61KNF9WAY9V3Y4FEM091CNW
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767930352Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1552672800000 maxt=1552867200000 ulid=01D67D28PTAB6JZVCZ8DWSZ3ZY
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.767986092Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1552867200000 maxt=1553061600000 ulid=01D6D6ERCPQD1D3KEW68JA15Z6
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.768043114Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1553061600000 maxt=1553256000000 ulid=01D6JZVECYBF628WFYVEP72PTS
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.768103039Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1553256000000 maxt=1553450400000 ulid=01D6RS7CFN1Q449KGPSEVJBKEP
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.768156464Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1553450400000 maxt=1553644800000 ulid=01D6YJKYSCBPZQ6CB9FYKP5GHK
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.76820942Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1553644800000 maxt=1553839200000 ulid=01D74C0YA1THXD8HRBQ6AWF0HS
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.768250873Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1553839200000 maxt=1553860800000 ulid=01D74ZZ51HYJ5XT7SVGAZZF5N1
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.768281941Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1553882400000 maxt=1553889600000 ulid=01D75M1446DK83H1PVC4K9R9ER
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.76832314Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1553860800000 maxt=1553882400000 ulid=01D75MJEQ990209K8JM6MVXAF3
Mar 29 23:50:41 server_A prometheus[16760]: level=info ts=2019-03-29T23:50:41.768354348Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1553889600000 maxt=1553896800000 ulid=01D75TWVC6GBK5EG54SMC2DNKP
Mar 29 23:57:27 server_A prometheus[16760]: level=warn ts=2019-03-29T23:57:27.680607312Z caller=head.go:450 component=tsdb msg="unknown series references" count=59855
Mar 29 23:57:38 server_A prometheus[16760]: level=info ts=2019-03-29T23:57:38.943430941Z caller=main.go:655 msg="TSDB started"
Mar 29 23:57:38 server_A prometheus[16760]: level=info ts=2019-03-29T23:57:38.94351575Z caller=main.go:724 msg="Loading configuration file" filename=/export/apps/prometheus/etc/prometheus.yml
Mar 29 23:57:38 server_A prometheus[16760]: ts=2019-03-29T23:57:38.946347956Z caller=dedupe.go:111 component=remote level=info queue=0:http://localhost:7201/api/v1/prom/remote/write msg="starting WAL watcher" queue=0:http://localhost:7201/api/v1/prom/remote/write
Mar 29 23:57:38 server_A prometheus[16760]: level=info ts=2019-03-29T23:57:38.987885196Z caller=main.go:751 msg="Completed loading of configuration file" filename=/export/apps/prometheus/etc/prometheus.yml
Mar 29 23:57:38 server_A prometheus[16760]: level=info ts=2019-03-29T23:57:38.987935847Z caller=main.go:609 msg="Server is ready to receive web requests."
Mar 29 23:58:48 server_A prometheus[16760]: ts=2019-03-29T23:58:48.963084404Z caller=dedupe.go:111 component=remote level=info queue=0:http://localhost:7201/api/v1/prom/remote/write msg="dropped sample for series that was not explicitly dropped via relabelling" ref=495005343
Mar 29 23:58:48 server_A prometheus[16760]: ts=2019-03-29T23:58:48.963226851Z caller=dedupe.go:111 component=remote level=info queue=0:http://localhost:7201/api/v1/prom/remote/write msg="dropped sample for series that was not explicitly dropped via relabelling" ref=488707286
Mar 29 23:58:48 server_A prometheus[16760]: ts=2019-03-29T23:58:48.963303215Z caller=dedupe.go:111 component=remote level=info queue=0:http://localhost:7201/api/v1/prom/remote/write msg="dropped sample for series that was not explicitly dropped via relabelling" ref=488709409
Mar 29 23:58:48 server_A prometheus[16760]: ts=2019-03-29T23:58:48.963323215Z caller=dedupe.go:111 component=remote level=info queue=0:http://localhost:7201/api/v1/prom/remote/write msg="dropped sample for series that was not explicitly dropped via relabelling" ref=488709410
Mar 29 23:58:48 server_A prometheus[16760]: ts=2019-03-29T23:58:48.963338419Z caller=dedupe.go:111 component=remote level=info queue=0:http://localhost:7201/api/v1/prom/remote/write msg="dropped sample for series that was not explicitly dropped via relabelling" ref=489004867

brian-brazil · Apr 1, 2019

Can you share a ls -l on the wal directory?

aned · Apr 1, 2019

Yep,

[wal]$ ls -l
total 58392296
-rw-r--r-- 1 app app 134217728 Apr  1 17:05 00386578
-rw-r--r-- 1 app app 134217728 Apr  1 17:05 00386579
-rw-r--r-- 1 app app 134184960 Apr  1 17:06 00386580
-rw-r--r-- 1 app app 134152192 Apr  1 17:07 00386581
-rw-r--r-- 1 app app 134119424 Apr  1 17:07 00386582
-rw-r--r-- 1 app app 134184960 Apr  1 17:08 00386583
-rw-r--r-- 1 app app 134184960 Apr  1 17:08 00386584
-rw-r--r-- 1 app app 134152192 Apr  1 17:09 00386585
-rw-r--r-- 1 app app 134217728 Apr  1 17:10 00386586
-rw-r--r-- 1 app app 134184960 Apr  1 17:10 00386587
-rw-r--r-- 1 app app 134217728 Apr  1 17:11 00386588
-rw-r--r-- 1 app app 134184960 Apr  1 17:11 00386589
-rw-r--r-- 1 app app 131760128 Apr  1 17:12 00386590
-rw-r--r-- 1 app app 132579328 Apr  1 17:13 00386591
-rw-r--r-- 1 app app 134217728 Apr  1 17:13 00386592
-rw-r--r-- 1 app app 134152192 Apr  1 17:14 00386593
-rw-r--r-- 1 app app 134152192 Apr  1 17:14 00386594
-rw-r--r-- 1 app app 134152192 Apr  1 17:15 00386595
-rw-r--r-- 1 app app 134217728 Apr  1 17:16 00386596
-rw-r--r-- 1 app app 134184960 Apr  1 17:16 00386597
-rw-r--r-- 1 app app 134184960 Apr  1 17:17 00386598
-rw-r--r-- 1 app app 134217728 Apr  1 17:17 00386599
-rw-r--r-- 1 app app 134217728 Apr  1 17:18 00386600
-rw-r--r-- 1 app app 134217728 Apr  1 17:19 00386601
-rw-r--r-- 1 app app 134184960 Apr  1 17:19 00386602
-rw-r--r-- 1 app app 130940928 Apr  1 17:20 00386603
-rw-r--r-- 1 app app 134217728 Apr  1 17:20 00386604
-rw-r--r-- 1 app app 134217728 Apr  1 17:21 00386605
-rw-r--r-- 1 app app 134184960 Apr  1 17:22 00386606
-rw-r--r-- 1 app app 134152192 Apr  1 17:22 00386607
-rw-r--r-- 1 app app 134152192 Apr  1 17:23 00386608
-rw-r--r-- 1 app app 133038080 Apr  1 17:24 00386609
-rw-r--r-- 1 app app 134217728 Apr  1 17:24 00386610
-rw-r--r-- 1 app app 134152192 Apr  1 17:25 00386611
-rw-r--r-- 1 app app 134217728 Apr  1 17:25 00386612
-rw-r--r-- 1 app app 131497984 Apr  1 17:26 00386613
-rw-r--r-- 1 app app 134217728 Apr  1 17:26 00386614
-rw-r--r-- 1 app app 134152192 Apr  1 17:27 00386615
-rw-r--r-- 1 app app 134217728 Apr  1 17:28 00386616
-rw-r--r-- 1 app app 134217728 Apr  1 17:28 00386617
-rw-r--r-- 1 app app 134184960 Apr  1 17:29 00386618
-rw-r--r-- 1 app app 133890048 Apr  1 17:30 00386619
-rw-r--r-- 1 app app 134217728 Apr  1 17:30 00386620
-rw-r--r-- 1 app app 134184960 Apr  1 17:31 00386621
-rw-r--r-- 1 app app 134217728 Apr  1 17:31 00386622
-rw-r--r-- 1 app app 134217728 Apr  1 17:32 00386623
-rw-r--r-- 1 app app 134217728 Apr  1 17:33 00386624
-rw-r--r-- 1 app app 133038080 Apr  1 17:33 00386625
-rw-r--r-- 1 app app 134217728 Apr  1 17:34 00386626
-rw-r--r-- 1 app app 130809856 Apr  1 17:34 00386627
-rw-r--r-- 1 app app 132022272 Apr  1 17:35 00386628
-rw-r--r-- 1 app app 134217728 Apr  1 17:36 00386629
-rw-r--r-- 1 app app 134217728 Apr  1 17:36 00386630
-rw-r--r-- 1 app app 134152192 Apr  1 17:37 00386631
-rw-r--r-- 1 app app 134217728 Apr  1 17:37 00386632
-rw-r--r-- 1 app app 134217728 Apr  1 17:38 00386633
-rw-r--r-- 1 app app 134152192 Apr  1 17:39 00386634
-rw-r--r-- 1 app app 134184960 Apr  1 17:39 00386635
-rw-r--r-- 1 app app 134152192 Apr  1 17:40 00386636
-rw-r--r-- 1 app app 134217728 Apr  1 17:40 00386637
-rw-r--r-- 1 app app 134184960 Apr  1 17:41 00386638
-rw-r--r-- 1 app app 133988352 Apr  1 17:42 00386639
-rw-r--r-- 1 app app 134086656 Apr  1 17:42 00386640
-rw-r--r-- 1 app app 134152192 Apr  1 17:43 00386641
-rw-r--r-- 1 app app 131072000 Apr  1 17:43 00386642
-rw-r--r-- 1 app app 134217728 Apr  1 17:44 00386643
-rw-r--r-- 1 app app 134184960 Apr  1 17:45 00386644
-rw-r--r-- 1 app app 134217728 Apr  1 17:45 00386645
-rw-r--r-- 1 app app 134184960 Apr  1 17:46 00386646
-rw-r--r-- 1 app app 134217728 Apr  1 17:46 00386647
-rw-r--r-- 1 app app 134184960 Apr  1 17:47 00386648
-rw-r--r-- 1 app app 134152192 Apr  1 17:48 00386649
-rw-r--r-- 1 app app 134152192 Apr  1 17:48 00386650
-rw-r--r-- 1 app app 134152192 Apr  1 17:49 00386651
-rw-r--r-- 1 app app 134184960 Apr  1 17:50 00386652
-rw-r--r-- 1 app app 134217728 Apr  1 17:50 00386653
-rw-r--r-- 1 app app 134217728 Apr  1 17:51 00386654
-rw-r--r-- 1 app app 134217728 Apr  1 17:51 00386655
-rw-r--r-- 1 app app 131301376 Apr  1 17:52 00386656
-rw-r--r-- 1 app app 134184960 Apr  1 17:53 00386657
-rw-r--r-- 1 app app 134217728 Apr  1 17:53 00386658
-rw-r--r-- 1 app app 134184960 Apr  1 17:54 00386659
-rw-r--r-- 1 app app 134184960 Apr  1 17:54 00386660
-rw-r--r-- 1 app app 134152192 Apr  1 17:55 00386661
-rw-r--r-- 1 app app 134184960 Apr  1 17:56 00386662
-rw-r--r-- 1 app app 134184960 Apr  1 17:56 00386663
-rw-r--r-- 1 app app 134152192 Apr  1 17:57 00386664
-rw-r--r-- 1 app app 134217728 Apr  1 17:58 00386665
-rw-r--r-- 1 app app 134184960 Apr  1 17:58 00386666
-rw-r--r-- 1 app app 134217728 Apr  1 17:59 00386667
-rw-r--r-- 1 app app 134152192 Apr  1 17:59 00386668
-rw-r--r-- 1 app app 134184960 Apr  1 18:00 00386669
-rw-r--r-- 1 app app 134217728 Apr  1 18:01 00386670
-rw-r--r-- 1 app app 134217728 Apr  1 18:01 00386671
-rw-r--r-- 1 app app 134152192 Apr  1 18:02 00386672
-rw-r--r-- 1 app app 133464064 Apr  1 18:02 00386673
-rw-r--r-- 1 app app 134217728 Apr  1 18:03 00386674
-rw-r--r-- 1 app app 134217728 Apr  1 18:04 00386675
-rw-r--r-- 1 app app 134152192 Apr  1 18:04 00386676
-rw-r--r-- 1 app app 134217728 Apr  1 18:05 00386677
-rw-r--r-- 1 app app 134217728 Apr  1 18:05 00386678
-rw-r--r-- 1 app app 130908160 Apr  1 18:06 00386679
-rw-r--r-- 1 app app 134217728 Apr  1 18:07 00386680
-rw-r--r-- 1 app app 132644864 Apr  1 18:07 00386681
-rw-r--r-- 1 app app 134184960 Apr  1 18:08 00386682
-rw-r--r-- 1 app app 131039232 Apr  1 18:08 00386683
-rw-r--r-- 1 app app 134184960 Apr  1 18:09 00386684
-rw-r--r-- 1 app app 134217728 Apr  1 18:10 00386685
-rw-r--r-- 1 app app 134184960 Apr  1 18:10 00386686
-rw-r--r-- 1 app app 134217728 Apr  1 18:11 00386687
-rw-r--r-- 1 app app 134184960 Apr  1 18:11 00386688
-rw-r--r-- 1 app app 134184960 Apr  1 18:12 00386689
-rw-r--r-- 1 app app 134184960 Apr  1 18:13 00386690
-rw-r--r-- 1 app app 134217728 Apr  1 18:13 00386691
-rw-r--r-- 1 app app 134184960 Apr  1 18:14 00386692
-rw-r--r-- 1 app app 134184960 Apr  1 18:14 00386693
-rw-r--r-- 1 app app 134217728 Apr  1 18:15 00386694
-rw-r--r-- 1 app app 134184960 Apr  1 18:16 00386695
-rw-r--r-- 1 app app 134217728 Apr  1 18:16 00386696
-rw-r--r-- 1 app app 134217728 Apr  1 18:17 00386697
-rw-r--r-- 1 app app 134152192 Apr  1 18:17 00386698
-rw-r--r-- 1 app app 133300224 Apr  1 18:18 00386699
-rw-r--r-- 1 app app 134184960 Apr  1 18:19 00386700
-rw-r--r-- 1 app app 134217728 Apr  1 18:19 00386701
-rw-r--r-- 1 app app 134217728 Apr  1 18:20 00386702
-rw-r--r-- 1 app app 134217728 Apr  1 18:20 00386703
-rw-r--r-- 1 app app 134184960 Apr  1 18:21 00386704
-rw-r--r-- 1 app app 134184960 Apr  1 18:22 00386705
-rw-r--r-- 1 app app 134184960 Apr  1 18:22 00386706
-rw-r--r-- 1 app app 133758976 Apr  1 18:23 00386707
-rw-r--r-- 1 app app 132972544 Apr  1 18:23 00386708
-rw-r--r-- 1 app app 134184960 Apr  1 18:24 00386709
-rw-r--r-- 1 app app 134152192 Apr  1 18:25 00386710
-rw-r--r-- 1 app app 134184960 Apr  1 18:25 00386711
-rw-r--r-- 1 app app 134184960 Apr  1 18:26 00386712
-rw-r--r-- 1 app app 134184960 Apr  1 18:26 00386713
-rw-r--r-- 1 app app 134217728 Apr  1 18:27 00386714
-rw-r--r-- 1 app app 134152192 Apr  1 18:28 00386715
-rw-r--r-- 1 app app 134184960 Apr  1 18:28 00386716
-rw-r--r-- 1 app app 134184960 Apr  1 18:29 00386717
-rw-r--r-- 1 app app 133365760 Apr  1 18:29 00386718
-rw-r--r-- 1 app app 134217728 Apr  1 18:30 00386719
-rw-r--r-- 1 app app 134184960 Apr  1 18:31 00386720
-rw-r--r-- 1 app app 134217728 Apr  1 18:31 00386721
-rw-r--r-- 1 app app 134184960 Apr  1 18:32 00386722
-rw-r--r-- 1 app app 134217728 Apr  1 18:32 00386723
-rw-r--r-- 1 app app 134217728 Apr  1 18:33 00386724
-rw-r--r-- 1 app app 134184960 Apr  1 18:34 00386725
-rw-r--r-- 1 app app 133038080 Apr  1 18:34 00386726
-rw-r--r-- 1 app app 134152192 Apr  1 18:35 00386727
-rw-r--r-- 1 app app 134217728 Apr  1 18:35 00386728
-rw-r--r-- 1 app app 134217728 Apr  1 18:36 00386729
-rw-r--r-- 1 app app 134184960 Apr  1 18:37 00386730
-rw-r--r-- 1 app app 132218880 Apr  1 18:37 00386731
-rw-r--r-- 1 app app 134217728 Apr  1 18:38 00386732
-rw-r--r-- 1 app app 134152192 Apr  1 18:38 00386733
-rw-r--r-- 1 app app 134184960 Apr  1 18:39 00386734
-rw-r--r-- 1 app app 131760128 Apr  1 18:40 00386735
-rw-r--r-- 1 app app 134152192 Apr  1 18:40 00386736
-rw-r--r-- 1 app app 134217728 Apr  1 18:41 00386737
-rw-r--r-- 1 app app 130514944 Apr  1 18:41 00386738
-rw-r--r-- 1 app app 134184960 Apr  1 18:42 00386739
-rw-r--r-- 1 app app 134184960 Apr  1 18:43 00386740
-rw-r--r-- 1 app app 132284416 Apr  1 18:43 00386741
-rw-r--r-- 1 app app 134184960 Apr  1 18:44 00386742
-rw-r--r-- 1 app app 134152192 Apr  1 18:44 00386743
-rw-r--r-- 1 app app 134217728 Apr  1 18:45 00386744
-rw-r--r-- 1 app app 134184960 Apr  1 18:46 00386745
-rw-r--r-- 1 app app 132644864 Apr  1 18:46 00386746
-rw-r--r-- 1 app app 134152192 Apr  1 18:47 00386747
-rw-r--r-- 1 app app 134152192 Apr  1 18:47 00386748
-rw-r--r-- 1 app app 134217728 Apr  1 18:48 00386749
-rw-r--r-- 1 app app 134217728 Apr  1 18:49 00386750
-rw-r--r-- 1 app app 134184960 Apr  1 18:49 00386751
-rw-r--r-- 1 app app 134184960 Apr  1 18:50 00386752
-rw-r--r-- 1 app app 134152192 Apr  1 18:51 00386753
-rw-r--r-- 1 app app 134217728 Apr  1 18:51 00386754
-rw-r--r-- 1 app app 131170304 Apr  1 18:52 00386755
-rw-r--r-- 1 app app 134184960 Apr  1 18:52 00386756
-rw-r--r-- 1 app app 134152192 Apr  1 18:53 00386757
-rw-r--r-- 1 app app 134184960 Apr  1 18:54 00386758
-rw-r--r-- 1 app app 134184960 Apr  1 18:54 00386759
-rw-r--r-- 1 app app 134217728 Apr  1 18:55 00386760
-rw-r--r-- 1 app app 134184960 Apr  1 18:55 00386761
-rw-r--r-- 1 app app 134217728 Apr  1 18:56 00386762
-rw-r--r-- 1 app app 132186112 Apr  1 18:57 00386763
-rw-r--r-- 1 app app 134152192 Apr  1 18:57 00386764
-rw-r--r-- 1 app app 134217728 Apr  1 18:58 00386765
-rw-r--r-- 1 app app 133136384 Apr  1 18:59 00386766
-rw-r--r-- 1 app app 134217728 Apr  1 18:59 00386767
-rw-r--r-- 1 app app 134152192 Apr  1 19:00 00386768
-rw-r--r-- 1 app app 134152192 Apr  1 19:00 00386769
-rw-r--r-- 1 app app 134152192 Apr  1 19:01 00386770
-rw-r--r-- 1 app app 134217728 Apr  1 19:02 00386771
-rw-r--r-- 1 app app 134217728 Apr  1 19:02 00386772
-rw-r--r-- 1 app app 134217728 Apr  1 19:03 00386773
-rw-r--r-- 1 app app 134152192 Apr  1 19:03 00386774
-rw-r--r-- 1 app app 134217728 Apr  1 19:04 00386775
-rw-r--r-- 1 app app 133201920 Apr  1 19:05 00386776
-rw-r--r-- 1 app app 134184960 Apr  1 19:05 00386777
-rw-r--r-- 1 app app 134217728 Apr  1 19:06 00386778
-rw-r--r-- 1 app app 134184960 Apr  1 19:07 00386779
-rw-r--r-- 1 app app 134184960 Apr  1 19:07 00386780
-rw-r--r-- 1 app app 134152192 Apr  1 19:08 00386781
-rw-r--r-- 1 app app 134184960 Apr  1 19:08 00386782
-rw-r--r-- 1 app app 131563520 Apr  1 19:09 00386783
-rw-r--r-- 1 app app 134217728 Apr  1 19:10 00386784
-rw-r--r-- 1 app app 134184960 Apr  1 19:10 00386785
-rw-r--r-- 1 app app 134217728 Apr  1 19:11 00386786
-rw-r--r-- 1 app app 134217728 Apr  1 19:11 00386787
-rw-r--r-- 1 app app 134217728 Apr  1 19:12 00386788
-rw-r--r-- 1 app app 134184960 Apr  1 19:12 00386789
-rw-r--r-- 1 app app 134217728 Apr  1 19:13 00386790
-rw-r--r-- 1 app app 134217728 Apr  1 19:14 00386791
-rw-r--r-- 1 app app 132612096 Apr  1 19:14 00386792
-rw-r--r-- 1 app app 134184960 Apr  1 19:15 00386793
-rw-r--r-- 1 app app 133595136 Apr  1 19:15 00386794
-rw-r--r-- 1 app app 134086656 Apr  1 19:16 00386795
-rw-r--r-- 1 app app 134184960 Apr  1 19:17 00386796
-rw-r--r-- 1 app app 134184960 Apr  1 19:17 00386797
-rw-r--r-- 1 app app 134152192 Apr  1 19:18 00386798
-rw-r--r-- 1 app app 134152192 Apr  1 19:18 00386799
-rw-r--r-- 1 app app 133857280 Apr  1 19:19 00386800
-rw-r--r-- 1 app app 134152192 Apr  1 19:20 00386801
-rw-r--r-- 1 app app 131399680 Apr  1 19:20 00386802
-rw-r--r-- 1 app app 131858432 Apr  1 19:21 00386803
-rw-r--r-- 1 app app 134152192 Apr  1 19:21 00386804
-rw-r--r-- 1 app app 134217728 Apr  1 19:22 00386805
-rw-r--r-- 1 app app 134152192 Apr  1 19:23 00386806
-rw-r--r-- 1 app app 134217728 Apr  1 19:23 00386807
-rw-r--r-- 1 app app 134217728 Apr  1 19:24 00386808
-rw-r--r-- 1 app app 134184960 Apr  1 19:24 00386809
-rw-r--r-- 1 app app 134184960 Apr  1 19:25 00386810
-rw-r--r-- 1 app app 134184960 Apr  1 19:26 00386811
-rw-r--r-- 1 app app 134217728 Apr  1 19:26 00386812
-rw-r--r-- 1 app app 134152192 Apr  1 19:27 00386813
-rw-r--r-- 1 app app 134184960 Apr  1 19:28 00386814
-rw-r--r-- 1 app app 134184960 Apr  1 19:28 00386815
-rw-r--r-- 1 app app 134217728 Apr  1 19:29 00386816
-rw-r--r-- 1 app app 134184960 Apr  1 19:29 00386817
-rw-r--r-- 1 app app 134184960 Apr  1 19:30 00386818
-rw-r--r-- 1 app app 134217728 Apr  1 19:31 00386819
-rw-r--r-- 1 app app 134217728 Apr  1 19:31 00386820
-rw-r--r-- 1 app app 131661824 Apr  1 19:32 00386821
-rw-r--r-- 1 app app 134217728 Apr  1 19:32 00386822
-rw-r--r-- 1 app app 132972544 Apr  1 19:33 00386823
-rw-r--r-- 1 app app 134184960 Apr  1 19:34 00386824
-rw-r--r-- 1 app app 133857280 Apr  1 19:34 00386825
-rw-r--r-- 1 app app 133365760 Apr  1 19:35 00386826
-rw-r--r-- 1 app app 134217728 Apr  1 19:35 00386827
-rw-r--r-- 1 app app 133103616 Apr  1 19:36 00386828
-rw-r--r-- 1 app app 134184960 Apr  1 19:37 00386829
-rw-r--r-- 1 app app 134184960 Apr  1 19:37 00386830
-rw-r--r-- 1 app app 134152192 Apr  1 19:38 00386831
-rw-r--r-- 1 app app 134217728 Apr  1 19:38 00386832
-rw-r--r-- 1 app app 134217728 Apr  1 19:39 00386833
-rw-r--r-- 1 app app 134217728 Apr  1 19:40 00386834
-rw-r--r-- 1 app app 133890048 Apr  1 19:40 00386835
-rw-r--r-- 1 app app 134217728 Apr  1 19:41 00386836
-rw-r--r-- 1 app app 134217728 Apr  1 19:41 00386837
-rw-r--r-- 1 app app 134184960 Apr  1 19:42 00386838
-rw-r--r-- 1 app app 134184960 Apr  1 19:43 00386839
-rw-r--r-- 1 app app 134184960 Apr  1 19:43 00386840
-rw-r--r-- 1 app app 134217728 Apr  1 19:44 00386841
-rw-r--r-- 1 app app 134184960 Apr  1 19:44 00386842
-rw-r--r-- 1 app app 134184960 Apr  1 19:45 00386843
-rw-r--r-- 1 app app 134217728 Apr  1 19:46 00386844
-rw-r--r-- 1 app app 134152192 Apr  1 19:46 00386845
-rw-r--r-- 1 app app 134217728 Apr  1 19:47 00386846
-rw-r--r-- 1 app app 134184960 Apr  1 19:48 00386847
-rw-r--r-- 1 app app 134217728 Apr  1 19:48 00386848
-rw-r--r-- 1 app app 134217728 Apr  1 19:49 00386849
-rw-r--r-- 1 app app 134217728 Apr  1 19:49 00386850
-rw-r--r-- 1 app app 131334144 Apr  1 19:50 00386851
-rw-r--r-- 1 app app 134184960 Apr  1 19:51 00386852
-rw-r--r-- 1 app app 134184960 Apr  1 19:51 00386853
-rw-r--r-- 1 app app 134184960 Apr  1 19:52 00386854
-rw-r--r-- 1 app app 134119424 Apr  1 19:52 00386855
-rw-r--r-- 1 app app 134184960 Apr  1 19:53 00386856
-rw-r--r-- 1 app app 134184960 Apr  1 19:54 00386857
-rw-r--r-- 1 app app 134184960 Apr  1 19:54 00386858
-rw-r--r-- 1 app app 134152192 Apr  1 19:55 00386859
-rw-r--r-- 1 app app 134217728 Apr  1 19:55 00386860
-rw-r--r-- 1 app app 132841472 Apr  1 19:56 00386861
-rw-r--r-- 1 app app 134152192 Apr  1 19:57 00386862
-rw-r--r-- 1 app app 134184960 Apr  1 19:57 00386863
-rw-r--r-- 1 app app 134184960 Apr  1 19:58 00386864
-rw-r--r-- 1 app app 134217728 Apr  1 19:59 00386865
-rw-r--r-- 1 app app 134184960 Apr  1 19:59 00386866
-rw-r--r-- 1 app app 134152192 Apr  1 20:00 00386867
-rw-r--r-- 1 app app 131137536 Apr  1 20:00 00386868
-rw-r--r-- 1 app app 132841472 Apr  1 20:01 00386869
-rw-r--r-- 1 app app 133070848 Apr  1 20:02 00386870
-rw-r--r-- 1 app app 134152192 Apr  1 20:02 00386871
-rw-r--r-- 1 app app 134217728 Apr  1 20:03 00386872
-rw-r--r-- 1 app app 134184960 Apr  1 20:03 00386873
-rw-r--r-- 1 app app 134217728 Apr  1 20:04 00386874
-rw-r--r-- 1 app app 134184960 Apr  1 20:05 00386875
-rw-r--r-- 1 app app 134217728 Apr  1 20:05 00386876
-rw-r--r-- 1 app app 134217728 Apr  1 20:06 00386877
-rw-r--r-- 1 app app 134184960 Apr  1 20:06 00386878
-rw-r--r-- 1 app app 134217728 Apr  1 20:07 00386879
-rw-r--r-- 1 app app 134152192 Apr  1 20:08 00386880
-rw-r--r-- 1 app app 133857280 Apr  1 20:08 00386881
-rw-r--r-- 1 app app 130711552 Apr  1 20:09 00386882
-rw-r--r-- 1 app app 134152192 Apr  1 20:09 00386883
-rw-r--r-- 1 app app 134217728 Apr  1 20:10 00386884
-rw-r--r-- 1 app app 134152192 Apr  1 20:11 00386885
-rw-r--r-- 1 app app 134217728 Apr  1 20:11 00386886
-rw-r--r-- 1 app app 134217728 Apr  1 20:12 00386887
-rw-r--r-- 1 app app 134152192 Apr  1 20:12 00386888
-rw-r--r-- 1 app app 134217728 Apr  1 20:13 00386889
-rw-r--r-- 1 app app 134217728 Apr  1 20:14 00386890
-rw-r--r-- 1 app app 134217728 Apr  1 20:14 00386891
-rw-r--r-- 1 app app 134184960 Apr  1 20:15 00386892
-rw-r--r-- 1 app app 133103616 Apr  1 20:15 00386893
-rw-r--r-- 1 app app 134217728 Apr  1 20:16 00386894
-rw-r--r-- 1 app app 134152192 Apr  1 20:16 00386895
-rw-r--r-- 1 app app 134184960 Apr  1 20:17 00386896
-rw-r--r-- 1 app app 134217728 Apr  1 20:18 00386897
-rw-r--r-- 1 app app 134152192 Apr  1 20:18 00386898
-rw-r--r-- 1 app app 134152192 Apr  1 20:19 00386899
-rw-r--r-- 1 app app 134152192 Apr  1 20:19 00386900
-rw-r--r-- 1 app app 134184960 Apr  1 20:20 00386901
-rw-r--r-- 1 app app 134184960 Apr  1 20:21 00386902
-rw-r--r-- 1 app app 134184960 Apr  1 20:21 00386903
-rw-r--r-- 1 app app 134217728 Apr  1 20:22 00386904
-rw-r--r-- 1 app app 134217728 Apr  1 20:22 00386905
-rw-r--r-- 1 app app 134184960 Apr  1 20:23 00386906
-rw-r--r-- 1 app app 134152192 Apr  1 20:24 00386907
-rw-r--r-- 1 app app 134217728 Apr  1 20:24 00386908
-rw-r--r-- 1 app app 134152192 Apr  1 20:25 00386909
-rw-r--r-- 1 app app 134217728 Apr  1 20:26 00386910
-rw-r--r-- 1 app app 134184960 Apr  1 20:26 00386911
-rw-r--r-- 1 app app 134184960 Apr  1 20:27 00386912
-rw-r--r-- 1 app app 134217728 Apr  1 20:27 00386913
-rw-r--r-- 1 app app 134184960 Apr  1 20:28 00386914
-rw-r--r-- 1 app app 134184960 Apr  1 20:29 00386915
-rw-r--r-- 1 app app 134152192 Apr  1 20:29 00386916
-rw-r--r-- 1 app app 134152192 Apr  1 20:30 00386917
-rw-r--r-- 1 app app 134184960 Apr  1 20:30 00386918
-rw-r--r-- 1 app app 134217728 Apr  1 20:31 00386919
-rw-r--r-- 1 app app 134152192 Apr  1 20:32 00386920
-rw-r--r-- 1 app app 134217728 Apr  1 20:32 00386921
-rw-r--r-- 1 app app 134217728 Apr  1 20:33 00386922
-rw-r--r-- 1 app app 134217728 Apr  1 20:33 00386923
-rw-r--r-- 1 app app 133234688 Apr  1 20:34 00386924
-rw-r--r-- 1 app app 134152192 Apr  1 20:35 00386925
-rw-r--r-- 1 app app 132382720 Apr  1 20:35 00386926
-rw-r--r-- 1 app app 134184960 Apr  1 20:36 00386927
-rw-r--r-- 1 app app 134184960 Apr  1 20:36 00386928
-rw-r--r-- 1 app app 134152192 Apr  1 20:37 00386929
-rw-r--r-- 1 app app 130809856 Apr  1 20:38 00386930
-rw-r--r-- 1 app app 134184960 Apr  1 20:38 00386931
-rw-r--r-- 1 app app 134217728 Apr  1 20:39 00386932
-rw-r--r-- 1 app app 131497984 Apr  1 20:39 00386933
-rw-r--r-- 1 app app 134184960 Apr  1 20:40 00386934
-rw-r--r-- 1 app app 134217728 Apr  1 20:41 00386935
-rw-r--r-- 1 app app 134152192 Apr  1 20:41 00386936
-rw-r--r-- 1 app app 134217728 Apr  1 20:42 00386937
-rw-r--r-- 1 app app 134184960 Apr  1 20:43 00386938
-rw-r--r-- 1 app app 134217728 Apr  1 20:43 00386939
-rw-r--r-- 1 app app 134217728 Apr  1 20:44 00386940
-rw-r--r-- 1 app app 134184960 Apr  1 20:44 00386941
-rw-r--r-- 1 app app 134184960 Apr  1 20:45 00386942
-rw-r--r-- 1 app app 134184960 Apr  1 20:46 00386943
-rw-r--r-- 1 app app 134217728 Apr  1 20:46 00386944
-rw-r--r-- 1 app app 133824512 Apr  1 20:47 00386945
-rw-r--r-- 1 app app 134217728 Apr  1 20:47 00386946
-rw-r--r-- 1 app app 133038080 Apr  1 20:48 00386947
-rw-r--r-- 1 app app 133300224 Apr  1 20:49 00386948
-rw-r--r-- 1 app app 134184960 Apr  1 20:49 00386949
-rw-r--r-- 1 app app 134152192 Apr  1 20:50 00386950
-rw-r--r-- 1 app app 133332992 Apr  1 20:50 00386951
-rw-r--r-- 1 app app 131268608 Apr  1 20:51 00386952
-rw-r--r-- 1 app app 134152192 Apr  1 20:52 00386953
-rw-r--r-- 1 app app 131235840 Apr  1 20:52 00386954
-rw-r--r-- 1 app app 134152192 Apr  1 20:53 00386955
-rw-r--r-- 1 app app 134217728 Apr  1 20:53 00386956
-rw-r--r-- 1 app app 134217728 Apr  1 20:54 00386957
-rw-r--r-- 1 app app 134152192 Apr  1 20:55 00386958
-rw-r--r-- 1 app app 134217728 Apr  1 20:55 00386959
-rw-r--r-- 1 app app 134217728 Apr  1 20:56 00386960
-rw-r--r-- 1 app app 134217728 Apr  1 20:57 00386961
-rw-r--r-- 1 app app 131596288 Apr  1 20:57 00386962
-rw-r--r-- 1 app app 133627904 Apr  1 20:58 00386963
-rw-r--r-- 1 app app 134217728 Apr  1 20:58 00386964
-rw-r--r-- 1 app app 134184960 Apr  1 20:59 00386965
-rw-r--r-- 1 app app 134217728 Apr  1 21:00 00386966
-rw-r--r-- 1 app app 130973696 Apr  1 21:00 00386967
-rw-r--r-- 1 app app 134217728 Apr  1 21:01 00386968
-rw-r--r-- 1 app app 134217728 Apr  1 21:01 00386969
-rw-r--r-- 1 app app 134184960 Apr  1 21:02 00386970
-rw-r--r-- 1 app app 133791744 Apr  1 21:03 00386971
-rw-r--r-- 1 app app 134152192 Apr  1 21:03 00386972
-rw-r--r-- 1 app app 134217728 Apr  1 21:04 00386973
-rw-r--r-- 1 app app 134184960 Apr  1 21:04 00386974
-rw-r--r-- 1 app app 134217728 Apr  1 21:05 00386975
-rw-r--r-- 1 app app 134184960 Apr  1 21:06 00386976
-rw-r--r-- 1 app app 132579328 Apr  1 21:06 00386977
-rw-r--r-- 1 app app 134184960 Apr  1 21:07 00386978
-rw-r--r-- 1 app app 134217728 Apr  1 21:08 00386979
-rw-r--r-- 1 app app 134217728 Apr  1 21:08 00386980
-rw-r--r-- 1 app app 134217728 Apr  1 21:09 00386981
-rw-r--r-- 1 app app 133955584 Apr  1 21:09 00386982
-rw-r--r-- 1 app app 134184960 Apr  1 21:10 00386983
-rw-r--r-- 1 app app 134184960 Apr  1 21:11 00386984
-rw-r--r-- 1 app app 130547712 Apr  1 21:11 00386985
-rw-r--r-- 1 app app 134152192 Apr  1 21:12 00386986
-rw-r--r-- 1 app app 134152192 Apr  1 21:12 00386987
-rw-r--r-- 1 app app 134217728 Apr  1 21:13 00386988
-rw-r--r-- 1 app app 134184960 Apr  1 21:14 00386989
-rw-r--r-- 1 app app 134152192 Apr  1 21:14 00386990
-rw-r--r-- 1 app app 133857280 Apr  1 21:15 00386991
-rw-r--r-- 1 app app 134184960 Apr  1 21:15 00386992
-rw-r--r-- 1 app app 131465216 Apr  1 21:16 00386993
-rw-r--r-- 1 app app 134184960 Apr  1 21:17 00386994
-rw-r--r-- 1 app app 134184960 Apr  1 21:17 00386995
-rw-r--r-- 1 app app 134217728 Apr  1 21:18 00386996
-rw-r--r-- 1 app app 134217728 Apr  1 21:18 00386997
-rw-r--r-- 1 app app 134152192 Apr  1 21:19 00386998
-rw-r--r-- 1 app app 134184960 Apr  1 21:20 00386999
-rw-r--r-- 1 app app 134217728 Apr  1 21:20 00387000
-rw-r--r-- 1 app app 133890048 Apr  1 21:21 00387001
-rw-r--r-- 1 app app 134152192 Apr  1 21:21 00387002
-rw-r--r-- 1 app app 134217728 Apr  1 21:22 00387003
-rw-r--r-- 1 app app 134184960 Apr  1 21:23 00387004
-rw-r--r-- 1 app app 134217728 Apr  1 21:23 00387005
-rw-r--r-- 1 app app 134152192 Apr  1 21:24 00387006
-rw-r--r-- 1 app app 134184960 Apr  1 21:24 00387007
-rw-r--r-- 1 app app 134217728 Apr  1 21:25 00387008
-rw-r--r-- 1 app app 134152192 Apr  1 21:26 00387009
-rw-r--r-- 1 app app 134184960 Apr  1 21:26 00387010
-rw-r--r-- 1 app app 134217728 Apr  1 21:27 00387011
-rw-r--r-- 1 app app 134184960 Apr  1 21:27 00387012
-rw-r--r-- 1 app app 134217728 Apr  1 21:28 00387013
-rw-r--r-- 1 app app 132153344 Apr  1 21:29 00387014
-rw-r--r-- 1 app app 132055040 Apr  1 21:29 00387015
-rw-r--r-- 1 app app 134184960 Apr  1 21:30 00387016
-rw-r--r-- 1 app app 134184960 Apr  1 21:30 00387017
-rw-r--r-- 1 app app 134184960 Apr  1 21:31 00387018
-rw-r--r-- 1 app app 134217728 Apr  1 21:32 00387019
-rw-r--r-- 1 app app 134217728 Apr  1 21:32 00387020
-rw-r--r-- 1 app app 134184960 Apr  1 21:33 00387021
-rw-r--r-- 1 app app 133464064 Apr  1 21:33 00387022
-rw-r--r-- 1 app app 134184960 Apr  1 21:34 00387023
-rw-r--r-- 1 app app  70417702 Apr  1 21:34 00387024
drwxr-xr-x 2 app app      4096 Apr  1 21:05 checkpoint.386577

brian-brazil · Apr 1, 2019

That adds up anyway.

tomwilkie · Apr 2, 2019

Mar 29 23:57:27 server_A prometheus[16760]: level=warn ts=2019-03-29T23:57:27.680607312Z caller=head.go:450 component=tsdb msg="unknown series references" count=59855

This looks the same as #5389. What are you thinking @brian-brazil ?

tomwilkie · Apr 2, 2019

@aned can you get a goroutine dump please? Go to /debug/pprof/goroutine?debug=2 in the browser.

tomwilkie · Apr 2, 2019

From the graphs it looks like you downgraded back to 2.7 shortly after trying 2.8 - is that correct?

brian-brazil · Apr 2, 2019

What are you thinking @brian-brazil ?

I was checking there wasn't anything crazy in the WAL file wise.

tomwilkie · Apr 2, 2019

I could do with a rubber ducking session with you and @cstyan to chat about this TBH, want to try and get some mitigation / fix in place for 2.9. Got some time this week?

brian-brazil · Apr 2, 2019

This week is fairly busy for me, I was planning on digging around today.

brian-brazil · Apr 2, 2019

Can you indicate if you're using the Delete api?

aned · Apr 2, 2019

I don't think m3coordinator deletes any series via api.

aned · Apr 16, 2019

The issue is still present in 2.9, no errors in the logs this time, but the remote write pretty much doesn't work, reverted back to 2.7.2

Also noticed a significant performance degradation on servers that do remote write:

brian-brazil · Apr 16, 2019

Delete your WAL, and try 2.9.0 again.

aned · Apr 16, 2019

2.9.1, deleted WAL, the same issue.

cstyan · Apr 17, 2019

@aned can I see the logs from your 2.9 instance?

Can you also post these graphs again from the instance that you're running with a new WAL?

Can you post the logs from the startup of your Prometheus please?

Also, can you post us screen shots of the following queries?

rate(prometheus_tsdb_head_samples_appended_total[1m])

rate(prometheus_remote_storage_samples_in_total[1m])

rate(prometheus_remote_storage_dropped_samples_total[1m])

prometheus_wal_watcher_current_segment

rate(prometheus_wal_watcher_records_read_total[1m])

I can look into this tomorrow.

aned · Apr 17, 2019

Thanks guys, attaching the screenshots (no errors in the logs):

brian-brazil · Apr 17, 2019

Hmm, it looks like everything is being dropped.

brian-brazil · Apr 17, 2019

Given the lack of logs about dropped series, this is possibly due to write_relabel_configs dropping everything. Can you share the remote_write part of your config?

aned · Apr 17, 2019

remote_write:
  - url: "http://localhost:7201/api/v1/prom/remote/write"
    write_relabel_configs:
     - source_labels: [__name__]
       regex: 'job:.*|record_local:.*'
       action: keep
     - source_labels: [__name__]
       regex: '.*Garbage.*'
       action: drop
    queue_config:
      capacity: 30000

brian-brazil · Apr 17, 2019

That's going to drop a lot of time series, so this result is as expected. Are there signs that data isn't making it to your remote write endpoint?

aned · Apr 17, 2019

Nothing changed but the version of prometheus, this is a snapshot on the remote write aggregation point - m3db:
2.7.2 has ~250k, 2.9.1 has 50k

Also, note the rule evaluation jumped 7x, prometheus basically becomes unusable on anything above 2.7.2:

Other performance issues:

aned · Apr 17, 2019

A small number of series was making it to the remote storage on 2.9.1 (but still unusable with all of the above)

brian-brazil · Apr 17, 2019

Based on that data my first thought is that relabelling was broken back in 2.7.2 for remote write.

What does count({__name__=~'job:.*|record_local:.*'}) return on the Prometheus and what's your evaluation interval?

If rule evaluations are failing, that'd also decrease the amount of data that'd be sent to remote write.

aned · Apr 17, 2019

The count is 1162458.

global:
  scrape_interval: 30s
  scrape_timeout: 30s
  evaluation_interval: 30s

brian-brazil · Apr 17, 2019

Okay, so we'd expect to see about 38k successes per second when everything is working correctly.

brian-brazil · Apr 17, 2019

Can you share with me what the Rules Status page is showing, preferably when there's some failures?

aned · Apr 17, 2019

I'd need to reproduce it tomorrow if really needed.

brian-brazil · Apr 17, 2019

At this point I suspect that remote_write is working fine, however there's something up with your rule evaluations.

brian-brazil · Apr 17, 2019

That'd be great.

aned · Apr 17, 2019

Nm, I'll dm you the page in a few.

brian-brazil · Apr 17, 2019

A snapshot of the same page from 2.7.2 would also be handy for comparison.

aned · Apr 17, 2019

Ok, reverting back, will dm you as well. Anything else might be useful from 2.9.1 before I revert?

brian-brazil · Apr 17, 2019

That's it offhand.

aned · Apr 17, 2019

CPU load of the machine, 2.9.1 in the middle:

brian-brazil · Apr 17, 2019

Given how Go GC works, 75% is the fullest you can safely be on average. You need to either cut CPU usage, or get a bigger machine.

brian-brazil · Apr 17, 2019

This doesn't look like the same thing based on hte graphs, but you might want to try the settings at #5166 (comment)

brian-brazil · Apr 17, 2019

Hmm, what's your GOMAXPROCS? It'll be on the Runtime Information page (should be the same for both versions).

aned · Apr 17, 2019

GOMAXPROCS is 24 in both cases. Tried 2.9.1 with

    queue_config:
      capacity: 300000
      max_shards: 100
      max_samples_per_send: 10000

the same results.

aned · Apr 18, 2019

Removed remote write completely, tested 2.9.1, the same performance degradation (read unusable), so the remote write is not related here I guess.
It's ingesting only 200k samples/s on a 24-core 256G RAM and SSD.

aned · Apr 18, 2019

Some snapshots without remote write, 2.7.2 vs 2.9.1

2.9.1 has 10k less samples, probably dropping them due to high CPU

brian-brazil · Apr 18, 2019

It's odd that you're hitting a 12 CPU limit on a 24 core box. What's your GOMAXPROCS showing as and are those real cores or hyperthreads?

…

On Thu 18 Apr 2019, 23:41 Artem Nedoshepa, ***@***.***> wrote: Some snapshots without remote write, 2.7.2 vs 2.9.1 [image: image] <https://user-images.githubusercontent.com/2947691/56392937-b267d680-61e7-11e9-94d1-c41a0abc521f.png> 2.9.1 has 10k less samples, probably dropping them due to high CPU [image: image] <https://user-images.githubusercontent.com/2947691/56393000-e2af7500-61e7-11e9-9c10-b7ef1c7de3f5.png> [image: image] <https://user-images.githubusercontent.com/2947691/56393051-08d51500-61e8-11e9-8594-1ae025735a07.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5424 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABWJG5SQSJUUCJARQDDZSYDPRDTKLANCNFSM4HCV637A> .

aned · Apr 18, 2019

12 real cores (24 hyperthreads).
GOMAXPROCS shows 24 in both cases.

brian-brazil · Apr 18, 2019

Okay, so really a 12 core box. You could try setting the GOMAXPROCS environment variable to 12 to reduce contention but likely you need a bigger box.

…

On Fri 19 Apr 2019, 00:14 Artem Nedoshepa, ***@***.***> wrote: 12 real cores (24 hyperthreads). GOMAXPROCS shows 24 in both cases. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5424 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABWJG5TN73JOGHSBEMVP4HLPRDXDDANCNFSM4HCV637A> .

aned · Apr 18, 2019

Why everything works just fine on 2.7.2 and when switching to 2.9 I need a "bigger box"?

brian-brazil · Apr 19, 2019

Changes in resource usage not unexpected between versions. If you can get a CPU profile we can see which change might have caused it.

…

On Fri 19 Apr 2019, 00:53 Artem Nedoshepa, ***@***.***> wrote: Why everything works just fine on 2.7.2 and when switching to 2.9 I need a "bigger box"? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5424 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABWJG5USEMM5JSQH2WAIXDLPRD3XRANCNFSM4HCV637A> .

aned · Apr 19, 2019

CPU profile, on 2.7.2 and 2.9.1

Archive.zip

brian-brazil · Apr 20, 2019

Looks like it's garbage collection, everything else is faster on 2.9.1. Can you share a heap profile with the -alloc_space flag. Preferably grab two 5m apart for each version.

aned · Apr 20, 2019

go tool pprof -alloc_space --svg http://localhost:7320/debug/pprof/heap > prof-heap.svg

Archive-heap.zip

brian-brazil · Apr 21, 2019

Looks like the allocation in mergedPostings.Seek is the issue.

tomwilkie added the component/remote storage label Apr 1, 2019

brian-brazil added a commit that referenced this issue Apr 2, 2019

Fixes #5424 #5389
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

Loading status checks…

1099999

cstyan referenced this issue Apr 3, 2019
Merged
Remote Write: fix checkpoint reading #5429

vsliouniaev referenced this issue Apr 11, 2019
Merged
[stable/prometheus-operator] Updated Prometheus Operator versions #12971
3 of 3 tasks complete

brian-brazil referenced this issue Apr 21, 2019
Open
Avoid allocation in mergedPostings.Seek. #586

krasi-georgiev added component/local storage kind/bug and removed component/remote storage labels Apr 21, 2019

prometheus/prometheus

Join GitHub today

Release 2.8+ remote storage doesn't work on ext4 bare metal, running RH7 #5424

Comments

aned commented Apr 1, 2019 • edited

tomwilkie added the component/remote storage label Apr 1, 2019

This comment has been minimized.

tomwilkie commented Apr 1, 2019 • edited

This comment has been minimized.

aned commented Apr 1, 2019

This comment has been minimized.

brian-brazil commented Apr 1, 2019

This comment has been minimized.

aned commented Apr 1, 2019

This comment has been minimized.

brian-brazil commented Apr 1, 2019

This comment has been minimized.

tomwilkie commented Apr 2, 2019

This comment has been minimized.

tomwilkie commented Apr 2, 2019

This comment has been minimized.

tomwilkie commented Apr 2, 2019

This comment has been minimized.

brian-brazil commented Apr 2, 2019

This comment has been minimized.

tomwilkie commented Apr 2, 2019

This comment has been minimized.

brian-brazil commented Apr 2, 2019

This comment has been minimized.

brian-brazil commented Apr 2, 2019

brian-brazil added a commit that referenced this issue Apr 2, 2019

This comment has been minimized.

aned commented Apr 2, 2019

cstyan referenced this issue Apr 3, 2019

Remote Write: fix checkpoint reading #5429

vsliouniaev referenced this issue Apr 11, 2019

[stable/prometheus-operator] Updated Prometheus Operator versions #12971

This comment has been minimized.

aned commented Apr 16, 2019

This comment has been minimized.

brian-brazil commented Apr 16, 2019

This comment has been minimized.

aned commented Apr 16, 2019

This comment has been minimized.

cstyan commented Apr 17, 2019

This comment has been minimized.

aned commented Apr 17, 2019

This comment has been minimized.

brian-brazil commented Apr 17, 2019

This comment has been minimized.

brian-brazil commented Apr 17, 2019

This comment has been minimized.

aned commented Apr 17, 2019 • edited

This comment has been minimized.

brian-brazil commented Apr 17, 2019

This comment has been minimized.

aned commented Apr 17, 2019 • edited

This comment has been minimized.

aned commented Apr 17, 2019

This comment has been minimized.

brian-brazil commented Apr 17, 2019

This comment has been minimized.

aned commented Apr 17, 2019

This comment has been minimized.

brian-brazil commented Apr 17, 2019

This comment has been minimized.

brian-brazil commented Apr 17, 2019

This comment has been minimized.

aned commented Apr 17, 2019

This comment has been minimized.

brian-brazil commented Apr 17, 2019

This comment has been minimized.

brian-brazil commented Apr 17, 2019

This comment has been minimized.

aned commented Apr 17, 2019

This comment has been minimized.

brian-brazil commented Apr 17, 2019

This comment has been minimized.

aned commented Apr 17, 2019

This comment has been minimized.

aned commented Apr 1, 2019 •

edited

tomwilkie commented Apr 1, 2019 •

edited

aned commented Apr 17, 2019 •

edited

aned commented Apr 17, 2019 •

edited

aned commented Apr 17, 2019 •

edited