Logstash reports a seq num error and is unable to start when persistent queue is enabled

[jsvd]

  3) conditionals in filter operators [message] != 'sample' "sample" when processed
     Failure/Error: sample("sample") { expect(subject.get("tags") ).to include("failure") }
     IOError:
       first seqNum=0 is different than minSeqNum=17773
     # org/logstash/ackedqueue/ext/JrubyAckedQueueMemoryExtLibrary.java:131:in `open'
     # ./logstash-core/lib/logstash/util/wrapped_acked_queue.rb:40:in `with_queue'
     # ./logstash-core/lib/logstash/util/wrapped_acked_queue.rb:23:in `create_memory_based'
     # ./logstash-core/lib/logstash/queue_factory.rb:21:in `create'
     # ./logstash-core/lib/logstash/pipeline.rb:98:in `initialize'
     # ./vendor/bundle/jruby/1.9/gems/logstash-devutils-1.2.1-java/lib/logstash/devutils/rspec/logstash_helpers.rb:34:in `pipeline'

rspec ./logstash-core/spec/conditionals_spec.rb:226 # conditionals in filter operators [message] != 'sample' "sample" when processed

Randomized with seed 39057

https://travis-ci.org/elastic/logstash/jobs/192716752#L1082-L1110

[ipsec]

I have a similar error in production.
:reason=>"first seqNum=0 is different than minSeqNum=3879808"

I'm unable to start logstash.
How to fix?

Version 5.1.2

[jsvd]

@ipsec are you using persistent queues (have you changed the logstash.yml file)?

[ipsec]

I'm using persisted queue.
My config has:

queue.type: persisted
path.queue: /data/logstash-queue
queue.max_bytes: 35gb

[guyboertje]

@ipsec
How many page.N files do you have in data/queue?

Can you see the cpdump utility in the logstash/bin folder?

Use this to printout the binary contents of a checkpoint file. Print the lowest and highest checkpoint.N files. Post you console output here. Use triple backticks.

DO NOT DELETE any page.N or checkpoint.N files!!

[ipsec]

# ls -1 page* | wc -l
93

I'm unable to run cpdump. I don't know about jruby :)

:/usr/share/logstash # ./bin/cpdump --help
Bundler::GemNotFound: Could not find rake-12.0.0 in any of the sources
      materialize at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/spec_set.rb:92
             map! at org/jruby/RubyArray.java:2446
      materialize at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/spec_set.rb:85
            specs at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:132
        specs_for at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:177
  requested_specs at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:166
  requested_specs at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/environment.rb:18
            setup at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/runtime.rb:13
            setup at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler.rb:122
           setup! at /usr/share/logstash/lib/bootstrap/bundler.rb:64
           (root) at ./bin/cpdump:4

[guyboertje]

I would zip/tar up the data/queue dir and copy to your dev machine.
Use a recent download of logstash.zip from github and do bin/bundle install --development
Then try bin/cpdump again.

Make sure you unzip the files into the data/queue folder of the downloaded logstash.

you should see a checkpoint.head file too - cpdump that one too.

The utility takes a relative path arg e.g. bin/cpdump checkpoint.head

[colinsurprenant]

@ipsec or you could also tar&gz/zip all the checkpoint files and send it to us/me for analysis. The checkpoint files only contains pointers and no actual data so there is no data privacy issues is sharing these.

Also, could you tell us how logstash terminated before your tried to restart it and got that error? if it crashed or stopped abnormally could you share the associated logs, and could you also share the exact log(s) you get when stating logstash?

Thanks.

[colinsurprenant]

@ipsec ok, we found why the bin/cpdump does not work outside the dev environment, it is easily fixable on your end, simply open bin/cpdump in an editor and replace the line:

LogStash::Bundler.setup!({:without => [:build]})

with

LogStash::Bundler.setup!({:without => [:build, :development]})

Once this is done could you please share with up the result of

$ bin/cpdump checkpoint.head

and also the result of

$ ls -la data/queue/

[ipsec]

I see kibana without data for two days.
Checking rabbitmq the queue is empty.
Checking logstash its not receiving events.
Then I run service logstash restart and the error starts.
Logstash starts only with memory, not persisted queue.

 # bin/cpdump checkpoint.head
checkpoint pageNum=216, firstUnackedPageNum=6, firstUnackedSeqNum=56317386, minSeqNum=56317386, elementCount=215639, isFullyAcked=no

# bin/cpdump checkpoint.6
checkpoint pageNum=6, firstUnackedPageNum=0, firstUnackedSeqNum=1440859, minSeqNum=1440859, elementCount=273138, isFullyAcked=no

# bin/cpdump checkpoint.215
checkpoint pageNum=215, firstUnackedPageNum=0, firstUnackedSeqNum=56052455, minSeqNum=56052455, elementCount=264931, isFullyAcked=no

# ls -la /data/logstash-queue/
total 23502536
drwxr-xr-x 2 root root     12288 Jan 20 11:42 .
drwxr-xr-x 1 root root        72 Jan 10 09:08 ..
-rw-r--r-- 1 root root        34 Jan 10 15:59 checkpoint.10
-rw-r--r-- 1 root root        34 Jan 15 02:56 checkpoint.100
-rw-r--r-- 1 root root        34 Jan 15 05:12 checkpoint.101
-rw-r--r-- 1 root root        34 Jan 15 06:12 checkpoint.102
-rw-r--r-- 1 root root        34 Jan 15 07:59 checkpoint.103
-rw-r--r-- 1 root root        34 Jan 15 10:43 checkpoint.104
-rw-r--r-- 1 root root        34 Jan 15 12:50 checkpoint.105
-rw-r--r-- 1 root root        34 Jan 15 13:54 checkpoint.106
-rw-r--r-- 1 root root        34 Jan 15 15:58 checkpoint.107
-rw-r--r-- 1 root root        34 Jan 15 17:57 checkpoint.108
-rw-r--r-- 1 root root        34 Jan 15 20:32 checkpoint.109
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.11
-rw-r--r-- 1 root root        34 Jan 15 23:09 checkpoint.110
-rw-r--r-- 1 root root        34 Jan 16 01:27 checkpoint.111
-rw-r--r-- 1 root root        34 Jan 16 04:35 checkpoint.112
-rw-r--r-- 1 root root        34 Jan 16 07:26 checkpoint.113
-rw-r--r-- 1 root root        34 Jan 16 09:00 checkpoint.114
-rw-r--r-- 1 root root        34 Jan 16 10:03 checkpoint.115
-rw-r--r-- 1 root root        34 Jan 16 10:48 checkpoint.116
-rw-r--r-- 1 root root        34 Jan 16 11:32 checkpoint.117
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.118
-rw-r--r-- 1 root root        34 Jan 16 12:17 checkpoint.119
-rw-r--r-- 1 root root        34 Jan 10 17:08 checkpoint.12
-rw-r--r-- 1 root root        34 Jan 16 12:39 checkpoint.120
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.121
-rw-r--r-- 1 root root        34 Jan 16 13:50 checkpoint.122
-rw-r--r-- 1 root root        34 Jan 16 14:22 checkpoint.123
-rw-r--r-- 1 root root        34 Jan 16 14:53 checkpoint.124
-rw-r--r-- 1 root root        34 Jan 16 15:24 checkpoint.125
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.126
-rw-r--r-- 1 root root        34 Jan 16 16:17 checkpoint.127
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.128
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.129
-rw-r--r-- 1 root root        34 Jan 10 17:46 checkpoint.13
-rw-r--r-- 1 root root        34 Jan 16 17:24 checkpoint.130
-rw-r--r-- 1 root root        34 Jan 16 18:04 checkpoint.131
-rw-r--r-- 1 root root        34 Jan 16 18:39 checkpoint.132
-rw-r--r-- 1 root root        34 Jan 16 19:23 checkpoint.133
-rw-r--r-- 1 root root        34 Jan 16 20:30 checkpoint.134
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.135
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.136
-rw-r--r-- 1 root root        34 Jan 16 23:59 checkpoint.137
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.138
-rw-r--r-- 1 root root        34 Jan 17 03:31 checkpoint.139
-rw-r--r-- 1 root root        34 Jan 10 18:24 checkpoint.14
-rw-r--r-- 1 root root        34 Jan 17 05:33 checkpoint.140
-rw-r--r-- 1 root root        34 Jan 17 07:34 checkpoint.141
-rw-r--r-- 1 root root        34 Jan 17 09:03 checkpoint.142
-rw-r--r-- 1 root root        34 Jan 17 09:47 checkpoint.143
-rw-r--r-- 1 root root        34 Jan 17 10:29 checkpoint.144
-rw-r--r-- 1 root root        34 Jan 17 11:17 checkpoint.145
-rw-r--r-- 1 root root        34 Jan 17 11:58 checkpoint.146
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.147
-rw-r--r-- 1 root root        34 Jan 17 13:08 checkpoint.148
-rw-r--r-- 1 root root        34 Jan 17 13:38 checkpoint.149
-rw-r--r-- 1 root root        34 Jan 10 19:06 checkpoint.15
-rw-r--r-- 1 root root        34 Jan 17 14:16 checkpoint.150
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.151
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.152
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.153
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.154
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.155
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.156
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.157
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.158
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.159
-rw-r--r-- 1 root root        34 Jan 10 20:01 checkpoint.16
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.160
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.161
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.162
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.163
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.164
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.165
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.166
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.167
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.168
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.169
-rw-r--r-- 1 root root        34 Jan 10 21:48 checkpoint.17
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.170
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.171
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.172
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.173
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.174
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.175
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.176
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.177
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.178
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.179
-rw-r--r-- 1 root root        34 Jan 11 00:14 checkpoint.18
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.180
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.181
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.182
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.183
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.184
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.185
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.186
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.187
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.188
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.189
-rw-r--r-- 1 root root        34 Jan 11 02:58 checkpoint.19
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.190
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.191
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.192
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.193
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.194
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.195
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.196
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.197
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.198
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.199
-rw-r--r-- 1 root root        34 Jan 11 05:55 checkpoint.20
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.200
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.201
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.202
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.203
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.204
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.205
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.206
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.207
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.208
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.209
-rw-r--r-- 1 root root        34 Jan 11 08:22 checkpoint.21
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.210
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.211
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.212
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.213
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.214
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.215
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.22
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.23
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.24
-rw-r--r-- 1 root root        34 Jan 11 12:18 checkpoint.25
-rw-r--r-- 1 root root        34 Jan 11 12:53 checkpoint.26
-rw-r--r-- 1 root root        34 Jan 11 13:28 checkpoint.27
-rw-r--r-- 1 root root        34 Jan 11 14:05 checkpoint.28
-rw-r--r-- 1 root root        34 Jan 11 14:43 checkpoint.29
-rw-r--r-- 1 root root        34 Jan 11 15:23 checkpoint.30
-rw-r--r-- 1 root root        34 Jan 11 15:55 checkpoint.31
-rw-r--r-- 1 root root        34 Jan 11 16:23 checkpoint.32
-rw-r--r-- 1 root root        34 Jan 11 16:54 checkpoint.33
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.34
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.35
-rw-r--r-- 1 root root        34 Jan 11 18:47 checkpoint.36
-rw-r--r-- 1 root root        34 Jan 11 19:27 checkpoint.37
-rw-r--r-- 1 root root        34 Jan 11 20:42 checkpoint.38
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.39
-rw-r--r-- 1 root root        34 Jan 12 00:17 checkpoint.40
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.41
-rw-r--r-- 1 root root        34 Jan 12 02:27 checkpoint.42
-rw-r--r-- 1 root root        34 Jan 12 05:19 checkpoint.43
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.44
-rw-r--r-- 1 root root        34 Jan 12 09:38 checkpoint.45
-rw-r--r-- 1 root root        34 Jan 12 10:52 checkpoint.46
-rw-r--r-- 1 root root        34 Jan 12 11:45 checkpoint.47
-rw-r--r-- 1 root root        34 Jan 12 12:25 checkpoint.48
-rw-r--r-- 1 root root        34 Jan 12 13:03 checkpoint.49
-rw-r--r-- 1 root root        34 Jan 12 13:37 checkpoint.50
-rw-r--r-- 1 root root        34 Jan 12 14:07 checkpoint.51
-rw-r--r-- 1 root root        34 Jan 12 14:33 checkpoint.52
-rw-r--r-- 1 root root        34 Jan 12 15:02 checkpoint.53
-rw-r--r-- 1 root root        34 Jan 12 15:35 checkpoint.54
-rw-r--r-- 1 root root        34 Jan 12 16:13 checkpoint.55
-rw-r--r-- 1 root root        34 Jan 12 16:43 checkpoint.56
-rw-r--r-- 1 root root        34 Jan 12 17:07 checkpoint.57
-rw-r--r-- 1 root root        34 Jan 12 17:42 checkpoint.58
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.59
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.6
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.60
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.61
-rw-r--r-- 1 root root        34 Jan 12 21:55 checkpoint.62
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.63
-rw-r--r-- 1 root root        34 Jan 13 01:46 checkpoint.64
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.65
-rw-r--r-- 1 root root        34 Jan 13 07:06 checkpoint.66
-rw-r--r-- 1 root root        34 Jan 13 08:54 checkpoint.67
-rw-r--r-- 1 root root        34 Jan 13 10:03 checkpoint.68
-rw-r--r-- 1 root root        34 Jan 13 11:02 checkpoint.69
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.7
-rw-r--r-- 1 root root        34 Jan 13 11:53 checkpoint.70
-rw-r--r-- 1 root root        34 Jan 13 12:26 checkpoint.71
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.72
-rw-r--r-- 1 root root        34 Jan 13 13:34 checkpoint.73
-rw-r--r-- 1 root root        34 Jan 13 14:07 checkpoint.74
-rw-r--r-- 1 root root        34 Jan 13 14:42 checkpoint.75
-rw-r--r-- 1 root root        34 Jan 13 15:18 checkpoint.76
-rw-r--r-- 1 root root        34 Jan 13 15:54 checkpoint.77
-rw-r--r-- 1 root root        34 Jan 13 16:31 checkpoint.78
-rw-r--r-- 1 root root        34 Jan 13 17:12 checkpoint.79
-rw-r--r-- 1 root root        34 Jan 10 14:45 checkpoint.8
-rw-r--r-- 1 root root        34 Jan 13 17:58 checkpoint.80
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.81
-rw-r--r-- 1 root root        34 Jan 13 19:47 checkpoint.82
-rw-r--r-- 1 root root        34 Jan 13 21:11 checkpoint.83
-rw-r--r-- 1 root root        34 Jan 13 22:06 checkpoint.84
-rw-r--r-- 1 root root        34 Jan 13 22:47 checkpoint.85
-rw-r--r-- 1 root root        34 Jan 13 23:32 checkpoint.86
-rw-r--r-- 1 root root        34 Jan 14 01:33 checkpoint.87
-rw-r--r-- 1 root root        34 Jan 14 04:34 checkpoint.88
-rw-r--r-- 1 root root        34 Jan 14 07:38 checkpoint.89
-rw-r--r-- 1 root root        34 Jan 10 15:23 checkpoint.9
-rw-r--r-- 1 root root        34 Jan 14 10:23 checkpoint.90
-rw-r--r-- 1 root root        34 Jan 14 13:02 checkpoint.91
-rw-r--r-- 1 root root        34 Jan 14 15:50 checkpoint.92
-rw-r--r-- 1 root root        34 Jan 14 17:59 checkpoint.93
-rw-r--r-- 1 root root        34 Jan 14 19:24 checkpoint.94
-rw-r--r-- 1 root root        34 Jan 14 20:28 checkpoint.95
-rw-r--r-- 1 root root        34 Jan 14 22:00 checkpoint.96
-rw-r--r-- 1 root root        34 Jan 14 22:30 checkpoint.97
-rw-r--r-- 1 root root        34 Jan 14 23:29 checkpoint.98
-rw-r--r-- 1 root root        34 Jan 15 00:37 checkpoint.99
-rw-r--r-- 1 root root        34 Jan 20 11:41 checkpoint.head
-rw-r--r-- 1 root root 262144000 Jan 10 16:31 page.11
-rw-r--r-- 1 root root 262144000 Jan 16 11:56 page.118
-rw-r--r-- 1 root root 262144000 Jan 16 13:16 page.121
-rw-r--r-- 1 root root 262144000 Jan 16 15:56 page.126
-rw-r--r-- 1 root root 262144000 Jan 16 16:37 page.128
-rw-r--r-- 1 root root 262144000 Jan 16 17:03 page.129
-rw-r--r-- 1 root root 262144000 Jan 16 22:10 page.135
-rw-r--r-- 1 root root 262144000 Jan 16 23:12 page.136
-rw-r--r-- 1 root root 262144000 Jan 17 02:02 page.138
-rw-r--r-- 1 root root 262144000 Jan 17 12:34 page.147
-rw-r--r-- 1 root root 262144000 Jan 20 11:42 page.15
-rw-r--r-- 1 root root 262144000 Jan 17 14:47 page.151
-rw-r--r-- 1 root root 262144000 Jan 17 15:22 page.152
-rw-r--r-- 1 root root 262144000 Jan 17 15:56 page.153
-rw-r--r-- 1 root root 262144000 Jan 17 16:31 page.154
-rw-r--r-- 1 root root 262144000 Jan 17 17:06 page.155
-rw-r--r-- 1 root root 262144000 Jan 17 17:46 page.156
-rw-r--r-- 1 root root 262144000 Jan 17 18:26 page.157
-rw-r--r-- 1 root root 262144000 Jan 17 19:07 page.158
-rw-r--r-- 1 root root 262144000 Jan 17 20:09 page.159
-rw-r--r-- 1 root root 262144000 Jan 17 21:58 page.160
-rw-r--r-- 1 root root 262144000 Jan 18 00:05 page.161
-rw-r--r-- 1 root root 262144000 Jan 18 02:23 page.162
-rw-r--r-- 1 root root 262144000 Jan 18 04:50 page.163
-rw-r--r-- 1 root root 262144000 Jan 18 07:18 page.164
-rw-r--r-- 1 root root 262144000 Jan 18 09:05 page.165
-rw-r--r-- 1 root root 262144000 Jan 18 10:16 page.166
-rw-r--r-- 1 root root 262144000 Jan 18 11:10 page.167
-rw-r--r-- 1 root root 262144000 Jan 18 11:53 page.168
-rw-r--r-- 1 root root 262144000 Jan 18 12:32 page.169
-rw-r--r-- 1 root root 262144000 Jan 18 13:04 page.170
-rw-r--r-- 1 root root 262144000 Jan 18 13:34 page.171
-rw-r--r-- 1 root root 262144000 Jan 18 14:04 page.172
-rw-r--r-- 1 root root 262144000 Jan 18 14:32 page.173
-rw-r--r-- 1 root root 262144000 Jan 18 15:02 page.174
-rw-r--r-- 1 root root 262144000 Jan 18 15:35 page.175
-rw-r--r-- 1 root root 262144000 Jan 18 16:08 page.176
-rw-r--r-- 1 root root 262144000 Jan 18 16:35 page.177
-rw-r--r-- 1 root root 262144000 Jan 18 17:07 page.178
-rw-r--r-- 1 root root 262144000 Jan 18 17:38 page.179
-rw-r--r-- 1 root root 262144000 Jan 18 18:16 page.180
-rw-r--r-- 1 root root 262144000 Jan 18 19:01 page.181
-rw-r--r-- 1 root root 262144000 Jan 18 19:53 page.182
-rw-r--r-- 1 root root 262144000 Jan 18 21:22 page.183
-rw-r--r-- 1 root root 262144000 Jan 18 23:25 page.184
-rw-r--r-- 1 root root 262144000 Jan 19 01:35 page.185
-rw-r--r-- 1 root root 262144000 Jan 19 03:55 page.186
-rw-r--r-- 1 root root 262144000 Jan 19 06:11 page.187
-rw-r--r-- 1 root root 262144000 Jan 19 08:06 page.188
-rw-r--r-- 1 root root 262144000 Jan 19 09:23 page.189
-rw-r--r-- 1 root root 262144000 Jan 19 10:21 page.190
-rw-r--r-- 1 root root 262144000 Jan 19 11:11 page.191
-rw-r--r-- 1 root root 262144000 Jan 19 11:57 page.192
-rw-r--r-- 1 root root 262144000 Jan 19 12:29 page.193
-rw-r--r-- 1 root root 262144000 Jan 19 13:02 page.194
-rw-r--r-- 1 root root 262144000 Jan 19 13:32 page.195
-rw-r--r-- 1 root root 262144000 Jan 19 14:04 page.196
-rw-r--r-- 1 root root 262144000 Jan 19 14:37 page.197
-rw-r--r-- 1 root root 262144000 Jan 19 15:06 page.198
-rw-r--r-- 1 root root 262144000 Jan 19 15:38 page.199
-rw-r--r-- 1 root root 262144000 Jan 19 16:12 page.200
-rw-r--r-- 1 root root 262144000 Jan 19 16:47 page.201
-rw-r--r-- 1 root root 262144000 Jan 19 17:19 page.202
-rw-r--r-- 1 root root 262144000 Jan 19 17:49 page.203
-rw-r--r-- 1 root root 262144000 Jan 19 18:24 page.204
-rw-r--r-- 1 root root 262144000 Jan 19 19:05 page.205
-rw-r--r-- 1 root root 262144000 Jan 19 19:49 page.206
-rw-r--r-- 1 root root 262144000 Jan 19 21:13 page.207
-rw-r--r-- 1 root root 262144000 Jan 19 23:19 page.208
-rw-r--r-- 1 root root 262144000 Jan 20 01:32 page.209
-rw-r--r-- 1 root root 262144000 Jan 20 03:54 page.210
-rw-r--r-- 1 root root 262144000 Jan 20 06:07 page.211
-rw-r--r-- 1 root root 262144000 Jan 20 08:07 page.212
-rw-r--r-- 1 root root 262144000 Jan 20 09:27 page.213
-rw-r--r-- 1 root root 262144000 Jan 20 10:24 page.214
-rw-r--r-- 1 root root 262144000 Jan 20 11:07 page.215
-rw-r--r-- 1 root root 262144000 Jan 20 11:41 page.216
-rw-r--r-- 1 root root 262144000 Jan 11 09:50 page.22
-rw-r--r-- 1 root root 262144000 Jan 11 10:50 page.23
-rw-r--r-- 1 root root 262144000 Jan 11 11:43 page.24
-rw-r--r-- 1 root root 262144000 Jan 11 17:24 page.34
-rw-r--r-- 1 root root 262144000 Jan 11 18:01 page.35
-rw-r--r-- 1 root root 262144000 Jan 11 22:19 page.39
-rw-r--r-- 1 root root 262144000 Jan 12 02:01 page.41
-rw-r--r-- 1 root root 262144000 Jan 12 07:56 page.44
-rw-r--r-- 1 root root 262144000 Jan 12 18:22 page.59
-rw-r--r-- 1 root root 262144000 Jan 10 13:35 page.6
-rw-r--r-- 1 root root 262144000 Jan 12 19:02 page.60
-rw-r--r-- 1 root root 262144000 Jan 12 20:01 page.61
-rw-r--r-- 1 root root 262144000 Jan 12 23:23 page.63
-rw-r--r-- 1 root root 262144000 Jan 13 04:27 page.65
-rw-r--r-- 1 root root 262144000 Jan 13 13:00 page.72
-rw-r--r-- 1 root root 262144000 Jan 13 18:47 page.81

[2017-01-20T11:41:38,748][WARN ][logstash.runner          ] SIGTERM received. Shutting down the agent.
[2017-01-20T11:41:38,776][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}
[2017-01-20T11:41:39,216][WARN ][logstash.inputs.udp      ] UDP listener died {:exception=>#<IOError: closed stream>, :backtrace=>["org/jruby/RubyIO.java:3705:in `select'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.0/lib/logstash/inputs/udp.rb:93:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.0/lib/logstash/inputs/udp.rb:56:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:365:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:359:in `start_input'"]}
[2017-01-20T11:41:43,513][WARN ][logstash.outputs.rabbitmq] RabbitMQ connection was closed! {:url=>"amqp://logstash-account:XXXXXX@srvrabbit01.mydomain.com:5672logstash", :automatic_recovery=>true, :cause=>com.rabbitmq.client.ShutdownSignalException: clean connection shutdown;protocol method: #method<connection.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0)}
[2017-01-20T11:42:17,353][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n  udp {\n    port => 514\n      type => syslog\n  }\n\n  tcp {\n    port => 514\n      type => syslog\n  }\n\n  beats {\n    port => 6000\n  }\n}\n\noutput { \n  rabbitmq {\n    exchange => \"logstash-exchange\"\n    exchange_type => \"direct\"\n    key => \"logstash-routing-key\"\n    host => \"srvrabbit01.mydomain.com\"\n    durable => true\n    persistent => true\n    user => \"logstash-account\"\n    password => \"lksdj0987123kmsdfy79012\"\n    vhost => \"logstash\"\n  }\n}\n\n", :reason=>"first seqNum=0 is different than minSeqNum=3879808"}
[2017-01-20T11:42:35,021][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n  udp {\n    port => 514\n      type => syslog\n  }\n\n  tcp {\n    port => 514\n      type => syslog\n  }\n\n  beats {\n    port => 6000\n  }\n}\n\noutput { \n  rabbitmq {\n    exchange => \"logstash-exchange\"\n    exchange_type => \"direct\"\n    key => \"logstash-routing-key\"\n    host => \"srvrabbit01.mydomain.com\"\n    durable => true\n    persistent => true\n    user => \"logstash-account\"\n    password => \"lksdj0987123kmsdfy79012\"\n    vhost => \"logstash\"\n  }\n}\n\n", :reason=>"first seqNum=0 is different than minSeqNum=3879808"}
[2017-01-20T11:42:52,723][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n  udp {\n    port => 514\n      type => syslog\n  }\n\n  tcp {\n    port => 514\n      type => syslog\n  }\n\n  beats {\n    port => 6000\n  }\n}\n\noutput { \n  rabbitmq {\n    exchange => \"logstash-exchange\"\n    exchange_type => \"direct\"\n    key => \"logstash-routing-key\"\n    host => \"srvrabbit01.mydomain.com\"\n    durable => true\n    persistent => true\n    user => \"logstash-account\"\n    password => \"lksdj0987123kmsdfy79012\"\n    vhost => \"logstash\"\n  }\n}\n\n", :reason=>"first seqNum=0 is different than minSeqNum=3879808"}
[2017-01-20T11:43:10,108][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n  udp {\n    port => 514\n      type => syslog\n  }\n\n  tcp {\n    port => 514\n      type => syslog\n  }\n\n  beats {\n    port => 6000\n  }\n}\n\noutput { \n  rabbitmq {\n    exchange => \"logstash-exchange\"\n    exchange_type => \"direct\"\n    key => \"logstash-routing-key\"\n    host => \"srvrabbit01.mydomain.com\"\n    durable => true\n    persistent => true\n    user => \"logstash-account\"\n    password => \"lksdj0987123kmsdfy79012\"\n    vhost => \"logstash\"\n  }\n}\n\n", :reason=>"first seqNum=0 is different than minSeqNum=3879808"}
[2017-01-20T11:43:27,870][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n  udp {\n    port => 514\n      type => syslog\n  }\n\n  tcp {\n    port => 514\n      type => syslog\n  }\n\n  beats {\n    port => 6000\n  }\n}\n\noutput { \n  rabbitmq {\n    exchange => \"logstash-exchange\"\n    exchange_type => \"direct\"\n    key => \"logstash-routing-key\"\n    host => \"srvrabbit01.mydomain.com\"\n    durable => true\n    persistent => true\n    user => \"logstash-account\"\n    password => \"lksdj0987123kmsdfy79012\"\n    vhost => \"logstash\"\n  }\n}\n\n", :reason=>"first seqNum=0 is different than minSeqNum=3879808"}
[2017-01-20T11:43:45,668][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n  udp {\n    port => 514\n      type => syslog\n  }\n\n  tcp {\n    port => 514\n      type => syslog\n  }\n\n  beats {\n    port => 6000\n  }\n}\n\noutput { \n  rabbitmq {\n    exchange => \"logstash-exchange\"\n    exchange_type => \"direct\"\n    key => \"logstash-routing-key\"\n    host => \"srvrabbit01.mydomain.com\"\n    durable => true\n    persistent => true\n    user => \"logstash-account\"\n    password => \"lksdj0987123kmsdfy79012\"\n    vhost => \"logstash\"\n  }\n}\n\n", :reason=>"first seqNum=0 is different than minSeqNum=3879808"}
[2017-01-20T11:44:03,729][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n  udp {\n    port => 514\n      type => syslog\n  }\n\n  tcp {\n    port => 514\n      type => syslog\n  }\n\n  beats {\n    port => 6000\n  }\n}\n\noutput { \n  rabbitmq {\n    exchange => \"logstash-exchange\"\n    exchange_type => \"direct\"\n    key => \"logstash-routing-key\"\n    host => \"srvrabbit01.mydomain.com\"\n    durable => true\n    persistent => true\n    user => \"logstash-account\"\n    password => \"lksdj0987123kmsdfy79012\"\n    vhost => \"logstash\"\n  }\n}\n\n", :reason=>"first seqNum=0 is different than minSeqNum=3879808"}
... repeated messages
[2017-01-20T11:52:29,540][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n  udp {\n    port => 514\n      type => syslog\n  }\n\n  tcp {\n    port => 514\n      type => syslog\n  }\n\n  beats {\n    port => 6000\n  }\n}\n\noutput { \n  rabbitmq {\n    exchange => \"logstash-exchange\"\n    exchange_type => \"direct\"\n    key => \"logstash-routing-key\"\n    host => \"srvrabbit01.mydomain.com\"\n    durable => true\n    persistent => true\n    user => \"logstash-account\"\n    password => \"lksdj0987123kmsdfy79012\"\n    vhost => \"logstash\"\n  }\n}\n\n", :reason=>"first seqNum=0 is different than minSeqNum=3879808"}
[2017-01-20T11:52:47,930][WARN ][logstash.runner          ] --config.debug was specified, but log.level was not set to 'debug'! No config info will be logged.
[2017-01-20T11:52:47,969][DEBUG][logstash.runner          ] -------- Logstash Settings (* means modified) ---------
[2017-01-20T11:52:47,969][DEBUG][logstash.runner          ] node.name: "svlp-logstash01.mydomain.com"
[2017-01-20T11:52:47,969][DEBUG][logstash.runner          ] *path.config: "/etc/logstash/conf.d/logstash.conf"
[2017-01-20T11:52:47,969][DEBUG][logstash.runner          ] *path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data")
[2017-01-20T11:52:47,969][DEBUG][logstash.runner          ] *config.test_and_exit: true (default: false)
[2017-01-20T11:52:47,970][DEBUG][logstash.runner          ] config.reload.automatic: false
[2017-01-20T11:52:47,970][DEBUG][logstash.runner          ] config.reload.interval: 3
[2017-01-20T11:52:47,970][DEBUG][logstash.runner          ] metric.collect: true
[2017-01-20T11:52:47,970][DEBUG][logstash.runner          ] pipeline.id: "main"
[2017-01-20T11:52:47,970][DEBUG][logstash.runner          ] pipeline.workers: 2
[2017-01-20T11:52:47,970][DEBUG][logstash.runner          ] pipeline.output.workers: 1
[2017-01-20T11:52:47,970][DEBUG][logstash.runner          ] pipeline.batch.size: 125
[2017-01-20T11:52:47,970][DEBUG][logstash.runner          ] pipeline.batch.delay: 5
[2017-01-20T11:52:47,970][DEBUG][logstash.runner          ] pipeline.unsafe_shutdown: false
[2017-01-20T11:52:47,971][DEBUG][logstash.runner          ] path.plugins: []
[2017-01-20T11:52:47,971][DEBUG][logstash.runner          ] *config.debug: true (default: false)
[2017-01-20T11:52:47,971][DEBUG][logstash.runner          ] *log.level: "debug" (default: "info")
[2017-01-20T11:52:47,971][DEBUG][logstash.runner          ] version: false
[2017-01-20T11:52:47,972][DEBUG][logstash.runner          ] help: false
[2017-01-20T11:52:47,972][DEBUG][logstash.runner          ] log.format: "plain"
[2017-01-20T11:52:47,972][DEBUG][logstash.runner          ] http.host: "127.0.0.1"
[2017-01-20T11:52:47,972][DEBUG][logstash.runner          ] http.port: 9600..9700
[2017-01-20T11:52:47,972][DEBUG][logstash.runner          ] http.environment: "production"
[2017-01-20T11:52:47,972][DEBUG][logstash.runner          ] *queue.type: "persisted" (default: "memory")
[2017-01-20T11:52:47,972][DEBUG][logstash.runner          ] queue.page_capacity: 262144000
[2017-01-20T11:52:47,972][DEBUG][logstash.runner          ] *queue.max_bytes: 37580963840 (default: 1073741824)
[2017-01-20T11:52:47,973][DEBUG][logstash.runner          ] queue.max_events: 0
[2017-01-20T11:52:47,973][DEBUG][logstash.runner          ] queue.checkpoint.acks: 1024
[2017-01-20T11:52:47,973][DEBUG][logstash.runner          ] queue.checkpoint.writes: 1024
[2017-01-20T11:52:47,973][DEBUG][logstash.runner          ] queue.checkpoint.interval: 1000
[2017-01-20T11:52:47,973][DEBUG][logstash.runner          ] slowlog.threshold.warn: -1
[2017-01-20T11:52:47,973][DEBUG][logstash.runner          ] slowlog.threshold.info: -1
[2017-01-20T11:52:47,973][DEBUG][logstash.runner          ] slowlog.threshold.debug: -1
[2017-01-20T11:52:47,973][DEBUG][logstash.runner          ] slowlog.threshold.trace: -1
[2017-01-20T11:52:47,973][DEBUG][logstash.runner          ] *path.queue: "/data/logstash-queue" (default: "/usr/share/logstash/data/queue")
[2017-01-20T11:52:47,973][DEBUG][logstash.runner          ] *path.settings: "/etc/logstash" (default: "/usr/share/logstash/config")
[2017-01-20T11:52:47,974][DEBUG][logstash.runner          ] *path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs")
[2017-01-20T11:52:47,974][DEBUG][logstash.runner          ] --------------- Logstash Settings -------------------
[2017-01-20T11:52:47,976][INFO ][logstash.runner          ] No config files found in path {:path=>"/etc/logstash/conf.d/logstash.conf"}
[2017-01-20T11:52:47,981][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<RuntimeError: No config files found: /etc/logstash/conf.d/logstash.conf. Can you make sure this path is a logstash config file?>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/loader.rb:18:in `format_config'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:245:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:178:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}
[2017-01-20T11:53:01,854][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n  udp {\n    port => 514\n      type => syslog\n  }\n\n  tcp {\n    port => 514\n      type => syslog\n  }\n\n  beats {\n    port => 6000\n  }\n}\n\noutput { \n  rabbitmq {\n    exchange => \"logstash-exchange\"\n    exchange_type => \"direct\"\n    key => \"logstash-routing-key\"\n    host => \"srvrabbit01.mydomain.com\"\n    durable => true\n    persistent => true\n    user => \"logstash-account\"\n    password => \"lksdj0987123kmsdfy79012\"\n    vhost => \"logstash\"\n  }\n}\n\n", :reason=>"first seqNum=0 is different than minSeqNum=3879808"}
[2017-01-20T11:53:19,411][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n  udp {\n    port => 514\n      type => syslog\n  }\n\n  tcp {\n    port => 514\n      type => syslog\n  }\n\n  beats {\n    port => 6000\n  }\n}\n\noutput { \n  rabbitmq {\n    exchange => \"logstash-exchange\"\n    exchange_type => \"direct\"\n    key => \"logstash-routing-key\"\n    host => \"srvrabbit01.mydomain.com\"\n    durable => true\n    persistent => true\n    user => \"logstash-account\"\n    password => \"lksdj0987123kmsdfy79012\"\n    vhost => \"logstash\"\n  }\n}\n\n", :reason=>"first seqNum=0 is different than minSeqNum=3879808"}
... repeated messages
[2017-01-20T12:11:33,161][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n  udp {\n    port => 514\n      type => syslog\n  }\n\n  tcp {\n    port => 514\n      type => syslog\n  }\n\n  beats {\n    port => 6000\n  }\n}\n\noutput { \n  rabbitmq {\n    exchange => \"logstash-exchange\"\n    exchange_type => \"direct\"\n    key => \"logstash-routing-key\"\n    host => \"srvrabbit01.mydomain.com\"\n    durable => true\n    persistent => true\n    user => \"logstash-account\"\n    password => \"lksdj0987123kmsdfy79012\"\n    vhost => \"logstash\"\n  }\n}\n\n", :reason=>"first seqNum=0 is different than minSeqNum=3879808"}
[2017-01-20T12:12:51,721][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:514"}
[2017-01-20T12:12:51,724][INFO ][logstash.inputs.tcp      ] Starting tcp input listener {:address=>"0.0.0.0:514"}
[2017-01-20T12:12:51,765][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:514", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2017-01-20T12:12:52,553][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:6000"}
[2017-01-20T12:12:52,638][INFO ][org.logstash.beats.Server] Starting server on port: 6000
[2017-01-20T12:12:52,812][INFO ][logstash.outputs.rabbitmq] Connected to RabbitMQ at
[2017-01-20T12:12:52,913][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-01-20T12:12:52,917][INFO ][logstash.pipeline        ] Pipeline main started
[2017-01-20T12:12:52,977][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

[colinsurprenant]

Ok thanks. the error you have is first seqNum=0 is different than minSeqNum=3879808 so let's try to find in which page this minSeqNum=3879808 is in by running:

for f in data/queue/checkpoint.*; do bin/cpdump $(basename $f) | grep 3879808; done

[ipsec]

:/usr/share/logstash # for f in data/queue/checkpoint.*; do bin/cpdump $(basename $f) | grep 3879808; done
checkpoint pageNum=14, firstUnackedPageNum=0, firstUnackedSeqNum=3879808, minSeqNum=3613637, elementCount=266171, isFullyAcked=yes
checkpoint pageNum=15, firstUnackedPageNum=0, firstUnackedSeqNum=4146085, minSeqNum=3879808, elementCount=266286, isFullyAcked=no

[colinsurprenant]

@ipsec ok, thanks - now could you do a dump of the first bytes of both pages 14 and 15 with:

hexdump data/queue/page.14 | head

hexdump data/queue/page.15 | head