Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[volume_loading] volumeDataIntegrityChecking failed verifyNeedleIntegrity size mismatch #2630

Closed
kmlebedev opened this issue Feb 3, 2022 · 5 comments
Closed

[volume_loading] volumeDataIntegrityChecking failed verifyNeedleIntegrity size mismatch #2630

kmlebedev opened this issue Feb 3, 2022 · 5 comments

Comments

@kmlebedev
Copy link
Contributor

kmlebedev commented Feb 3, 2022 •
edited

weed version

2.87

I unexpectedly massively (more than 20 volumes) received volume loading errors after another update with restarting volumes

Jan 28, 2022 @ 13:00:59.912	I0128 08:00:53     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_210.idx failed: read data [24264,32) : size mismatch

Jan 28, 2022 @ 13:00:59.917	I0128 08:00:53     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_258.idx failed: read data [179803856,32) : size mismatch

Jan 28, 2022 @ 13:00:59.917	I0128 08:00:53     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_146.idx failed: read data [4988464,32) : size mismatch

Jan 28, 2022 @ 13:00:59.917	I0128 08:00:53     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_256.idx failed: read data [257705480,32) : size mismatch

Jan 28, 2022 @ 13:04:21.699	I0128 08:04:13     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_210.idx failed: read data [24264,32) : size mismatch

Jan 28, 2022 @ 13:04:21.702	I0128 08:04:13     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_256.idx failed: read data [257705480,32) : size mismatch

Jan 28, 2022 @ 13:07:42.939	I0128 08:07:33     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_94.idx failed: read data [1384568,32) : size mismatch

Jan 28, 2022 @ 13:07:42.945	I0128 08:07:33     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_118.idx failed: read data [38372008,32) : size mismatch

Jan 28, 2022 @ 13:07:42.946	I0128 08:07:33     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_257.idx failed: read data [223556776,32) : size mismatch

Jan 28, 2022 @ 13:07:42.947	I0128 08:07:33     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_123.idx failed: read data [3327824,32) : size mismatch

Jan 28, 2022 @ 13:11:10.969	I0128 08:11:03     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_95.idx failed: read data [1709848,32) : size mismatch

Jan 28, 2022 @ 13:11:10.977	I0128 08:11:03     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_257.idx failed: read data [223556776,32) : size mismatch

Jan 28, 2022 @ 13:11:10.977	I0128 08:11:03     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_146.idx failed: read data [4988464,32) : size mismatch

Jan 28, 2022 @ 13:11:10.978	I0128 08:11:03     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_123.idx failed: read data [3327824,32) : size mismatch

Jan 28, 2022 @ 13:28:50.993	I0128 08:28:44     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_95.idx failed: read data [1709848,32) : size mismatch

Jan 28, 2022 @ 13:28:50.999	I0128 08:28:44     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_146.idx failed: read data [4988464,32) : size mismatch

Jan 28, 2022 @ 13:28:51.000	I0128 08:28:44     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_257.idx failed: read data [223556776,32) : size mismatch

Jan 28, 2022 @ 13:28:51.001	I0128 08:28:44     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_123.idx failed: read data [3327824,32) : size mismatch

Jan 28, 2022 @ 13:30:22.946	I0128 08:30:17     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_94.idx failed: read data [1384568,32) : size mismatch

Jan 28, 2022 @ 13:30:22.954	I0128 08:30:17     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_118.idx failed: read data [38372008,32) : size mismatch

Jan 28, 2022 @ 13:30:22.955	I0128 08:30:17     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_257.idx failed: read data [223556776,32) : size mismatch

Jan 28, 2022 @ 13:30:22.957	I0128 08:30:17     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_123.idx failed: read data [3327824,32) : size mismatch

Jan 28, 2022 @ 13:32:41.690	I0128 08:32:34     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_210.idx failed: read data [24264,32) : size mismatch

Jan 28, 2022 @ 13:32:41.694	I0128 08:32:34     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_256.idx failed: read data [257705480,32) : size mismatch

Jan 28, 2022 @ 13:34:30.976	I0128 08:34:25     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_95.idx failed: read data [1709848,32) : size mismatch

Jan 28, 2022 @ 13:34:30.983	I0128 08:34:25     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_146.idx failed: read data [4988464,32) : size mismatch

Jan 28, 2022 @ 13:34:30.984	I0128 08:34:25     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_257.idx failed: read data [223556776,32) : size mismatch

Jan 28, 2022 @ 13:34:30.985	I0128 08:34:25     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_123.idx failed: read data [3327824,32) : size mismatch

Jan 28, 2022 @ 13:38:40.985	I0128 08:38:35     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_95.idx failed: read data [1709848,32) : size mismatch

Jan 28, 2022 @ 13:38:40.995	I0128 08:38:35     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_146.idx failed: read data [4988464,32) : size mismatch

Jan 28, 2022 @ 13:38:40.995	I0128 08:38:35     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_257.idx failed: read data [223556776,32) : size mismatch

Jan 28, 2022 @ 13:38:40.996	I0128 08:38:35     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/_123.idx failed: read data [3327824,32) : size mismatch

https://github.com/chrislusf/seaweedfs/blob/master/weed/storage/volume_checking.go#L57
@chrislusf
Copy link
Collaborator

It is trying to check the last entry is valid or not. But it is a deleted entry. Can you one pair of .dat and .idx files?

@kmlebedev
Copy link
Contributor Author

kmlebedev commented Feb 7, 2022 •
edited

It is trying to check the last entry is valid or not. But it is a deleted entry. Can you one pair of .dat and .idx files?

@chrislusf I send pair to https://seaweedfs.slack.com/archives/D01ETGL89CL/p1644229014304329

Also I tried to do

Example: weed fix [-volumeId=234] [-collection=bigData] /tmp
Default Usage:
  -collection string
    	an optional volume collection name, if specified only it will be processed
  -options string
    	a file of command line options, each line in optionName=optionValue format
  -volumeId int
    	an optional volume id, if not 0 (default) only it will be processed
Description:
  Fix runs the SeaweedFS fix command on dat files or whole folders to re-create the index .idx file.
/data # weed fix -volumeId 123 /data
/data # ls -l /data/*123*
-rw-rw-r--    1 root     99         3327856 Jan 17 15:22 /data/doc_123.dat
-rw-rw-r--    1 root     99            1296 Jan 17 15:22 /data/doc_123.idx
-rw-rw-r--    1 root     99             848 Jan 18 08:14 /data/doc_123.sdx
-rw-rw-r--    1 root     99              57 Sep 16 13:22 /data/doc_123.vif

after restarting the volume again received

kubectl --context stage-dcdp -n s3 logs fast-volume-1 -c seaweedfs | grep -e failed -e error
I0207 09:54:30     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/210.idx failed: read data [163080,32) : size mismatch
I0207 09:54:30     1 volume_loading.go:126] volumeDataIntegrityChecking failed size mismatch
I0207 09:54:30     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/118.idx failed: read data [39748408,32) : size mismatch
I0207 09:54:30     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/146.idx failed: read data [5862016,32) : size mismatch
I0207 09:54:30     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/256.idx failed: read data [257705480,32) : size mismatch
I0207 09:54:30     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/257.idx failed: read data [223556776,32) : size mismatch
I0207 09:54:30     1 volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/123.idx failed: read data [3327824,32) : size mismatch

I also tried to mark volumes for recording and made a volume check.

volume.check.disk -v -force -slow

with missed 0 entries

@chrislusf
Copy link
Collaborator

The volume .dat file does miss the last few deleted entries. You can ignore this for now.

@kmlebedev kmlebedev mentioned this issue Feb 7, 2022
Merged
kmlebedev added a commit to kmlebedev/seaweedfs that referenced this issue Feb 8, 2022
@kmlebedev
graceful stop cluster http server default timeout 60 sec
22a3ef5 
seaweedfs#2630
@kmlebedev kmlebedev mentioned this issue Feb 8, 2022
Merged
@kmlebedev
Copy link
Contributor Author

kmlebedev commented Feb 13, 2022 •
edited

be reproduced when an OOM killer arrives

volume_loading.go:126] volumeDataIntegrityChecking failed verifyNeedleIntegrity /data/1090.idx failed: read data [10562905296,32) : size mismatch

@kmlebedev kmlebedev mentioned this issue Feb 14, 2022
Merged
@kmlebedev
Copy link
Contributor Author

fsync added

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@chrislusf @kmlebedev