!URGENT URGENT URGENT! Wrong volume size reported => upload error

[brstgt]

Since 0.73 volume sizes seem to be reported * 1024.
So volume size exceeds max volume size and upload is denied but volume is marked as writeable.
Unfortunately we cannot switch back to 0.7 or 0.72 as these throw other critical errors.

Please fix this ASAP. Our production system with hundrets of millions of files is currently fucked.
As a workaround we try to raise max_volume_size * 1024 but I don't know if there are any side effects.

Example - uploaded files in collection have approx 1MB, not 1GB:
https://cl.ly/3h0i0220032L

[ijunaid8989]

Switch to 0.70 ? why it will send errors? on which version you were working already?

[brstgt]

0.70 crashed due to concurrent map things We used 0.7 before but when rolling back it crashes all the time. Currently 0.73 installed. I already found a bug in store.go if MaxPossibleVolumeSize >= v.ContentSize()+uint64(size) { Should not use constant but s.VolumeSizeLimit But this does not fix the wrong volume sizes 2017-01-18 11:27 GMT+01:00 Junaid Farooq <notifications@github.com>:

…

Switch to 0.70 ? why it will send errors? on which version you were working already? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#441 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUQZ5YaeiVG7_X2e-yC2Rm_-PZdwip8ks5rTekMgaJpZM4LmtFU> .

[brstgt]

Please take a look at needle_map_memory.go LoadNeedleMap

	nm.FileByteCounter = nm.FileByteCounter + uint64(size)
	if offset > 0 && size != TombstoneFileSize {

as size is MaxUint32 it MUST NOT increase the FileByteCounter. This is what i found but i dont know if there are other places where this has more effects

[chrislusf]

As an urgent fix, please change the value of TombstoneFileSize to 0 for now.

[brstgt]

I already rolled out the pull request I've sent you. It seems to work from what I can see. Let me know if you see any problems with this patch. 2017-01-18 16:34 GMT+01:00 Chris Lu <notifications@github.com>:

…

As an urgent fix, please change the value of TombstoneFileSize to 0 for now. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#441 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUQZ27_UcE5PSMD2i96kqM_SuLOICsKks5rTjD8gaJpZM4LmtFU> .

[brstgt]

Thanks but it still does not fix the hardcoded max volume size Am 18.01.2017 17:40 schrieb "Chris Lu" <notifications@github.com>:

…

Closed #441 <#441> via 59022b6 <59022b6> . — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#441 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUQZ7vFGcpCqiQaldFyXbr2iFMjvO1uks5rTkCRgaJpZM4LmtFU> .

[chrislusf]

That's by design. The system allows some extra files, for example, new deletion after the size limit is reached.

[brstgt]

Sorry but to me it really seems like a bug! Today we were not able to raise max vol size above 32gb. Am 18.01.2017 17:45 schrieb "Chris Lu" <notifications@github.com>:

…

That's by design. The system allows some extra files, for example, new deletion after the size limit is reached. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#441 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUQZy0WNHjg4gznJD7n22f_NtRBnklfks5rTkGqgaJpZM4LmtFU> .

[chrislusf]

For more than 32GB, it is more than fixing the limit. Current offset can only address to 32GB.

[chrislusf]

What's your intended volume size, btw?

[brstgt]

As a workaround we wanted to increase volsize to 30000gb. Of course it is insane but if there is a setting for it I expect no hidden hardcoded limit. Also when I read the code it looks like unintentional. The check is against hardcoded 32gb but the error message shows the configured value Am 18.01.2017 17:51 schrieb "Chris Lu" <notifications@github.com>:

…

What's your intended volume size, btw? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#441 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUQZ1b9ql5dO9A7ymKeGhTSSQ3N7pP9ks5rTkMegaJpZM4LmtFU> .

[brstgt]

I see. Is this well documented? If 32gb is max the server should not allow more as startup parameter Am 18.01.2017 17:50 schrieb "Chris Lu" <notifications@github.com>:

…

For more than 32GB, it is more than fixing the limit. Current offset can only address to 32GB. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#441 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUQZ9plxexZ13Y1xvsLEby1rGIeL5rqks5rTkLGgaJpZM4LmtFU> .

[brstgt]

Is there a chance that this bug corrupted data or index file and if so is there a job that can be run to fix this? Am 18.01.2017 17:40 schrieb "Chris Lu" <notifications@github.com>:

…

Closed #441 <#441> via 59022b6 <59022b6> . — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#441 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUQZ7vFGcpCqiQaldFyXbr2iFMjvO1uks5rTkCRgaJpZM4LmtFU> .

[chrislusf]

I think it happens only during volume server startup and wrong volume size is counted. If wrong, the write to the volumes are failed. So the dat and idx files are still correct.

[brstgt]

I see. Unfortunately we had the 32gb removed for some time until I found the root cause. Could that have corrupted the DAT/idx files? Is there sth like a fsck command to check integrity of a file? Am 18.01.2017 18:09 schrieb "Chris Lu" <notifications@github.com>:

…

I think it happens only during volume server startup and wrong volume size is counted. If wrong, the write to the volumes are failed. So the dat and idx files are still correct. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#441 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUQZ5gEA6rMZYUjb37iX3HQDUEn4-YFks5rTkdJgaJpZM4LmtFU> .

[chrislusf]

Just check the actual .dat file size. But if the size is over 32GB, it will need non trivial amount of code to fix the .dat file.

[brstgt]

Hm there are actually some dat files that exceeded that size:

br@dev1:~$ sudo dsh -Mcg seaweed find /weedfs/ -type f -size +32G -exec 'ls -lh {} ;'
seaweed4: -rw-r--r-- 1 root root 35G Jan 18 18:04 /weedfs/d73825be-406a-4050-a36d-ad98ad23292d/mwps_624.cpd
seaweed4: -rw-r--r-- 1 root root 392G Jan 18 18:04 /weedfs/f4bc475a-4da6-4a89-9064-5e7f9b3164fd/mwps_622.dat
seaweed3: -rw-r--r-- 1 root root 58G Jan 18 14:05 /weedfs/6e50ee2d-b9aa-47e7-9020-1b1deb651713/mwpxs_65.dat
seaweed4: -rw-r--r-- 1 root root 72G Jan 18 18:04 /weedfs/0f9c1bb9-b4d2-4482-8a2e-80c46ee59915/gwpcl_402.dat
seaweed4: -rw-r--r-- 1 root root 68G Jan 18 14:05 /weedfs/efaacb2b-10f4-40f4-b817-414766685d9e/mwpcs_297.dat
seaweed1: -rw-r--r-- 1 root root 40G Jan 18 13:50 /weedfs/dda17747-c987-4fae-ba26-8377d34e8348/gwpcm_499.dat
seaweed4: -rw-r--r-- 1 root root 75G Jan 18 14:23 /weedfs/efaacb2b-10f4-40f4-b817-414766685d9e/mwpcm_600.cpd
seaweed3: -rw-r--r-- 1 root root 44G Jan 18 18:04 /weedfs/dba9c1e6-11c7-4af2-ac73-7d5ec0ad158d/mpm_390.dat
seaweed1: -rw-r--r-- 1 root root 83G Jan 18 18:04 /weedfs/dda17747-c987-4fae-ba26-8377d34e8348/gwpm_695.dat
seaweed4: -rw-r--r-- 1 root root 103G Jan 18 18:04 /weedfs/86823a03-e3c6-46c6-92a6-b569bbc856f5/gwpm_697.dat
seaweed3: -rw-r--r-- 1 root root 72G Jan 18 18:04 /weedfs/dba9c1e6-11c7-4af2-ac73-7d5ec0ad158d/gwpcl_402.dat
seaweed2: -rw-r--r-- 1 root root 34G Jan 18 11:49 /weedfs/399b17c1-9d57-430c-8fe3-73f10a488e38/mwpxs_68.dat
seaweed3: -rw-r--r-- 1 root root 68G Jan 18 18:04 /weedfs/33d28675-f4f8-45ee-b03c-36e670cd0b11/mwpcs_297.dat
seaweed3: -rw-r--r-- 1 root root 34G Jan 18 11:49 /weedfs/65a45a2e-3bfd-4377-8ff7-28bc25304f54/mwpxs_68.dat

What would you recommend?