Is LizardFS development still alive?

[hradec]

The last commit on github was JUNE/2008! It's being 7 months without ANY bug fixes or new development. Even the main source code depot (http://cr.skytechnology.pl:8081/#/q/status:open), is being stale since Oct/2018. (the only update in November was an update to the UPGRADE text file)

Is this project still being developed at all?

Due to all the frustrations I had in the last 3 years with LizardFS, and the lack of development in the past 7 months, I start testing the newer version of MooseFS... and HUAAAU!! just HUAU!

MooseFS seems miles ahead of LizardFS today, waaay faster read/write (same hardware as LizardFS), and way better speeds for reading small files, which finally makes it possible to use as home folders in our studio.

NO TIMEOUTS!! transfer of files (read/write) just works!! Same hardware, same network as LizardFS, there are NO "timeout" messages or "unknown error" in the logs of any of the machines, chunkserver or master/shadow... You start writing a 300GB file, and the files transfer CONSISTENTLY at 60mb/sec from start to end! LizardFS just halts, drop down to 5mb/sec, jumps to 10-20... but NEVER I was able to write 300GB at more than 20mb/sec with LizardFS... MooseFS gives 60MB/sec OUT OF THE BOX... no hassle, no setup, no adjustments in sysctl... nothing!

The WebUI is so much better, with chunkserver "maintenance mode" switch, a detailed description of storage classes (their approach to file goal) which informs if a class can be fulfilled or not before hand, based on the chunkserver setup you have... not to mention that the class storage system is just AMAZING!!!

Unfortunately, the open source version of MooseFS doesn't have EC(D,P) support, so I can't comment on that performance... But we are seriously considering not only switch to MooseFS, but actually pay for the "Pro" version to have access to EC(D,P), since we're loosing faith on LizardFS at this point.

Apart from the low read/write performance with LizardFS, we're also finding a lot of wasted disk space happening where chunks over chunks are just "forgot" on disks. The Web UI says everything is fine, no undergoal/overgoal files... nothing to be deleted!
By removing a chunk, formatting it's drives and adding the chunk back (essentially forcing LizardFS to re-replicate that chunk), we ended up with 25% more free disk space on that chunk after replication finishes!! 25%!!!!!

We're basically doing that on ALL our chunks now... kind of a forced "scrub", and we're just seeing more and more free space showing up! (and also a few missing chunks showing up for ec(2,1) and 2 goals, when LFS was saying everything as "safe"... not good!!!)

So, we're actually seeing an extra 15TB free disk space starting to appear out of the 60TB storage we have on LizardFS... just by removing/adding chunks.

Because all of that, for the past 2 years we have being using LizardFS JUST for backup storage. It's IMPOSSIBLE to use it in production at the studio, and now I also can't trust on it for backup since I have to explain the boss that we lost a few ec(2,1)/2 goal files that got silently corrupted by LFS.

And NO public development on LizardFS since June!

Not to mention the infamous "lto tape server" support which I even tried contact your company to known more about, and got NO information about whatsoever. Funny thing, the alleged tape support was about 40% the reason we bet on LizardFS over MooseFS 2 year ago... so sad!

anyhow... I would really like to hear from you guys about the future of LizardFS... if there's one at all!
MooseFS is way ahead of you guys, no question about it! So there's a lot of "catch-up" you guys need to do before even thinking about new and better features.

---

PS: sorry for rumbling... but it's being really frustrating experience so far with LFS... the whole "timeout" thing between servers in LFS is just ridiculous... and more ridiculous the fact you guys keep blaming on hardware or network instead of acknowledge the issue (just read the answers to this issue here in github... it's all documented).

What a surprised I had with MooseFS when I couldn't find ANY timeout messages or performance drops whatsoever, no matter what you do USING THE SAME HARDWARE/NETWORK as LFS!!! Consistent transfer from start to end, all the time, with multiple clients... It's actually more consistent than NFS! (still 25% slower than NFS, but more consistent speeds for sure).

That's more than prove that the problem is not on our side (hardware/network) but in fact with LFS code itself. Sorry to say!

[onlyjob]

IMHO bug fixing dynamics in the last few years also show how unhealthy this project has become. Even if development eventually resumes I doubt they'll pick up the pace and will be able to compete... LizardFS team should issue a public statement about status of the project. It makes me sad to say that my prognosis about the future of LizardFS is far from positive...

[matthiaz]

I agree with the comments made about the lack of activity.

But I do want to point out that the loss of storage space might be caused by HDD_PUNCH_HOLES on the chunkservers. As you know, lizardfs increases the size of the chunks to the nearest 64KB to have them in a consistent size. But... if you have HDD_PUNCH_HOLES = 0, those 0's will actually take space. So make sure you have HDD_PUNCH_HOLES = 1 on each chunkserver.

I too was burned by EC in lizardfs, and recommend against using ec encoding.

[hradec]

I do have HDD_PUNCH_HOLES = 1 since day one! It's not that!

once you take a hard drive out, re-format (or replace it by a new one to be on the safe side), you'll see that LFS WILL use LESS space to replicate the missing drive than it actually was on said drive!

For me, it's being around 25% increase in free space for each drive!

[blink69]

Hi all,
We would like to inform you that we are still here :), of course there has been a noticeable silence from our side, unfortunately this was unavoidable. But now we want share with you some details and our future plan.

To move LizardFS forward we decided to not actively develop 3.x branch and go to LizardFS 5.x which is totally redesigned and rewritten from scratch. Because version 5 will be totally incompatible with version 3 and thinking about you guys (our community) we will also publish LizardFS 4.x version (somewhere around Q3 and Q4 of 2019) which will be a direct bridge between 3.x and 5.x this version will be the only way to upgrade your current cluster.

In coming days we will delegate a person to work on resolving all the current issues that appear here on github, this developer will work with you to help LizardFS get back to the strong position it held before we had to neglect it.

Thank you for sticking with us!

[jkiebzak]

Q4 of 2019 is almost a year away.

…

On Tue, Feb 26, 2019 at 7:58 AM Adam Ochmański ***@***.***> wrote: Hi all, We would like to inform you that we are still here :), of course there has been a noticeable silence from our side, unfortunately this was unavoidable. But now we want share with you some details and our future plan. To move LizardFS forward we decided to not actively develop 3.x branch and go to LizardFS 5.x which is totally redesigned and rewritten from scratch. Because version 5 will be totally incompatible with version 3 and thinking about you guys (our community) we will also publish LizardFS 4.x version (somewhere around Q3 and Q4 of 2019) which will be a direct bridge between 3.x and 5.x this version will be the only way to upgrade your current cluster. In coming days we will delegate a person to work on resolving all the current issues that appear here on github, this developer will work with you to help LizardFS get back to the strong position it held before we had to neglect it. Thank you for sticking with us! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#805 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEynix3ef6EJF3Fo5OqnsHRjIAlaikS8ks5vRS9qgaJpZM4aj0-y> .

[onlyjob]

In other words let us all forget about hope for bug fixing... What a slap to the face of the community. :(
Who would trust you, LizardFS team, to maintain anything with such attitude?

I think I will be long gone by the time you will be ready to deliver anything.

This project will not survive without community and leadership.
Community is almost lost and leadership is weak.

Before promising new great releases you have to repair the trust first and show your ability to maintain what's already there, especially after fiasco with 3.13.0~rc1.

[blink69]

True, we will do our best to repair that we broke here.

[onlyjob]

By breaking continuity? By giving up on existing code base and writing v5 from scratch?
How many years are you going so spend stabilising v5 before it becomes usable?

Secret development behind closed doors will not repair trust. Only transparency can but this project have no transparency any more. :(