Add database abstraction to avoid hard dependency

[Quix0r]

I would love to see some database-abstraction being added so PeerTube can not just run with MongoDB but with any database (even SQLite).

[Chocobozzz]

It would be interesting but the maintenance would require big efforts. I don't think it is very useful for now.

[ldidry]

As you say "All pods have an index of all videos of the network", I think MongoDB is a terrible choice.

• the DB files don't shrink when records are removed (you can shrink them, but it needs stopping Peertube)
• more videos means more time to start the database. At Framasoft, we used MongoDB for our Etherpad instances and some took 10 minutes to start because of MongoDB (now, with PostgreSQL, it takes one minute max)

It may be a good thing to consider PostgreSQL: it can do NoSQL too, and better than MongoDB. Have a look at http://www.aptuz.com/blog/is-postgres-nosql-database-better-than-mongodb/ 😄

[Chocobozzz]

Thanks @ldidry to share your experience. I'll study this more deeply :)

[faddat]

If we are sharing-- I love redis?

Well, redis, or we just scrap node.js for the back end altogether and do it in go. There are webtorrent goodies for go, so might be able to implement in go using storm/boltdb. In the end this means you distribute a single executable that includes the database, and it will go fast.

[ldidry]

Well, having the database outside the executable allows you to run it on a different server, which is a good thing when you're beginning to scale. One server for backend (and videos for our case), with a large disk space, and a server with fast disks for the database is a pretty common thing.

My 2 cts.

[Chocobozzz]

I'm still benchmarking different DBMS but it seems that PostgreSQL offers better performances than MongoDB or MySQL (with 1 000 000 videos, 3 000 000 tags). And this is a relational DBMS which will simplify things in the future (for playlists etc).
Moreover there are very interesting feature like the Trigram module for the video search for example.

I agree @ldidry and I don't really stay with a NoSQL database. I chose MongoDB because I thought SQL DBMS were not effective enough with big tables but I was wrong (with PostgreSQL at least). And NoSQL databases will make things difficult for PeerTube.

I need to make some other tests but I think I'll switch to PostgreSQL soon. Of course, I'm open to discussions :)

[faddat]

What all does the back end need to do in total? Golang+storm is quite a way to build a whatever.

…

On Dec 9, 2016 6:54 PM, "Bigard Florian" ***@***.***> wrote: I'm still benchmarking different DBMS but it seems that PostgreSQL offers better performances than MongoDB or MySQL (with 1 000 000 videos, 3 000 000 tags). And this is a relational DBMS which will simplify things in the future (for playlists etc). Moreover there are very interesting feature like the Trigram module <https://www.postgresql.org/docs/current/static/pgtrgm.html> for the video search for example. I agree @ldidry <https://github.com/ldidry> and I don't really stay with a NoSQL database. I chose MongoDB because I thought SQL DBMS were not effective enough with big tables but I was wrong (with PostgreSQL at least). And NoSQL databases will make things difficult for PeerTube. I need to make some other tests but I think I'll switch to PostgreSQL soon. Of course, I'm open to discussions :) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#9 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGz6iRK8K439sGPgIeAB5VmV5W9w7XIcks5rGUGFgaJpZM4J6jNn> .

[Chocobozzz]

PostgreSQL is now on master/develop branches. I keep this issue opened since PeerTube only support PostgreSQL for now.

The ORM is sequelize which supports PostgreSQL, MySQL, SQLite and MSSQL so we will be able to add other DBMS in the future.

[Quix0r]

Looks promising. :-) And nice work.

[Chocobozzz]

After some consideration, I don't think it's a good idea to support other databases than PostgreSQL. It requires a lot of effort and complicates maintenance. We can see such issues in Diaspora that supports PostgreSQL and MySQL (performance problems with some queries between these DMBS for example).

[ldidry]

And it adds a problem: if server A has Pg and is a big instance but server B has SQLite or MySQL, the server B will have poor performances.

So, avoiding other databases will improve performances consistency.