New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide Postgresql support for Piwik #500

Open
anonymous-piwik-user opened this Issue Jan 13, 2009 · 112 comments

Comments

Projects
None yet
@anonymous-piwik-user

anonymous-piwik-user commented Jan 13, 2009

I want to be able to use a postgres database (instead of mysql).

please see http://piwik.org/faq/how-to-install/#faq_55

Latest update: see the [Piwik fork with Postgresql support](https://github.com/sri-soham/piwik). Great contribution by Sridhar from the community
Keywords: wishlist

@robocoder

This comment has been minimized.

Contributor

robocoder commented Jan 13, 2009

See ticket #425.

@mattab

This comment has been minimized.

Member

mattab commented Feb 2, 2009

The piwik team decided to not support Postgresql as it would make the development process much harder for us, for a tiny number of users requesting Postgresql.

Piwik officially supports MySQL

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented Feb 28, 2009

Let’s go here : http://github.com/klando/pgpiwik/tree/master

If you are interesting, then :

git clone git://github.com/klando/pgpiwik.git

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented Mar 2, 2009

too bad. I would have loved to see this happen. I would say that people that would use postgres would rival that of mysql.

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented Mar 5, 2009

Do I interpret right that you expect to have more tickets opened to request postgres? :-) I usually don’t even bother to check a mysql-only project, let alone open a bug about it in their project, so your assumption about the size of the postgresql userbase may be extremely skewed.

Apart from letting you know that many people use db other than mysql it is a bit more important to note that even php makes it possible to write db backend independent code, which could support nearly any database backends, including, but not limited to psql and sqlite. I do not believe it would visibly make development harder to use standard SQL and standard DB API, since at a first blick your code is pretty much mysql independent (only a few places may require tuning) and you seem to use backend independent code (at least you include it in the distribution). So it may only required a few changes here and there. Pity I do not like php at all, so I cannot offer to patch it up, and you’d rejct anyway from an outsider. ;-)

So, please consider to use database backend independent code and API. It would be nice and The Right Thing®™.

@robocoder

This comment has been minimized.

Contributor

robocoder commented Mar 6, 2009

Re-closing. This is a resource issue; not a techical one.

We need people to step up to:
- implement the changes to core
- provide ongoing support to end users
- test releases for regressions
- maintain the backend as development continues

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented Mar 6, 2009

Replying to [vipsoft](comment:7):

> Re-closing. This is a resource issue; not a techical one.
>
> We need people to step up to:
> – implement the changes to core
> – provide ongoing support to end users
> – test releases for regressions
> – maintain the backend as development continues
- I have implement change to core,
- I provide ongoing support via mail and via irc freenode #piwik (well, it is irc, ask, wait, wait,wait, answer)
- I didn’t have test yet, but I saw that your are in the pocess of Zendify so, it will be easy.
- I merge with the svn as soon as I am aware of svnchanges (despite the fact that the ML for svncommits is down, so I have to check from time to time)

@mattab

This comment has been minimized.

Member

mattab commented Mar 11, 2009

A good ressource to read by wordpress team: http://codex.wordpress.org/Using_Alternative_Databases

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented Aug 3, 2009

Well, since Piwik uses ZF for database handling, it wouldn't be that hard, "just" editing the SQL statements to be compatible with other DBMS. I see further problems in plugin development, there will be coder who make their plugins only MySQL compatible.
I don't know how many people here would like to see their Piwik with postgres, but I think it can be done if we work together.

@robocoder

This comment has been minimized.

Contributor

robocoder commented Sep 11, 2009

Blockers:

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented May 6, 2011

I'm keeping MySQL on my VPS just because Piwik requires it. I decided to start support for PostgreSQL in Piwik, the plugin approach seems the right first try. When it stabilizes in my production VPS, I'll report here.

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented Nov 9, 2011

I strongly ask for Postgres support as MySQL has various bugs and critical problems (like security, speed, scaling).

Supporting any database is very easy. Everybody knows that plain SQL queries within sourcecode are a big risks, because some people know how to write good queries, others don't. Injection and speed issues are unavoidable.

There must be something like http://search.cpan.org/~abraxxa/DBIx-Class-0.08195/lib/DBIx/Class.pm for PHP. Switching shouldn't take that much time, if you still use plain queries in sourcecode, and will greatly improve security and portability.

PS: MySQL is already on it's way to become payware and I doubt that many users would pay for it just to use Piwik :-)

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented Jan 24, 2012

I'm interested to use Piwik with Postgresql too !

Right now, the only reason why i'm still using Google Analytics is because Piwik don't support pgsql...

Is there any chance to use piwik with alternatives databases soon ?

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented Jun 2, 2012

Replying to grin:

I usually don't even bother to check a mysql-only project, let alone open a bug about it in their project, so your assumption about the size of the postgresql userbase may be extremely skewed.

+1 for Postgres support
(I registered just to reduce the skewing effect)

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented Dec 23, 2012

Another +1 for PostgreSQL support. Piwik is a really nice piece of software, but I definitely won't install another DBMS so I can use it.

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented Dec 23, 2012

Another +1 for a non mysql version of Piwik here! PostgreSQL would be great! Also other great software like Drupal is supporting postgresql nowadays.

@mattab

This comment has been minimized.

Member

mattab commented Jan 2, 2013

like we said we are keen to do it if someone submits a patch. There is clearly a lot of interest out there but not yet someone who has the skills and time to invest in such a huge coding spree!

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented Jan 25, 2013

Reopening, as it appears that the devs are now welcoming patches for this request.

Also +1'ing, because I refuse to deal with MySQL and Oracle anymore. I could switch to Maria DB, but Postgres seems to have all the momentum.

Would love this feature. Without it, I'm going to go looking for another solution.

@reedstrm

This comment has been minimized.

reedstrm commented Mar 27, 2013

+1 here as well. I've looked at Piwik several times, and mysql has been the deal breaker each time.

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented Jun 10, 2013

+1 here as well. I'm using Piwik for a volunteer organization, and we are sensitive to the privacy issues posed by for instance Google analytics.

We're using Piwik with mysql locally right now, but all other resources are running against a postgres cluster, with all the bells and whistles, most importantly proper data backup and protection routines. It would be great to get rid of the lone mysql service running locally.

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented Dec 18, 2013

+1. We have a small VPS that we use for hosting our site (Apache+Postgresql), and it's working quite nicely, but having to add MySQL just for Piwik uses up almost all memory, even if we configure MySQL to use as little as possible.

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented Jan 24, 2014

+1

MySQL is a big joke. (It never makes me laugh though.)

Unfortunately MariaDB has to have all the MySQL strangeness in order to work as a drop-in replacement. This, however, makes it just as irrelevant as an alternative.

I want to get rid of that installation for the same reasons already mentioned by others:

@Jon-IB

This comment has been minimized.

Jon-IB commented Jan 24, 2014

I'm interested in taking on the work of adding support to PostgreSQL.

The actual amount of SQL that needs to be changed doesn't seem too dramatic. I'm more concerned with the following:

  1. There isn't much of a db abstraction layer, as far as I can tell.
  2. Plugin developers would need to support both MySQL and PostgreSQL. Or alternatively, each plugin would need meta-data to indicate which database(s) it supports.
  3. As noted by others, testing and CI would need to cover both databases.

Most importantly (for me), does anyone on the core team have the time to map out an approach for a database abstraction layer? Once that's done, I think we can handle the pgsql-specific changes.

Also, any thoughts on the other concerns above?

EDIT: Looks like there's some active discussion on this in the mailing list. I'll follow up there.

@mattab

This comment has been minimized.

Member

mattab commented Jan 27, 2014

Thanks for showing your interest in this topic! While we are not actively engaged in this work (yet) many team members are interested in running other DBs with Piwik. the issue is, that it's super complex.

Most importantly (for me), does anyone on the core team have the time to map out an approach for a database abstraction layer? Once that's done, I think we can handle the pgsql-specific changes.

We could map out an approach, but honestly, unless we do the full work, do you think anyone would do it?

If anyone is really interested, yes we could work for a few days / weeks on the basics of DB abstraction layer. But converting the whole codebase to use such layer, is rather huge and complicated task.

If you know any software engineer/developer with 3-4 weeks available and a big brain willing to explode, and a big interest in this work, please let us know here!

@Jon-IB

This comment has been minimized.

Jon-IB commented Jan 28, 2014

(Apologies, I think I misspoke -- it looks like Piwik has a db abstraction layer, in the form of the Db class and the Zend_Db_Adapter. What's missing is a data-access layer, since all of the SQL code is intermingled with the business logic.)

Matt, I can spare a developer for 3-4 weeks, but I doubt we'll have time to do much more than that right now. So I'm hesitant to dive in, given that we might realize after 2 or 3 weeks that we'd need to spend a lot more time on this. If that happened, I would need to find an alternative solution quickly.

One creative solution is to do this incrementally, by updating the code file-by-file to use a new data access layer. This could only work if some of the code could be updated to use a new data access layer, but the rest of the code was left as-is. I'm not sure how exactly to do this, but I'm guessing the two approaches could co-exist, since most or all queries seem to go through the Db class.

This would allow the core team to implement and test an approach that you're happy with, since you'd only have to implement a data-access tier for a few files. That would prove that it works, and your code would give us a template to follow. You wouldn't even need a PostgreSQL database to test those changes; just set up two different MySQL logins with access to the same schema.

Once you're done with the proof of concept, we would come in and move the SQL into a new data-access tier for as much of the code as possible, following your template. Our initial focus would be all of the SQL that is specific to MySQL, since the app would be usable for both PostgreSQL and MySQL if we get through all of that code and leave the rest of the files as-is. But we'd try to get through all of the SQL while we had focus.

I can offer at least 3 person-weeks of work, and hopefully a bit more, which might be enough to get through all of the code. Other developers could help with this as well, if any are available. And any remaining files could be handled by the community or by us as time permits.

I'd be more comfortable diving in if we followed this approach, because we'd be more likely to have a production-quality solution within 3-4 weeks that we could deploy.

The big downside is that the code might end up in a Frankenstein-like state for some period of time, with some code calling to a data-access tier, and other code with inline SQL. You might not consider that to be "production-quality".

Having said that, a data-access tier might help structure the code more cleanly. If so, then the incremental approach would let you migrate the code towards that structure without having to take on all of the work within a single release.

@Jon-IB

This comment has been minimized.

Jon-IB commented Jan 28, 2014

Ah, I see a few classes under the Piwik\DataAccess namespace. Is this the start of a full data-access tier? Or is this a solution for a much smaller problem?

@anonymous-piwik-user

This comment has been minimized.

anonymous-piwik-user commented Jan 28, 2014

Please see https://github.com/sri-soham/piwik . This is an (almost) working version of Piwik with PostgreSQL suppport.

@Jon-IB

This comment has been minimized.

Jon-IB commented Jan 29, 2014

Thanks, @low. I saw a link to that repo in another ticket, so our developers have been looking closely at it for the past few days. I myself looked too quickly; I saw a change to a file that commingled a MySQL and PostgreSQL error code (core/DataAccess/ArchiveTableCreator.php maybe?), so I quickly assumed that there was no db abstraction layer in this repo. I assumed incorrectly that @sri-soham was working initially on a proof of concept, without changing the code structure.

One of my developers mentioned that this repo uses DAOs to encapsulate both data access and database abstraction, so I looked more closely. The DAO implementation looks pretty darn solid to me. The pgsql classes extending the MySQL classes to take advantage of common logic. It would be straightforward to add support for additional DBMSes, if that was desirable.

Matt et al, have you been following this work? What do you think of this implementation? If the remaining issues are resolved, is it reasonable to assume that this code or something like it will make it into the upstream repo in the foreseeable future?

@mattab

This comment has been minimized.

Member

mattab commented Feb 1, 2014

Increasing priority to reflect the very high interest of the community in supporting other databases! Please keep commenting.

I am a bit swamped at the moment, but I'll do my best to review the code and give a feedback here in the next week or so. Thanks for your patience and trying to make this happen!

@mattab

This comment has been minimized.

Member

mattab commented Feb 6, 2014

hey Jon! I have posted a message to Sridhar on the piwik-hackers mailing list: http://lists.piwik.org/pipermail/piwik-hackers/2014-February/001452.html

I updated the description of this ticket with: see the Piwik fork with Postgresql support. Great contribution by Sridhar from the community

@sridhar if you read this, maybe you could update the port to sync with master? I am looking at this diff in particular: sri-soham/piwik@piwik:master...master

If you confirm this is the right place to look, I will discuss the fork with the Piwik team and will come back to you here about our feedback!

@somenet

This comment has been minimized.

somenet commented Aug 18, 2016

Its 2016 ... and THIS is this still an issue?
DBMS-Agnostic features like PDO have been there for like ages and no sane person uses mysql (or maria) anymore.
Its either fully relational (oracle, mssql, pgsql) or nosql nowadays.

+1 for PDO (or pgsql for that case)

@mattab

This comment has been minimized.

Member

mattab commented Aug 22, 2016

Hi everyone,

if you need Piwik to support Postgresql and your company or business can sponsor this improvement, please get in touch with the Piwik core team here: https://piwik.org/development/ - we would love to build Postgresql support in Piwik but we would need serious monetary or engineering support to make this project happen and maintain Postgresql going forward.

@law

This comment has been minimized.

law commented Oct 5, 2016

+1 for Postgresql from me, as well.

@eydunn

This comment has been minimized.

eydunn commented Oct 19, 2016

+1 for postgresql :-)

@fazalmajid

This comment has been minimized.

fazalmajid commented Oct 19, 2016

@mattab: how much money are we talking about? I doubt any one entity would front something like this, but it might be possible to crowdfund it.

@law

This comment has been minimized.

law commented Oct 19, 2016

I'd chip in for this.

@advn

This comment has been minimized.

advn commented Oct 31, 2016

@mattab It costs money to implement support for postrresql. We all get it. But how much money? 10k? 100k? More? Can anyone give a cost estimate?

@augustf

This comment has been minimized.

augustf commented Oct 31, 2016

@advn It's been awhile since I jumped into this crazy +1-happy thread, but my guess is money is only a minor part of the issue. To make this happen, the product would need to essentially be torn apart and re-architected to remove all of the MySQL-proprietary evilness. And that's not something they're willing to do for really any price at all.

@fazalmajid

This comment has been minimized.

fazalmajid commented Oct 31, 2016

@augustf that's not quite the impression I got from mattab's comment, which sounds guardedly open as long as there is some funding. Obviously rewriting, e.g. to use PDO, is a significant engineering effort and I would understand if the Piwik team closed this ticket with a WONTFIX status code.

@tsteur

This comment has been minimized.

Member

tsteur commented Nov 1, 2016

I would say it is about funding, having the resources and priority. It's not only making Piwik itself compatible because it even affects the Piwik eco system with plugins etc. It makes everything a bit more complicated as plugin developers suddenly need to mark whether their plugin is compatible with Postgres or MySQL. Users need to filter plugins by database backend etc. It's quite hard to estimate something like this without having a proper look which would take a lot of time. Just to mention a rough number because you asked I would roughly estimate like a mid five digit number but as said, it's really hard to say something about it.

@mattab mattab modified the milestones: Mid term, Long term Dec 5, 2016

@karo-github

This comment has been minimized.

karo-github commented Feb 2, 2017

Can some one explain why it is not enough to rewrite (or write another class) like Piwik/Db, Piwik/DbHelper, \Piwik\Db\Schema\Pgsql and Piwik\Db\Adapter\Pdo\Pgsql to have piwik integrated with PostgreSQL?

@sgiehl

This comment has been minimized.

Member

sgiehl commented Feb 2, 2017

SQL queries written for MySQL might not work on PostgeSQL

@karo-github

This comment has been minimized.

karo-github commented Feb 2, 2017

But why is it necessary to use native sql queries? We can havean DataAccess layer that encapsulates any sql logic in itself and provides an API for executing any db query without using sql queries at all.
I mean something like ORM in Java

@sgiehl

This comment has been minimized.

Member

sgiehl commented Feb 2, 2017

Most of our queries are optimized to run faster on MySQL. That isn't really possible when using ORMs that create the queries dynamicaly. And as speed is something that matters quite much in that kind of application, it's imho not a good option to loose it only to support multiple databases.

@karo-github

This comment has been minimized.

karo-github commented Feb 2, 2017

I agree with you that " not a good option to loose it only to support multiple databases", but it would be great to have one or several classes , where all these queries are accumulated , but not hardcode queries in code . It really make a big problem for me as programmer to integrate piwik with another DB.

@smscotten

This comment has been minimized.

smscotten commented Feb 2, 2017

Of course, performance at scale is why many of us prefer PostgreSQL over MySQL.

My suggestion: drop MySQL support entirely and make everyone happier.

Yes, I'm kidding. If MySQL optimization is your specialty it will take you a little while to get PostgreSQL queries to purr like you want.

The part I'm not kidding about is: we're all on the side of better performance.

@kvordf

This comment has been minimized.

kvordf commented May 3, 2017

Hello everyone,
Where can I get the latest PIWIK/PostgreSQL version for testing.

Thanks
Khanh

@sgiehl

This comment has been minimized.

Member

sgiehl commented May 3, 2017

There is no official PostgreSQL version

@kvordf

This comment has been minimized.

kvordf commented May 3, 2017

I understand that there is no official version for PostgreSQL. I thought that there was a beta or some version out there. Is this it? https://github.com/klando/piwik ?

@RuslanAyupov

This comment has been minimized.

RuslanAyupov commented Mar 12, 2018

+1 for postgresql support

@wenerme

This comment has been minimized.

wenerme commented Mar 27, 2018

Matomo now, the first thing that I hope to happen is support PG, kind of frustrated. Is there any possibility that this may happen, for 9ys, we all know PG is good, so what?

@clement-igonet

This comment has been minimized.

clement-igonet commented Apr 8, 2018

My (big) company support PS but not MySQL, because of policy and ressources constraints.
So, Matomo would be welcome if PS supported, but currently excluded until the day it will supports PS...

@siva538

This comment has been minimized.

siva538 commented Aug 22, 2018

+1 PostgreSQL

@fazalmajid

This comment has been minimized.

fazalmajid commented Sep 17, 2018

Never mind, i switched to Fathom
https://usefathom.com

Much more basic, but I like the UI better and it’s written in Go, not PHP/

@clement-igonet

This comment has been minimized.

clement-igonet commented Sep 18, 2018

@rasebo

This comment has been minimized.

rasebo commented Sep 18, 2018

Hey, fathom looks nice, it's worth a try if one only needs basic visitor tracking (which I do). Thanks, I'll look into it and consider switching to it if it means I'll be able to remove the overhead generated by having both mysql and postgres on my infrastructure.

@ofaurax

This comment has been minimized.

ofaurax commented Oct 26, 2018

Can the travis-ci tests be run on an instance with postgresql?
Using the unit tests could help to ensure that a postgresql branch is functional.

@Jon-IB

This comment has been minimized.

Jon-IB commented Nov 13, 2018

I'm running the old 2.0.b4 fork for Postgres on about 15k low-traffic sites. Thanks to the EOL on PHP 5.6, I need to migrate these sites to Matomo 3.x on MySQL. Postgres is running fine with about 550 million rows in the log_link_action table. But I don't have time to port the Postgres fork to the 3.x branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment