New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let users import Google Analytics data in Piwik #6095

Open
mattab opened this Issue Aug 31, 2014 · 51 comments

Comments

Projects
None yet
@mattab
Member

mattab commented Aug 31, 2014

The goal of this ticket is to create an easy to use tool that will let anyone import in Piwik their Google Analytics data.

Features expected in this tool

  • Import one or several websites
  • Import data for all periods, or only import for a custom date range
  • For each website, we import the report data: Top pages, top events, top browsers, etc. (for the 50 reports available in Piwik or so)
  • Note: will not import the raw log visitor data since it is not available via Google Analytics API
  • Create a simple user interface, which uses a console command

Tasks

  • Proposed name: Google Analytics Importer
  • Create new project at piwik/importer-google-analytics
  • This is a Piwik plugin published on the Marketplace
  • Let user connect via oAuth to the Google APIs, fetch automatically the profiles, websites available.
  • Run import via the UI or directly let user execute the console command
  • Test the importer on a few websites,
    • including a website with medium traffic to check that we work around Google API limits
  • Release the tool to public:
    • update this blog post, this FAQ
    • write a new blog post about the tool

This issue requires #6094 (Create an API to let users import historical report data in Piwik)

@mcollopy

This comment has been minimized.

Show comment
Hide comment
@mcollopy

mcollopy Jan 12, 2015

Will importing history from GA cause duplication in Piwik? e.g. For a site which has been using GA for a year, but only 3 months in Piwik, we might want to import the previous 9 months to get 1 year of history.
But not the overlapping recent 3 months.
Do you plan on deduping or providing a data range filter?
Thanks

Will importing history from GA cause duplication in Piwik? e.g. For a site which has been using GA for a year, but only 3 months in Piwik, we might want to import the previous 9 months to get 1 year of history.
But not the overlapping recent 3 months.
Do you plan on deduping or providing a data range filter?
Thanks

@liebana

This comment has been minimized.

Show comment
Hide comment
@liebana

liebana Mar 11, 2015

What's the overall status of this? It seems that there's no activity at all in both issues. Do you have this in your short term roadmap?

liebana commented Mar 11, 2015

What's the overall status of this? It seems that there's no activity at all in both issues. Do you have this in your short term roadmap?

@mattab

This comment has been minimized.

Show comment
Hide comment
@mattab

mattab Mar 11, 2015

Member

There is no update and while it is currently in the Short term it may not be done in short term due to the complexity of the task. If you need this and have a budget, please contact Piwik PRO who may be able to help!

Member

mattab commented Mar 11, 2015

There is no update and while it is currently in the Short term it may not be done in short term due to the complexity of the task. If you need this and have a budget, please contact Piwik PRO who may be able to help!

@dbrgn

This comment has been minimized.

Show comment
Hide comment
@dbrgn

dbrgn Mar 25, 2015

This feature would be great!

dbrgn commented Mar 25, 2015

This feature would be great!

@mattab mattab modified the milestones: Short term, Mid term Apr 7, 2015

@erotavlas85

This comment has been minimized.

Show comment
Hide comment
@erotavlas85

erotavlas85 Apr 10, 2015

This is great feature!

This is great feature!

@JagerBomb

This comment has been minimized.

Show comment
Hide comment
@JagerBomb

JagerBomb Apr 20, 2015

We are really looking forward to this feature in our company. We cannot ask our clients to use our central Piwik installation until this has been released, since they have been relying on Google Analytics for too long and wish to keep their historical data.

We are really looking forward to this feature in our company. We cannot ask our clients to use our central Piwik installation until this has been released, since they have been relying on Google Analytics for too long and wish to keep their historical data.

@baptistedonaux

This comment has been minimized.

Show comment
Hide comment
@jdreesen

This comment has been minimized.

Show comment
Hide comment

+1

@paul-cross

This comment has been minimized.

Show comment
Hide comment

+1

@Maestro2k5

This comment has been minimized.

Show comment
Hide comment

+1

@phpdave11

This comment has been minimized.

Show comment
Hide comment
@phpdave11

phpdave11 Jul 30, 2015

What about importing raw google analytics data? I have a copy of every hit saved, can these be imported into piwik?

What about importing raw google analytics data? I have a copy of every hit saved, can these be imported into piwik?

@nickwright80

This comment has been minimized.

Show comment
Hide comment

+1

@berniezhao

This comment has been minimized.

Show comment
Hide comment

+1

@Tomcraft1980

This comment has been minimized.

Show comment
Hide comment

+1

@aaronDeveloper

This comment has been minimized.

Show comment
Hide comment
@VelNZ

This comment has been minimized.

Show comment
Hide comment

VelNZ commented Sep 2, 2015

+1

@gelldur

This comment has been minimized.

Show comment
Hide comment

gelldur commented Sep 8, 2015

+1

@movak

This comment has been minimized.

Show comment
Hide comment

movak commented Sep 9, 2015

+1

@ghost

This comment has been minimized.

Show comment
Hide comment

ghost commented Sep 18, 2015

+1

@MatthieuScarset

This comment has been minimized.

Show comment
Hide comment
@MatthieuScarset

MatthieuScarset Oct 20, 2015

I'm in! What task need to be done first? :)

I'm in! What task need to be done first? :)

@L0garithmic

This comment has been minimized.

Show comment
Hide comment
@L0garithmic

L0garithmic Nov 18, 2015

Come on guys, we need this!!!

Come on guys, we need this!!!

@unknown23person

This comment has been minimized.

Show comment
Hide comment
@mgonera

This comment has been minimized.

Show comment
Hide comment

mgonera commented Feb 17, 2016

+1

@Reamer

This comment has been minimized.

Show comment
Hide comment

Reamer commented Apr 11, 2016

+1

@anacona16

This comment has been minimized.

Show comment
Hide comment
@anacona16

anacona16 May 11, 2016

Hi, can you tell me the status for this issue?

Hi, can you tell me the status for this issue?

@sgiehl

This comment has been minimized.

Show comment
Hide comment
@sgiehl

sgiehl May 11, 2016

Member

Not sure if anyone is working on that at all 😞

Member

sgiehl commented May 11, 2016

Not sure if anyone is working on that at all 😞

@mattab mattab modified the milestones: Mid term, Long term May 27, 2016

@Draky50110

This comment has been minimized.

Show comment
Hide comment

+1

@mattab

This comment has been minimized.

Show comment
Hide comment
@mattab

mattab Sep 27, 2016

Member

We won't be able to work on this, unless someone can sponsor us to work on it. Please contact us via https://piwik.org/development/ if you're able to sponsor the team!

Member

mattab commented Sep 27, 2016

We won't be able to work on this, unless someone can sponsor us to work on it. Please contact us via https://piwik.org/development/ if you're able to sponsor the team!

@erotavlas85

This comment has been minimized.

Show comment
Hide comment
@erotavlas85

erotavlas85 Sep 27, 2016

Why do not we/you create a crowdfounding campaign? I think it would be of great success...

Why do not we/you create a crowdfounding campaign? I think it would be of great success...

@dbrgn

This comment has been minimized.

Show comment
Hide comment
@dbrgn

dbrgn Sep 27, 2016

I'd certainly give a few dollars :)

dbrgn commented Sep 27, 2016

I'd certainly give a few dollars :)

@josephernest

This comment has been minimized.

Show comment
Hide comment
@josephernest

josephernest Oct 23, 2016

+10 (+10$ at least in a crowdfunding)

Possible already-available solution:

Is there a tool that allows us to export from Google Analytics to Apache-log similar access.log files?

Then it would be easy to re-import to Piwik, with import_logs.py...

What do you think?

+10 (+10$ at least in a crowdfunding)

Possible already-available solution:

Is there a tool that allows us to export from Google Analytics to Apache-log similar access.log files?

Then it would be easy to re-import to Piwik, with import_logs.py...

What do you think?

@rosa2

This comment has been minimized.

Show comment
Hide comment

rosa2 commented Nov 27, 2016

+1

@rosa2

This comment has been minimized.

Show comment
Hide comment
@rosa2

rosa2 Nov 27, 2016

@mattab how much money do you need to make the programming? I am very interested to get it.

rosa2 commented Nov 27, 2016

@mattab how much money do you need to make the programming? I am very interested to get it.

@rosa2 rosa2 referenced this issue Nov 27, 2016

Closed

Piwik #774

@cisarpavel

This comment has been minimized.

Show comment
Hide comment
@cisarpavel

cisarpavel Dec 21, 2016

Do you have some estimate price/hours to develop this feature?

Do you have some estimate price/hours to develop this feature?

@zody0210

This comment has been minimized.

Show comment
Hide comment
@zody0210

zody0210 Dec 25, 2016

+1
I am absolutely willing to pay for such a plugin/script.

zody0210 commented Dec 25, 2016

+1
I am absolutely willing to pay for such a plugin/script.

@ryanhamilton

This comment has been minimized.

Show comment
Hide comment
@ryanhamilton

ryanhamilton Dec 31, 2016

+1 would pay.

+1 would pay.

@viezel

This comment has been minimized.

Show comment
Hide comment
@viezel

viezel Jan 13, 2017

+1 I would pay. @mattab can you create a kickstarter or similar?

viezel commented Jan 13, 2017

+1 I would pay. @mattab can you create a kickstarter or similar?

@shaun-of-the-dead

This comment has been minimized.

Show comment
Hide comment
@shaun-of-the-dead

shaun-of-the-dead Mar 5, 2017

Same for me. I'd be happy to contribute financially to help this feature / tool occur!

I have years of Google Analytics data logged before I learnt about Piwik, and I'm sure I'm not the only one. There'd be great value in implementing a tool like this if you guys can pull it off.

Same for me. I'd be happy to contribute financially to help this feature / tool occur!

I have years of Google Analytics data logged before I learnt about Piwik, and I'm sure I'm not the only one. There'd be great value in implementing a tool like this if you guys can pull it off.

@machampagne

This comment has been minimized.

Show comment
Hide comment
@machampagne

machampagne Mar 28, 2017

We are also willing to chip in for this feature.

We are also willing to chip in for this feature.

@DJLunacy

This comment has been minimized.

Show comment
Hide comment

Bueller?

@nopynoop

This comment has been minimized.

Show comment
Hide comment

+1

@erotavlas85

This comment has been minimized.

Show comment
Hide comment
@erotavlas85

erotavlas85 Jul 21, 2017

We could use patreon.
Take a look at this or this.

We could use patreon.
Take a look at this or this.

@SzymonDukla

This comment has been minimized.

Show comment
Hide comment
@SzymonDukla

SzymonDukla Sep 26, 2017

I would like to offer a development of such functionalities (that includes the API and the plugin to import GA data). A monthly financial contribution is crucial. Do you want me to set up a Patreon (or similar)?
In case of questions, please ask on sd@szymondukla.co.uk

I would like to offer a development of such functionalities (that includes the API and the plugin to import GA data). A monthly financial contribution is crucial. Do you want me to set up a Patreon (or similar)?
In case of questions, please ask on sd@szymondukla.co.uk

@mattab

This comment has been minimized.

Show comment
Hide comment
@mattab

mattab Sep 26, 2017

Member

We'd also like to work on this, but so far the problem is how would it technically work? @SzymonDukla do you have some thoughts on this? we haven't found a GA API that we can use to import all the data safely and accurately yet...

Member

mattab commented Sep 26, 2017

We'd also like to work on this, but so far the problem is how would it technically work? @SzymonDukla do you have some thoughts on this? we haven't found a GA API that we can use to import all the data safely and accurately yet...

@SzymonDukla

This comment has been minimized.

Show comment
Hide comment
@SzymonDukla

SzymonDukla Sep 27, 2017

I have a project behind me now that involves Analytics data gathering through GA API. It's nearly (as close as we can really get) to raw data. I have a nice and flexible API wrapper written in PHP I'm going to use.
I've set up a Patreon: https://www.patreon.com/ga_for_piwik
I'll be answering all the questions on there.

I have a project behind me now that involves Analytics data gathering through GA API. It's nearly (as close as we can really get) to raw data. I have a nice and flexible API wrapper written in PHP I'm going to use.
I've set up a Patreon: https://www.patreon.com/ga_for_piwik
I'll be answering all the questions on there.

@mattab

This comment has been minimized.

Show comment
Hide comment
@mattab

mattab Sep 29, 2017

Member

Hi @SzymonDukla what's your idea of getting as close as you can get raw data? would you call the GA API with many dimensions to get raw data or so?

Member

mattab commented Sep 29, 2017

Hi @SzymonDukla what's your idea of getting as close as you can get raw data? would you call the GA API with many dimensions to get raw data or so?

@fabrice-regnier

This comment has been minimized.

Show comment
Hide comment
@fabrice-regnier

fabrice-regnier Oct 11, 2017

+1 i'd pay too. Please,please, help us to opt out of GA. I guess it would be a great functionnality of piwik!

I can see chats are going on on this subject, nice.

regards.
f.

+1 i'd pay too. Please,please, help us to opt out of GA. I guess it would be a great functionnality of piwik!

I can see chats are going on on this subject, nice.

regards.
f.

@SzymonDukla

This comment has been minimized.

Show comment
Hide comment
@SzymonDukla

SzymonDukla Oct 13, 2017

Sorry for the delay, last two weeks have been crazy...
Getting back on this project, that's correct @mattab - I have a side project that involves getting and storing all the possible data from GA accounts and then generating monthly/weekly/yearly SEO reports based on that so I think I will have the same approach which is getting as many metrics in as many configurations (as some metrics are now working with other ones) exported into a JSON or similar file that then will be imported via Piwik plugin, processed and merged into Piwik itself.

@fabrice-regnier, thank you very much for your interest! I do that because I have the same problem - years worth of data on Google I have no way to export.

As mentioned, I've set up a Patreon (https://www.patreon.com/ga_for_piwik), let's move all further discussion on this project there for those interested :)

Sorry for the delay, last two weeks have been crazy...
Getting back on this project, that's correct @mattab - I have a side project that involves getting and storing all the possible data from GA accounts and then generating monthly/weekly/yearly SEO reports based on that so I think I will have the same approach which is getting as many metrics in as many configurations (as some metrics are now working with other ones) exported into a JSON or similar file that then will be imported via Piwik plugin, processed and merged into Piwik itself.

@fabrice-regnier, thank you very much for your interest! I do that because I have the same problem - years worth of data on Google I have no way to export.

As mentioned, I've set up a Patreon (https://www.patreon.com/ga_for_piwik), let's move all further discussion on this project there for those interested :)

@mattab

This comment has been minimized.

Show comment
Hide comment
@mattab

mattab Oct 15, 2017

Member

will have the same approach which is getting as many metrics in as many configurations (as some metrics are now working with other ones) exported into a JSON or similar file that then will be imported via Piwik plugin, processed and merged into Piwik itself.

Did you mean 'metrics' or rather 'dimensions'? Our previous version of the GA importer (since deprecated) used this trick, the problem was that back in the days there were not enough precise dimensions, so we couldn't get useful "Per visitor" information and import all visitors.
Could you show us an example of how you'll export the GA data to then import it in Piwik?

Member

mattab commented Oct 15, 2017

will have the same approach which is getting as many metrics in as many configurations (as some metrics are now working with other ones) exported into a JSON or similar file that then will be imported via Piwik plugin, processed and merged into Piwik itself.

Did you mean 'metrics' or rather 'dimensions'? Our previous version of the GA importer (since deprecated) used this trick, the problem was that back in the days there were not enough precise dimensions, so we couldn't get useful "Per visitor" information and import all visitors.
Could you show us an example of how you'll export the GA data to then import it in Piwik?

@geekbrat

This comment has been minimized.

Show comment
Hide comment
@geekbrat

geekbrat Apr 15, 2018

@SzymonDukla

As mentioned, I've set up a Patreon (https://www.patreon.com/ga_for_piwik), let's move all further discussion on this project there for those interested :)

I was really interested in this and went to back your patreon and then I noticed that you are asking for over $2000/month to create the plugin, and then expect most of us to purchase it on top of that. I have a great appreciation for the amount of work involved and needing to be able to divert your attention from other projects, but I really prefer a game to have either free to play, pay to upgrade or pay to play, free to upgrade models of pricing, and REALLY hate when they stick it to me on both ends, which is exactly what this feels like.

@SzymonDukla

As mentioned, I've set up a Patreon (https://www.patreon.com/ga_for_piwik), let's move all further discussion on this project there for those interested :)

I was really interested in this and went to back your patreon and then I noticed that you are asking for over $2000/month to create the plugin, and then expect most of us to purchase it on top of that. I have a great appreciation for the amount of work involved and needing to be able to divert your attention from other projects, but I really prefer a game to have either free to play, pay to upgrade or pay to play, free to upgrade models of pricing, and REALLY hate when they stick it to me on both ends, which is exactly what this feels like.

@voarsh

This comment has been minimized.

Show comment
Hide comment
@voarsh

voarsh Aug 13, 2018

It looks like @SzymonDukla is no longer supporting this issue and the patreon is not running. Can someone update the issue, outline who is responsible for creating or outsourcing this piece of work?

voarsh commented Aug 13, 2018

It looks like @SzymonDukla is no longer supporting this issue and the patreon is not running. Can someone update the issue, outline who is responsible for creating or outsourcing this piece of work?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment