Add GeoIp extension

[cloudlena]

What version of Cloud Foundry are you using?
246

What version of the buildpack you are using?
4.3.23

If you were attempting to accomplish a task, what was it you were attempting to do?
Push Piwik using this tutorial: https://berndsgn.ch/run-piwik-on-cloud-foundry

What did you expect to happen?
I was attempting to install the geoip extension of PHP which plays a major role in Piwik. Here's my .bp-config/options.json for that:

{
  "PHP_EXTENSIONS": ["bz2", "zlib", "curl", "mcrypt", "gd", "cli", "geoip", "pdo", "pdo_mysql", "mbstring", "openssl"],
  "PHP_VERSION": "{PHP_70_LATEST}"
}

What was the actual behavior?

During staging, I get the following message:

The extension 'geoip' is not provided by this buildpack.

Please confirm where necessary:

• I have included a log output
• My log includes an error message
• I have included steps for reproduction

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/135465045

The labels on this github issue will be updated when the story is started.

[RochesterinNYC]

Hi @mastertinner, the extension seems like a useful PHP extension.

• Is this the correct pecl package for the geoip extension: https://pecl.php.net/package/geoip ?
• Additionally, looking at the discussions on the installations page for the geoip extension, there seems to be mention of a geoip2 that has superseded the geoip extension. Is this a PHP extension and should it be used in lieu of the geoip extension? Any clarifications/insight here would be helpful.

[cloudlena]

Thanks for investigating, @RochesterinNYC! The extension you mentioned is in fact the correct one. I am not sure about GeoIP2 mentioned on the site you posted. It sounds like a good idea but I have no clue whether it's backwards compatible and if Piwik would work with it. Also, I'm not sure if it can still be installed as an extension because it's not listed here: https://secure.php.net/manual/en/extensions.alphabetical.php

The Piwik page where you can set the method of how geolocations should be determined says the following about using GeoIP from PECL:

This location provider uses a GeoIP database and a PECL module to accurately and efficiently determine the location of your visitors.

There are no limitations with this provider, so it is the one we recommend using.

How do I install the GeoIP PECL extension?

So it looks like they're recommending the one you suggested.

[dmikusa]

It seems like while the extension is free and open source, it requires a database that is not.

https://dev.maxmind.com/geoip/geoipupdate/
https://piwik.org/faq/how-to/faq_163/

Not sure that should block us from pulling in the extension, but something to consider since the extension doesn't really do anything without the DB.

Perhaps there is some way that we could auto download and install the DB as well. Maybe if a user binds a user provided service with his or her maxmind creds, then we could configure geoipupdate (linked above) to download and keep the db up-to-date (or something along those lines).

Another option might be to have the user bind a service (or set an env variable) with the path to the database. Then we could just configure php.ini to point to the correct location for the database.

[athornton2012]

I agree with just pulling in the extension. Ascertaining the DB could be done via the .profile script potentially ? (i.e. downloads the appropriate setup/install scripts, installs the DB, sets up or starts whatever necessary binaries). Though, a DB solely inside the app container does not have any kind of persistence at the moment.

The part about auto downloading and installing the DB sounds out of the scope of the buildpack.

The last option sounds a bit more in line with what we do with other buildpacks/services like NewRelic.

[dmikusa]

Ascertaining the DB could be done via the .profile script potentially ? (i.e. downloads the appropriate setup/install scripts, installs the DB, sets up or starts whatever necessary binaries).

This does not fit with the general philosophy of build packs, which is that things should "just work". We should try to make this work without requiring the user to code or do anything beyond binding a service or setting a specific env variable.

Though, a DB solely inside the app container does not have any kind of persistence at the moment.

Persistence in the container doesn't really matter here. The DB is read-only and just needs to exist so that the geoip extension can look up stuff. The only changes would be if the DB is updated and if this happened, the entire DB would be replaced with a new one.

I think if we set the build pack to pull a DB at staging time that's probably sufficient. Users can then restage the app to pull in the latest copy of the DB.

The part about auto downloading and installing the DB sounds out of the scope of the buildpack.
The last option sounds a bit more in line with what we do with other buildpacks/services like NewRelic.

My $0.02. Add a PHP build pack extension that looks for the presence of a variable, say GEOIP_DB_URL. If it exists, the extension would download the pointed to URL and then proceed to configure the geoip extension. This should support both file:// and http:// URLs.

It looks like you can get a direct download URL for the database from MaxMind, so if your CF is on the public Internet you could point directly to MaxMind. If you're not on the public Internet, you could point to a file uploaded with the app or provide a URL to an internal location with the database.

Should be easy enough to implement and easy for users to enable. Similar to NewRelic PHP build pack extension, where it looks for the presence of a license key.

@mastertinner - Thoughts? How are you obtaining and configuring the DB in other environments?

[sesmith177]

@mastertinner is there further information you could give us regarding how you plan to use the GeoIp extension? Specifically in regard to @dmikusa-pivotal's comments about the DB

[sesmith177]

@mastertinner thoughts on above comments?

[dgodd]

@mastertinner thoughts on above comments?

[cloudlena]

Sorry that I took so long for me to answer.

I would like to use it so that Piwik can look up the geolocations of the IP addresses of visitors of my sites. I agree that persistence is not an issue because the database needs to be read only.

My $0.02. Add a PHP build pack extension that looks for the presence of a variable, say GEOIP_DB_URL. If it exists, the extension would download the pointed to URL and then proceed to configure the geoip extension. This should support both file:// and http:// URLs.

I think that's a great idea. Furthermore, we could set the variable to use the official download URL by default so that most users won't have to configure the variable. In that way, we could keep the mentioned buildpack philosophy about them "just working". The user would just have to enable the extension in .bp-config/options.json and (as long as they're connected to the internet), it would work. If they are not connected to the internet, they would additionally have to declare the GEOIP_DB_URL to specify an alternate URL.

I'm not a PHP developer at all so I haven't used this database anywhere else yet...

[dmikusa]

Just an fyi, I have some changes to binary-builder which I think will suffice for building the geoip library and optionally packaging the "lite" version of the databases, which are available under a creative commons license (default is not to though). I'll try to share this soon.

[dmikusa]

I've submitted a binary-builder PR that adds support for geoip & the dbs.

cloudfoundry/binary-builder#24

I have a PR in mind for the build pack too, but first step is to get the extension built and included.

[cloudlena]

Awesome! Thanks so much, @dmikusa-pivotal!

[sclevine]

This extension should be available in the latest version of the buildpack.

[cloudlena]

Nice! Thx, @sclevine!

[cloudlena]

@sclevine, I included it when deploying Piwik and the database is there. However, when I look at Piwik's health check, it tells me GeoIP (PECL): Broken. The reason behind it appears to be the following:

Error: The PECL module is looking for databases in '/usr/share/GeoIP/', but this directory does not exist. Please create it and add the GeoIP databases to it. Alternatively, you can set 'geoip.custom_directory' to the correct directory in your php.ini file.
Note: The 'geoip.custom_directory' PHP ini option is set to ''.

This GeoIP implementation has access to the following types of databases: .

Should we add the database to the folder /usr/share/GeoIP/ which appears to be the default location? Or can you tell me where the database is located in our case?

[sclevine]

@dmikusa-pivotal thoughts?

[dmikusa]

I think you can make this work with a .user.ini as geoip.custom_directory can be set in any config file. It just needs to point to the directory where the geoip databases are installed. By default, the build pack will put them in /home/vcap/app/php/geoipdb/dbs.

Ex:

geoip.custom_directory=/home/vcap/app/php/geoipdb/dbs

The other point to note is that we do not bundle the databases with the build pack. You'll either need to upload those with your app or run the script that we bundle to download them at runtime.

Ex: /home/vcap/app/php/geoipdb/bin/download_geoip_db.rb -h

USAGE: download_geoip_db [options]
    -u, --user=USER                  User Id from MaxMind.  Default "999999".
    -l, --license=LICENSE            License from MaxMind.  Default "000000000000".
    -o, --output_dir=OUTPUTDIR       Directory where databases might exist and will be written / updated.  Default "."
    -p, --products=PRODUCTS          Space separated list of product ids.  Default "GeoLite-Legacy-IPv6-City GeoLite-Legacy-IPv6-Country 506 517 533".

Unless you have a license file, you probably want to run /home/vcap/app/php/geoipdb/bin/download_geoip_db.rb -o /home/vcap/app/php/geoipdb/dbs.

You can run this by setting -c '$HOME/php/geoipdb/bin/download_geoip_db.rb -o $HOME/app/php/geoipdb/dbs && $HOME/.bp/bin/start' (I would use $HOME so that you're not hard coding the app path) for cf push or adding it in your manifest.yml.

@sclevine - I want to update the build pack so that it does this automatically. I just haven't gotten around to putting together a patch. I think the basic idea is that when the geoip extension has been enabled by the user, the build pack would set that php.ini option and somehow add the databases.

I haven't through it through all the way though and there are concerns:

• how will it work for offline users, they won't be able to run the script so there needs to be a way for them to supply the database with the app
• is it better to download staging or runtime or both? staging can cache the files, plus runtime takes away from the app startup time or could cause timeouts, but runtime will keep them more up-to-date. Both might be a good option as the script will only download when there are changes server side, so an initial download during staging and a follow up download to update at runtime might be the best of both worlds

[dmikusa]

I submitted this PR which should make using geoip easier. #204