Google Cloud Datastore is a great NoSQL solution (hosted, scalable, free up to a point), but it can be tricky (i.e. there's lots of code glue needed) to get even the "Hello World" of data persistence up and running in PHP.
This library is intended to make it easier for you to get started with and to use Datastore in your applications.
composer require "tomwalder/php-gds:^6.1"// Build a new entity
$obj_book = new GDS\Entity();
$obj_book->title = 'Romeo and Juliet';
$obj_book->author = 'William Shakespeare';
$obj_book->isbn = '1840224339';
// Write it to Datastore
$obj_store = new GDS\Store('Book');
$obj_store->upsert($obj_book);
// Fetch all books
foreach($obj_store->fetchAll() as $obj_book) {
echo "Title: {$obj_book->title}, ISBN: {$obj_book->isbn} <br />", PHP_EOL;
}Support for automated exponential backoff for some types of errors. See documentation here: https://cloud.google.com/datastore/docs/concepts/errors
To enable:
\GDS\Gateway::exponentialBackoff(true);Version 6.0.0 introduced better (but different) support for NULL values.
As of version 5 (May 2021), this library provides support for
- PHP 7 second-generation App Engine runtimes - using the REST API by default
- PHP 7 "anywhere" (e.g. Google Compute Engine, Cloud Run, GKE) - using REST or gRPC
Key features removed from version 5 onwards
- PHP 5 support
- Support for the legacy "Protocol Buffer" API built into first-generation App Engine runtimes
If you need to continue running applications on that infrastructure, stick to version 4.x or earlier.
- Examples
- Getting Started including installation with Composer and setup for GDS Emulator
- Defining Your Model
- Creating Records
- Timezones and DateTime
- Geopoint Support
- Queries, GQL & The Default Query
- Multi-tenant Applications & Data Namespaces
- Entity Groups, Hierarchy & Ancestors
- Transactions
- Data Migrations
- More About Google Cloud Datastore
- Unit Tests
- Footnotes
As of PHP-GDS version 5, the REST API Gateway is the default.
It will attempt to auto-detect your Google Project ID - and usually the Google auth library will use the default application credentials.
You might need to set an environment variable with the path to your JSON credentials file first, usually if you're running outside of App Engine or Google Compute Engine.
putenv('GOOGLE_APPLICATION_CREDENTIALS=/path/to/my/credentials.json');
// A regular Store, but with a custom Gateway
$obj_book_store = new GDS\Store('Book', new \GDS\Gateway\RESTv1('my-project-id'));You can find out more about the auth system here: Google Auth Library for PHP
You can download a service account JSON file from the Google Cloud Console API Manager > Credentials.
If you are using PHP-GDS version 4 or earlier, and Firestore in Datastore mode from App Engine standard (first generation), you may run into Internal Errors from the Protobuf gateway.
You can resolve this by using the RESTv1 Gateway & APIs. See here for basic guidance
More details and recommended upgrade paths to come. Along with better gRPC support for outside App Engine.
I find examples a great way to decide if I want to even try out a library, so here's a couple for you.
// Build a new entity
$obj_book = new GDS\Entity();
$obj_book->title = 'Romeo and Juliet';
$obj_book->author = 'William Shakespeare';
$obj_book->isbn = '1840224339';
// Write it to Datastore
$obj_store = new GDS\Store('Book');
$obj_store->upsert($obj_book);You can also use the Alternative Array Syntax for creating Entity objects, like this
$obj_book = $obj_store->createEntity([
'title' => 'The Merchant of Venice',
'author' => 'William Shakespeare',
'isbn' => '1840224312'
]);Now let's fetch all the Books from the Datastore and display their titles and ISBN numbers
$obj_store = new GDS\Store('Book');
foreach($obj_store->fetchAll() as $obj_book) {
echo "Title: {$obj_book->title}, ISBN: {$obj_book->isbn} <br />", PHP_EOL;
}These initial examples assume you are either running a Google AppEngine application or in a local AppEngine dev environment. In both of these cases, we can auto detect the dataset.
We use a GDS\Store to read and write GDS\Entity objects to and from Datastore.
These examples use the generic GDS\Entity class with a dynamic Schema. See Defining Your Model below for more details on custom Schemas and indexed fields.
A simple guest book application
Application: http://php-gds-demo.appspot.com/
Code: https://github.com/tomwalder/php-gds-demo
- Add PHP 7 support
- Remove PHP 5 support
- Remove App Engine first-generation runtime support (inc direct Protocol Buffer API)
In 5.1, timezone support has been improved for DateTime objects going in & out of Datastore.
Datastore keeps the data recorded as UTC. When you browse data in the Google Cloud Console, they represent it in your locale.
You can now expect any DateTime object coming out of Datastore from PHP-GDS to have your current PHP default timezone applied. Example follows:
date_default_timezone_set('America/New_York');
$obj_store = new GDS\Store('Book');
$obj_book = $obj_store->fetchOne();
echo $obj_book->published->format('c'); // 2004-02-12T15:19:21-05:00
echo $obj_book->published->getTimezone()->getName(); // America/New_YorkIf you pass in a DateTime object (or anything matching DateTimeInterface), we will respect the timezone set on it.
Any other string-based value passed in for a datetime field will be converted to a DateTimeImmutable object before being converted to UTC, using the standard PHP methods:
https://www.php.net/manual/en/datetime.construct.php
This means that unless using a timestamp value (e.g. @946684800), or a value with a timezone already stated (e.g. 2010-01-28T15:00:00+02:00), we will assume the value is in your current timezone context.
- More consistent use of
DateTimeobjects - now all result sets will use them instead ofY-m-d H:i:sstrings - Move the
google/authto an optional dependency - if you need the REST API
- Support for the new Datastore API, v1 - via REST
- Removal of support for the old 1.x series "PHP Google API Client"
- GeoPoint data is now supported over the REST API v1 as well as ProtoBuf
Are you sitting comfortably? Before we begin, you will need:
- a Google Account (doh), usually for running AppEngine - but not always
- a Project to work on with the "Google Cloud Datastore API" turned ON Google Developer Console
If you want to use the JSON API from remote or non-App Engine environments, you will also need
- Application default credentials OR
- a "Service account" and the JSON service key file, downloadable from the Developer Console
To install using Composer
composer require "tomwalder/php-gds:^5.1"Local development is supported using the REST Gateway and the Datastore Emulator.
Detailed instructions can be found here: https://cloud.google.com/datastore/docs/tools/datastore-emulator
Because Datastore is schemaless, the library also supports fields/properties that are not explicitly defined. But it often makes a lot of sense to define your Entity Schema up front.
Here is how we might build the Schema for our examples, with a Datastore Entity Kind of "Book" and 3 fields.
$obj_schema = (new GDS\Schema('Book'))
->addString('title')
->addString('author')
->addString('isbn');
// The Store accepts a Schema object or Kind name as its first parameter
$obj_book_store = new GDS\Store($obj_schema);By default, all fields are indexed. An indexed field can be used in a WHERE clause. You can explicitly configure a field to be not indexed by passing in FALSE as the second parameter to addString().
If you use a dynamic schema (i.e. do not define on, but just use the Entity name) then all fields will be indexed for that record.
Available Schema configuration methods:
GDS\Schema::addStringGDS\Schema::addIntegerGDS\Schema::addDatetimeGDS\Schema::addFloatGDS\Schema::addBooleanGDS\Schema::addStringListGDS\Schema::addGeopoint
Take a look at the examples folder for a fully operational set of code.
There is an alternative to directly constructing a new GDS\Entity and setting its member data, which is to use the GDS\Store::createEntity factory method as follows.
$obj_book = $obj_book_store->createEntity([
'title' => 'The Merchant of Venice',
'author' => 'William Shakespeare',
'isbn' => '1840224312'
]);Other than scalar values, there are two "object" data types supported:
Support for DateTime object binding (also see query parameter binding below)
$obj_book = $obj_book_store->createEntity([
'title' => 'Some Book',
'author' => 'A N Other Guy',
'isbn' => '1840224313',
'published' => new DateTime('-5 years')
]);The library has recently had support added for Geopoint properties.
$obj_schema->addGeopoint('location');Then when setting data, use the Geopoint object
$obj_person->location = new GDS\Property\Geopoint(53.4723272, -2.2936314);And when pulling geopoint data out of a result:
echo $obj_person->location->getLatitude();
echo $obj_person->location->getLongitude();It is not currently possible to query Geopoint fields, although this feature is in Alpha with Google
The GDS\Store object uses Datastore GQL as its query language. Here is an example:
$obj_book_store->fetchOne("SELECT * FROM Book WHERE isbn = '1853260304'");And with support for named parameter binding (strings, integers) (this is recommended)
$obj_book_store->fetchOne("SELECT * FROM Book WHERE isbn = @isbnNumber", [
'isbnNumber' => '1853260304'
]);Support for DateTime object binding
$obj_book_store->fetchOne("SELECT * FROM Task WHERE date_date < @now", [
'now' => new DateTime()
]);We provide a couple of helper methods for some common (root Entity) queries, single and batch (much more efficient than many individual fetch calls):
GDS\Store::fetchByIdGDS\Store::fetchByIds- batch fetchingGDS\Store::fetchByNameGDS\Store::fetchByNames- batch fetching
When you instantiate a store object, like BookStore in our example, it comes pre-loaded with a default GQL query of the following form (this is "The Default Query")
SELECT * FROM <Kind> ORDER BY __key__ ASCWhich means you can quickly and easily get one or many records without needing to write any GQL, like this:
$obj_store->fetchOne(); // Gets the first book
$obj_store->fetchAll(); // Gets all books
$obj_store->fetchPage(10); // Gets the first 10 booksBy default, this library will include a 1,000 record "batch size".
This means calling fetchAll() will only return 1,000 records.
I suggest paging your results if you need more than 1,000 records using fetchPage().
At the time of writing, the Google App Engine local development server does not support GQL. So, as of 2.0 I have included a basic GQL parser, which is only used in local development environments and should mean you can run most application scenarios locally as you can on live.
The GQL parser should be considered a "for fun" tool, rather than a production-ready service.
Feedback very much appreciated - if you have GQL queries that fail to run, just raise an issue and I'll see what I can do (or fork & PR!).
When working with larger data sets, it can be useful to page through results in smaller batches. Here's an example paging through all Books in 50's.
$obj_book_store->query('SELECT * FROM Book');
while($arr_page = $obj_book_store->fetchPage(50)) {
echo "Page contains ", count($arr_page), " records", PHP_EOL;
}In a standard SQL environment, the above pagination would look something like this:
SELECT * FROM Book LIMIT 0, 50for the first pageSELECT * FROM Book LIMIT 50, 50for the second, and so on.
Although you can use a very similar syntax with Datastore GQL, it can be unnecessarily costly. This is because each row scanned when running a query is charged for. So, doing the equivalent of LIMIT 5000, 50 will count as 5,050 reads - not just the 50 we actually get back.
This is all fixed by using Cursors. The implementation is all encapsulated within the GDS\Gateway class so you don't need to worry about it.
Bototm line: the bult-in pagination uses Cursors whenever possible for fastest & cheapest results.
Do not supply a LIMIT clause when calling
GDS\Store::fetchOne- it's done for you (we addLIMIT 1)GDS\Store::fetchPage- again, it's done for you and it will cause a conflict.
Google Datastore supports segregating data within a single "Dataset" using something called Namespaces.
Generally, this is intended for multi-tenant applications where each customer would have separate data, even within the same "Kind".
This library supports namespaces, and they are be configured per Gateway instance by passing in the optional 3rd namespace parameter.
ALL operations carried out through a Gateway with a namespace configured are done in the context of that namespace. The namespace is automatically applied to Keys when doing upsert/delete/fetch-by-key and to Requests when running GQL queries.
// Create a store for a particular customer or 'application namespace'
$obj_gateway = new \GDS\Gateway\RESTv1('project-id', 'namespace');
$obj_store = new \GDS\Store('Book', $obj_gateway);Further examples are included in the examples folder.
Google Datastore allows for (and encourages) Entities to be organised in a hierarchy.
The hierarchy allows for some amount of "relational" data. e.g. a ForumThread entity might have one more more ForumPosts entities as children.
Entity groups are quite an advanced topic, but can positively affect your application in a number of areas including
- Transactional integrity
- Strongly consistent data
At the time of writing, I support working with entity groups through the following methods
GDS\Entity::setAncestryGDS\Entity::getAncestryGDS\Store::fetchEntityGroup
The GDS\Store supports running updates and deletes in transactions.
To start a transaction
$obj_store->beginTransaction();Then, any operation that changes data will commit and consume the transaction. So an immediate call to another operation WILL NOT BE TRANSACTIONAL.
// Data changed within a transaction
$obj_store->upsert($obj_entity);
// Not transactional
$obj_store->delete($obj_entity);Watch out for GDS\Exception\Contention exceptions - they should be thrown by the library if you manage to hit Datastore contention locally in development or through the live Gateways.
Whilst you can use the GDS\Entity and GDS\Store classes directly, as per the examples above, you may find it useful to extend one or the other.
For example
class Book extends GDS\Entity { /* ... */ }
$obj_store->setEntityClass('\\Book');This way, when you pull objects out of Datastore, they are objects of your defined Entity class.
The Schema holds the custom entity class name - this can be set directly, or via the Store object.
When you change a field from non-indexed to indexed you will need to "re-index" all your existing entities before they will be returned in queries run against that index by Datastore. This is due to the way Google update their BigTable indexes.
I've included a simple example (paginated) re-index script in the examples folder, reindex.php.
Using multiple Gateway classes, you can move data between namespaces
// Name-spaced Gateways
$obj_gateway_one = new \GDS\Gateway\RESTv1('project-id', 'namespace_one');
$obj_gateway_two = new \GDS\Gateway\RESTv1('project-id', 'namespace_two');
// Grab some books from one
$arr_books = (new \GDS\Store('Book', $obj_gateway_one))->fetchPage(20);
// And insert to two
(new \GDS\Store('Book', $obj_gateway_two))->upsert($arr_books);and between local and live environments.
// Local and Remote Gateways
$obj_gateway_local = new \GDS\Gateway\ProtoBuf();
$obj_gateway_remote = new \GDS\Gateway\RESTv1('project-name');
// Grab some books from local
$arr_books = (new \GDS\Store('Book', $obj_gateway_local))->fetchPage(20);
// And insert to remote
(new \GDS\Store('Book', $obj_gateway_remote))->upsert($arr_books);Note: In both of these examples, the entities will be inserted with the same KeyID or KeyName
What Google says:
"Use a managed, NoSQL, schemaless database for storing non-relational data. Cloud Datastore automatically scales as you need it and supports transactions as well as robust, SQL-like queries."
https://cloud.google.com/datastore/
A few highlighted topics you might want to read up on
- Entities, Data Types etc.
- More information on GQL
- GQL Reference
- Indexes
- Ancestors
- More about Datastore Transactions
A full suite of unit tests is in the works. Assuming you've installed php-gds and its dependencies with Composer, you can run
vendor/bin/phpunitOr, if you need to run containerised tests, you can use the runphp image (or any you choose)
docker run --rm -it -v`pwd`:/app -w /app fluentthinking/runphp:7.4.33-v0.9.0 php /app/vendor/bin/phpunitI am certainly more familiar with SQL and relational data models so I think that may end up coming across in the code - rightly so or not!
Thanks to @sjlangley for any and all input - especially around unit tests for Protocol Buffers.
Whilst I am using this library in production, it is my hope that other people find it of use. Feedback appreciated.
If you've enjoyed this, you might be interested in my Full Text Search Library for PHP on Google App Engine