Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Firestore #1865

Closed
wants to merge 30 commits into from
Closed

Firestore #1865

wants to merge 30 commits into from

Conversation

blowmage
Copy link
Contributor

@blowmage blowmage commented Dec 4, 2017

This PR adds the Firestore Ruby client library.

require "google/cloud/firestore"

firestore = Google::Cloud::Firestore.new

# Get the cities collection
cities_col = firestore.col "cities"

# Get the document for NYC
nyc_ref = cities_col.doc "NYC"

# Set the name for NYC
firestore.set(nyc_ref, { "name": "New York City" })

The library also implements writing multiple changes in a batch:

firestore.batch do |b|
  # Set the data for NYC
  b.col("cities").doc("NYC").set({ name: "New York City" })

  # Update the population for SF
  b.col("cities").doc("SF").update({ population: 1000000 })

  # Delete LA
  b.col("cities").doc("LA").delete
end

It also implements support for transactions

cities_col = firestore.col("cities").doc("SF")
cities_col.set({ name: "San Francisco",
           state: "CA",
           country: "USA",
           capital: false,
           population: 860000 })

firestore.transaction do |tx|
  new_population = tx.get(cities_col).data[:population] + 1
  tx.update(city, { population: new_population })
end

And implements read-only transactions:

cities_col = firestore.col "cities"
nyc_ref = cities_col.doc "NYC"
sf_ref  = cities_col.doc "SF"
la_ref  = cities_col.doc "LA"

firestore.read_only_transaction do |rtx|
  # Get each city's population
  nyc_population = rtx.get(nyc_ref).data[:population]
  sf_population  = rtx.get(sf_ref).data[:population]
  ls_population  = rtx.get(la_ref).data[:population]
end

@blowmage blowmage added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Dec 4, 2017
@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Dec 4, 2017
@blowmage blowmage force-pushed the firestore branch 2 times, most recently from 0492a0e to cf4a6f5 Compare December 8, 2017 05:03
@coveralls
Copy link

Coverage Status

Coverage increased (+0.05%) to 93.677% when pulling cf4a6f57e7fbe9500d3e7a2ef001b3107bace96e on firestore into 8b6dc68 on master.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.05%) to 93.677% when pulling f38d92dd369e7252788cd339cb04ddc147a50f46 on firestore into 8b6dc68 on master.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.05%) to 93.677% when pulling fd67403f85012678a137c284f95b0b10cb28df28 on firestore into 8b6dc68 on master.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.07%) to 93.692% when pulling ca7b8574e99b0143e9079dfcc8e07631468151aa on firestore into 8b6dc68 on master.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.1%) to 93.692% when pulling 0b2960a on firestore into c26b404 on master.

@blowmage blowmage removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Dec 8, 2017
@blowmage blowmage changed the title WIP Firestore Firestore Dec 8, 2017
@blowmage
Copy link
Contributor Author

blowmage commented Dec 8, 2017

@sebastian-schmidt @frankyn Can you please take a look at this now?

@blowmage blowmage self-assigned this Dec 8, 2017

Gem::Specification.new do |gem|
gem.name = "google-cloud-firestore"
gem.version = "0.6.8"

This comment was marked as spam.

This comment was marked as spam.

{
"title": "V1beta1",
"type": "google/cloud/firestore/v1",
"patterns": ["\/firestore\/v1$", "\/firestore\/v1\/"],

This comment was marked as spam.

This comment was marked as spam.

@frankyn
Copy link
Member

frankyn commented Dec 11, 2017

Will Review today.

@frankyn
Copy link
Member

frankyn commented Dec 11, 2017

@blowmage I'm seeing a Gem::LoadError when I try to require the Firestore gem. This failure doesn't occur when I create a Gemfile and execute bundle install.

Gem::LoadError: google/rpc/status_pb found in multiple gems: googleapis-common-protos, googleapis-common-protos-types
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:102:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/grpc-1.7.3-x86_64-linux/src/ruby/lib/grpc/google_rpc_status_utils.rb:16:in `<top (required)>'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/google-gax-0.10.2/lib/google/gax/grpc.rb:31:in `<top (required)>'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/google-gax-0.10.2/lib/google/gax/errors.rb:32:in `<top (required)>'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/google-gax-0.10.2/lib/google/gax/api_callable.rb:32:in `<top (required)>'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/google-gax-0.10.2/lib/google/gax.rb:30:in `<top (required)>'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:in `require'
... 8 levels...
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/google-cloud-firestore-0.6.8/lib/google/cloud/firestore/collection.rb:17:in `<top (required)>'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/google-cloud-firestore-0.6.8/lib/google/cloud/firestore/database.rb:16:in `<top (required)>'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/google-cloud-firestore-0.6.8/lib/google/cloud/firestore/project.rb:17:in `<top (required)>'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/google-cloud-firestore-0.6.8/lib/google/cloud/firestore.rb:17:in `<top (required)>'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:133:in `require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:133:in `rescue in require'
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:40:in `require'
        from (irb):1
        from /usr/local/google/home/franknatividad/.rbenv/versions/2.3.1/bin/irb:11:in `<main>'

Steps to reproduce:
Install gem

gem install google-cloud-firestore-0.6.8.gem

Run IRB

require "google/cloud/firestore"

Ruby version: ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]

@frankyn
Copy link
Member

frankyn commented Dec 12, 2017

@blowmage I'm still reviewing, and should have comments tomorrow. I'm running through samples written for other langues presented on https://cloud.google.com/firestore/docs/ to review the client library. I do have an open question, where are Firestore samples/documentation for the client library similar to other clients?

Update to match the current gem dependencies.
Remove host, channel, and chan_creds as we are using the credentials object now.
The main object is going to be Database, so return it from the constructor.
At some point in the future users will be able to create databases.
Call list_collection_ids and create Collection objects.
Add Database#doc to create a new Document::Reference object.
@frankyn
Copy link
Member

frankyn commented Dec 13, 2017

@blowmage follow-up on the gem install question. I tried requiring the gem again and then I was able to use the gem as expected. I'm trying to understand the omg workflow. What is the omg command? Is it a Ruby specific tool?

There was an issue where an empty response with the transaction_id would be
treated as a Document result. Fix implementation and add test coverage.
@blowmage
Copy link
Contributor Author

@frankyn The omg command is ohmygems. It is functionally equivalent to rvm gemset and rbenv gemset. It creates an empty environment where you have no gems installed. I try to optimize my ruby response time, so I use a combination of chruby and ohmygems because it is faster than rvm, rbenv, or chruby/chgem.

the-more-you-know

The documentation was showing the wrong return type.
It returns Document::Snapshot objects, not Document::Refernce objects.
Update the code examples to show accessing the snapshot data.
@coveralls
Copy link

Coverage Status

Coverage increased (+0.09%) to 93.71% when pulling 803625b on firestore into 37dde6c on master.

@quartzmo
Copy link
Member

@frankyn fwiw, I just use rbenv gemset.

@quartzmo
Copy link
Member

@blowmage At which level (alpha, beta) is this library being released? (I noticed the generated overview.rb lists "Alpha". And do you want to add Firestore to the top-level README in this PR?

@blowmage
Copy link
Contributor Author

I honestly don’t know what level it will be released. I was expecting another PR doing a documentation pass and polish before it is released.

Copy link
Member

@quartzmo quartzmo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I didn't know (or forgot) that there would be a subsequent PR for docs. I'll just submit these comments now, I think they are all for documentation.

end
alias_method :document, :doc

def add data = nil

This comment was marked as spam.

This comment was marked as spam.

end

##
# The project resource the Cloud Firestore batch belongs to.

This comment was marked as spam.

This comment was marked as spam.

#
# All changes are accumulated in memory until the block passed to
# {Database#batch} completes. Unlike transactions, batches are not
# automatically retried. See {Database#batch}.

This comment was marked as spam.

This comment was marked as spam.

# path of the document, or a document reference object.
# @param [Hash] data The document's fields and values.
# @param [true, String|Symbol, Array<String|Symbol>] merge When provided
# and `true` all data is merged with the existing docuemnt data. When

This comment was marked as spam.

This comment was marked as spam.

# Write to document with the provided object values. If the document
# does not exist, it will be created. By default, the provided data
# overwrites existing data, but the provided data can be merged into the
# existing document using the `merge` argument.

This comment was marked as spam.

This comment was marked as spam.

##
# Create a document with the provided object values.
#
# The batch will fail if the document already exists.

This comment was marked as spam.

This comment was marked as spam.

@quartzmo
Copy link
Member

It seems potentially misleading for Batch to offer read methods such as get, select, etc:

        #   firestore.batch do |b|
        #     # Get a document reference
        #     nyc_ref = b.doc "cities/NYC"
        #
        #     # Create a document
        #     b.create(nyc_ref, { name: "New York City" })
        #   end

The semantics here are identical to that of Transaction, but in fact the reads are not part of the batch operation at all:

If you do not need to read any documents in your operation set, you can execute multiple write operations as a single batch that contains any combination of set(), update(), or delete() operations.

A batch can be understood as the complement of a read-only transaction. ReadOnlyTransaction does not contain write methods, however.

Make changes requested by the review process.
@blowmage
Copy link
Contributor Author

And do you want to add Firestore to the top-level README in this PR?

I do not. I think it would be confusing to add to the top-level README until there is a gem that users can install.

@blowmage
Copy link
Contributor Author

It seems potentially misleading for Batch to offer read methods such as get, select, etc

I understand why you say that, but the ruby library is doing something extra that I think is more in-line with Rubyist expectations. But yes, the requirements document does indicate that the write batch operations should be specific to write operations:

# Get a document reference
nyc_ref = firestore.doc "cities/NYC"

# Create a document in a batch
firestore.batch do |b|
  b.create(nyc_ref, { name: "New York City" })
end

This PR delivers that usage. However, I believe a more idiomatic approach is to make all writes that occur within a closure use the batch. Similar to the approach taken by ActiveRecord's transactions. To deliver something like this, objects created or retrieved within the batch will make writes on the batch as long as they are made within the closure:

# Functionally equivalent to the previous example
firestore.batch do |b|
  b.doc("cities/NYC").create({ name: "New York City" })
end

@coveralls
Copy link

Coverage Status

Coverage increased (+0.04%) to 93.665% when pulling 7086aec on firestore into 37dde6c on master.

@quartzmo
Copy link
Member

Similar to the approach taken by ActiveRecord's transactions.

It is the similarity to transactions that concerns me: Batches are not read-write transactions, but if the semantics are identical, with reads performed on the object yielded to the block, people unfamiliar with the difference may be misled. That's why I recommend removing the read methods from Batch and showing examples similar to your first one, above:

# Read existing documents
nyc_ref = firestore.doc "cities/NYC"
sf_ref = firestore.doc "cities/SF"

# Update multiple document in a batch rather than in a 
# transaction to avoid contention
firestore.batch do |b|
  b.update(nyc_ref, { name: "Big Apple" })
  b.update(sf_ref, { name: "San Fran" })
end

@blowmage
Copy link
Contributor Author

Your code comments are misleading. That code does not read the documents, it creates document reference objects. No reads are happening there.

@quartzmo
Copy link
Member

That's correct.

@@ -0,0 +1,230 @@
# Copyright 2017, Google Inc. All rights reserved.

This comment was marked as spam.

This comment was marked as spam.

@googleapis googleapis deleted a comment from blowmage Dec 18, 2017
Copy link
Member

@frankyn frankyn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had one open question about custom objects with te Ruby client library. It's not possible. Is this an expected state?

Reference: https://cloud.google.com/firestore/docs/manage-data/add-data#custom_objects

I also commented on the documentation and samples.

#
# firestore.batch do |b|
# # Create a query
# query = b.where(:population, :>=, 1000000).

This comment was marked as spam.

# @param [String, Symbol] direction The direction to order the results
# by. Optional. Default is ascending.
#
# @return [Query] A query with `order` called on it.

This comment was marked as spam.

This comment was marked as spam.

#
# firestore.batch do |b|
# # Create a document
# b.update("cities/NYC", { name: "New York City" },

This comment was marked as spam.

##
# Create a document with random document identifier.
#
# The batch will fail if the document already exists.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

# Write to document with the provided object values. If the document
# does not exist, it will be created. By default, the provided data
# overwrites existing data, but the provided data can be merged into
# the existing document using the `merge` argument.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

#
# All changes are accumulated in memory until the block passed to
# {Database#batch} completes. Unlike transactions, batches are not
# automatically retried. See {Database#batch}.

This comment was marked as spam.

@blowmage
Copy link
Contributor Author

I had one open question about custom objects with the Ruby client library.

I didn't see that in the original Firestore requirements. Is there any guidance on how this is supposed to work? Does the library automatically map a Document::Snapshot's data to the class? Is this something that is needed to release? Or can it be added afterwards?

Here are some quick thoughts on how to possibly implement this: Probably the best is to include a module in your class. Another option is to see of the object can be represented as a Hash to use it as a Hash. The simplest way is to check if the object responds to the to_h as method, but so many objects can respond to that method that it might easily be a false positive. It would be better if it responded to to_hash, but common libraries such as Struct and OpenStruct don't do this by default.

Copy link

@schmidt-sebastian schmidt-sebastian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First couple of comments. Note that I only looked at the API surface and don't understand Ruby. More comments to follow on the rest of the files.

#
# Batched writes have fewer failure cases than transactions and use
# simpler code. They are not affected by contention issues, because they
# don't depend on consistently reading any documents.

This comment was marked as spam.

# The database resource the Cloud Firestore batch belongs to.
#
# @return [Database] database resource.
def database

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

# end
# end
#
def cols

This comment was marked as spam.

This comment was marked as spam.

# b.set(nyc_ref, { name: "New York City" })
# end
#
def col collection_path

This comment was marked as spam.

@closed = true
return nil if @writes.empty?
resp = service.commit @writes
Convert.timestamp_to_time resp.commit_time

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

# end
#
def docs &block
query.run(&block)

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

# puts "#{city.document_id} has #{city[:population]} residents."
# end
#
def get_all *docs, mask: nil, &block

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

def query
ensure_context!

Query.start(parent_path, context).from(collection_id)

This comment was marked as spam.

# @example
# require "google/cloud/firestore"
#
# firestore = Google::Cloud::Firestore.new

This comment was marked as spam.

This comment was marked as spam.

end
end

def create_writes doc_path, data

This comment was marked as spam.

This comment was marked as spam.

@schmidt-sebastian
Copy link

I had one open question about custom objects with the Ruby client library.

I didn't see that in the original Firestore requirements. Is there any guidance on how this is supposed to work? Does the library automatically map a Document::Snapshot's data to the class? Is this something that is needed to release? Or can it be added afterwards?

Here are some quick thoughts on how to possibly implement this: Probably the best is to include a module in your class. Another option is to see of the object can be represented as a Hash to use it as a Hash. The simplest way is to check if the object responds to the to_h as method, but so many objects can respond to that method that it might easily be a false positive. It would be better if it responded to to_hash, but common libraries such as Struct and OpenStruct don't do this by default.

Whether this is a requirement really depends on how we expect this client to be used. Is it easy to create Firestore objects without it? Or do users commonly have their own types that they want to serialize? We added in Java, since POJOs are very common, but left it out of most other clients.

@blowmage
Copy link
Contributor Author

Whether this is a requirement really depends on how we expect this client to be used.

So I looked at the Python example again, and they don't actually accept the custom object, they call to_dict on it and pass the Dict object to the Firestore code. We can easily do the same with Ruby's Hash as well. Here are what our two code examples would be in that case:

City = Struct.new(:name, :state, :country, :capital, :population)
city = City.new("Los Angeles", "CA", "USA", false, 3900000)
db.col("cities").doc("LA").set(city.to_h)

Would this approach be acceptable?

Copy link
Member

@frankyn frankyn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the Python implementation a user should only need to implement a to_h method for a custom class and possibly a Hash to custom class method. I think documented examples showing how to do this using a Struct with to_h and Hash to a Struct is acceptable.

# Write to document with the provided object values. If the document
# does not exist, it will be created. By default, the provided data
# overwrites existing data, but the provided data can be merged into
# the existing document using the `merge` argument.

This comment was marked as spam.

The documentation was a duplicate of the set method documentation.
# firestore.set(nyc_ref, { name: "New York City" })
#
def col collection_path
if collection_path.to_s.split("/").count.even?

This comment was marked as spam.

# puts "#{city.document_id} has #{city[:population]} residents."
# end
#
def docs collection_path, &block

This comment was marked as spam.

# @param [String, Document::Reference] docs One or more strings
# representing the path of the document, or document reference
# objects.
# @param [Array<String|Symbol>, String|Symbol] mask A list of field

This comment was marked as spam.

# # Create a document
# firestore.create(nyc_ref, { name: "New York City" })
#
def create doc, data

This comment was marked as spam.

# firestore.set("cities/NYC", { name: "New York City" },
# merge: [:name])
#
def set doc, data, merge: nil

This comment was marked as spam.

retry
rescue Google::Cloud::InvalidArgumentError => err
# Return if a previous call has succeeded
return nil if retries > 0

This comment was marked as spam.

# end
# end
#
def read_only_transaction read_time: nil

This comment was marked as spam.

# puts "#{city.document_id} has #{city[:population]} residents."
# end
#
def query

This comment was marked as spam.

# puts "#{city.document_id} has #{city[:population]} residents."
# end
#
def select *fields

This comment was marked as spam.

# puts "#{city.document_id} has #{city[:population]} residents."
# end
#
def get obj

This comment was marked as spam.

@blowmage blowmage closed this Dec 20, 2017
@alixhami alixhami deleted the firestore branch June 22, 2018 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants