Driver test to confirm Firestore client behaviors #39

suztomo · 2020-03-10T05:22:45Z

Fixes #17 .

The example/test_driver/cloud_firestore_behaviors driver test ensures that the behavior of
cloud_firestore_mocks follows the real Firestore client for both Cloud Firestore backend and Firestore emulator backend.

suztomo · 2020-03-10T05:24:44Z

example/ios/Flutter/Generated.xcconfig

@@ -1,9 +0,0 @@
-// This is a generated file; do not edit or check into version control.


This file says it should not be checked into version control.

suztomo · 2020-03-10T05:24:53Z

example/ios/Flutter/flutter_export_environment.sh

@@ -1,10 +0,0 @@
-#!/bin/sh
-# This is a generated file; do not edit or check into version control.


This file says it should not be checked into version control.

suztomo · 2020-03-10T05:25:27Z

example/ios/Podfile

@@ -15,49 +15,64 @@ def parse_KV_file(file, separator='=')
  if !File.exists? file_abs_path


Flutter command said the Podfile was out-of-date and suggested regenerating; I didn't manually modify this.

suztomo · 2020-03-10T05:25:43Z

example/ios/Runner.xcodeproj/project.pbxproj

@@ -44,6 +44,7 @@
 		3B80C3931E831B6300D905FE /* App.framework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; name = App.framework; path = Flutter/App.framework; sourceTree = "<group>"; };


Flutter updated this file; I didn't manually modify this.

suztomo · 2020-03-10T05:39:26Z

example/test_driver/cloud_firestore_behaviors.dart

+  tearDownAll(() => completer.complete(null));
+
+  group('Firestore behavior comparison:', () {
+    Map<String, Future<Firestore>> firestoreFutures = {


I had to make the value futures, because we need to call 'test' function without async.

If the value were without futures, it would have been initialized somewhere in setup method using future/async. When I tried that Flutter did not recognize test function calls.

suztomo · 2020-03-10T05:42:14Z

example/test_driver/cloud_firestore_behaviors.dart

+    firestoreFutures.forEach((name, firestoreFuture) {
+      test('Unsaved documens ($name)', () async {
+        final firestore = await firestoreFuture;
+
+        final CollectionReference recipients = firestore.collection('messages');
+
+        final DocumentReference doc = recipients.document();
+        final String documentId = doc.documentID;
+
+        expect(documentId.length >= 20, true);
+
+        final result = await doc.get();
+        expect(doc.path, 'messages/$documentId');
+        expect(result.data, null);
+      });


This is the motivation for this PR. Asserting the same behavior for the following three Firestore instances:

cloud_firestore with Cloud Firestore backend

cloud_firestore with Firestore emulator backend

cloud_firestore_mocks

suztomo · 2020-03-10T05:44:39Z

example/README.md

@@ -3,6 +3,47 @@
 Demonstrates how to use the cloud_firestore_mocks plugin.

 The example project comes from
-https://github.com/FirebaseExtended/flutterfire/tree/master/packages/cloud_firestore/example,


This URL has changed.

suztomo · 2020-03-10T13:50:14Z

@atn832 The idea of the driver test worked in my environment. Thank you for the advice. When you have time, would you try to confirm it works in your Mac (I assume you have iOS simulator).

atn832 · 2020-03-13T13:18:56Z

This looks amazing. Let me try it this weekend.

atn832 · 2020-03-16T02:50:23Z

I gave your PR a try.

I was able to run the integration tests on an Android emulator and iOS simulator for the real firebase and the mock library (woohoo!), but not the emulator.

When running tests with the emulator, it times out on Android, and returns this error instantly on iOS:

flutter: 00:00 +5 -5: Firestore behavior comparison: Nested objects update (Firestore Emulator) [E]
flutter:   PlatformException(Error 7, FIRFirestoreErrorDomain, 
  false for 'create' @ L16)
flutter:   package:flutter/src/services/message_codecs.dart 569:7        StandardMethodCodec.decodeEnvelope
  package:flutter/src/services/platform_channel.dart 156:18     MethodChannel._invokeMethod
  ===== asynchronous gap ===========================
  dart:async/zone.dart 1064:19                                  _CustomZone.registerBinaryCallback
  dart:async-patch/async_patch.dart 84:23                       _asyncErrorWrapperHelper
  package:test_api/src/backend/invoker.dart                     Invoker.waitForOutstandingCallbacks.<fn>
  dart:async/zone.dart 1126:13                                  _rootRun
  dart:async/zone.dart 1023:19                                  _CustomZone.run
  dart:async/zone.dart 1518:10                                  _runZoned
  dart:async/zone.dart 1465:12                                  runZoned
  package:test_api/src/backend/invoker.dart 239:5               Invoker.waitForOutstandingCallbacks

Yet the emulator seems to be running ok:

% firebase emulators:start --only firestore
i  Starting emulators: ["firestore"]
i  firestore: Logging to firestore-debug.log
✔  firestore: Emulator started at http://localhost:8080
i  firestore: For testing set FIRESTORE_EMULATOR_HOST=localhost:8080

Maybe I need to configure the emulator/simulator in some special way, so it can access my mac's port 8080?

atn832 · 2020-03-16T02:51:18Z

example/README.md

+...
+~/Documents/cloud_firestore_mocks $ which firebase
+/usr/local/bin/firebase
+```


I also had to set up the emulator:

firebase setup:emulators:firestore

And also run firebase init using some project of mine so that it could initialize a firebase.json file.

Added firebase setup:emulators:firestore, and clarified that the test does not expect firebase.json.

My bad, we don't actually need to run setup:emulators:firestore with the latest version! I can remove this comment after the merge if you want.

atn832 · 2020-03-16T02:53:32Z

example/README.md

+```
+
+`test_driver/cloud_firestore_behaviors` assumes the emulator listen on port
+8080 (default) on localhost. This works for iOS simulator running in the same


Did you require any special config for the iOS simulator? The tests for the emulator failed on my machine. I get this error:

flutter: 00:00 +5 -5: Firestore behavior comparison: Nested objects update (Firestore Emulator) [E] flutter: PlatformException(Error 7, FIRFirestoreErrorDomain, false for 'create' @ L16) flutter: package:flutter/src/services/message_codecs.dart 569:7 StandardMethodCodec.decodeEnvelope package:flutter/src/services/platform_channel.dart 156:18 MethodChannel._invokeMethod ===== asynchronous gap =========================== dart:async/zone.dart 1064:19 _CustomZone.registerBinaryCallback dart:async-patch/async_patch.dart 84:23 _asyncErrorWrapperHelper package:test_api/src/backend/invoker.dart Invoker.waitForOutstandingCallbacks.<fn> dart:async/zone.dart 1126:13 _rootRun dart:async/zone.dart 1023:19 _CustomZone.run dart:async/zone.dart 1518:10 _runZoned dart:async/zone.dart 1465:12 runZoned package:test_api/src/backend/invoker.dart 239:5 Invoker.waitForOutstandingCallbacks

@atn832 I didn't setup anything special for my iOS simulator or Firestore emulator (or firebase.json).

My environment

Simulator version: Version 11.3.1 (SimulatorApp-912.5.1 SimulatorKit-570.3 CoreSimulator-681.17.2)

My Mac OS Catalina: 10.15.3

XCode 11.3.1

firebase command 7.15.0 (latest)

Firestore emulator

In your emulator startup output, I don't see "Could not find config (firebase.json) so using defaults.". Aren't you using some firebase.json that defines security rules?

Mine:

~ $ firebase emulators:start --only firestore ⚠ Could not find config (firebase.json) so using defaults. i emulators: Starting emulators: firestore ⚠ firestore: Did not find a Cloud Firestore rules file specified in a firebase.json config file. ⚠ firestore: The emulator will default to allowing all reads and writes. Learn more about this option: https://firebase.google.com/docs/emulator-suite/install_and_configure#security_rules_configuration. i firestore: Serving ALL traffic (including WebChannel) on http://localhost:8080 ⚠ firestore: Support for WebChannel on a separate port (8081) is DEPRECATED and will go away soon. Please use port above instead. i firestore: firestore emulator logging to firestore-debug.log ✔ firestore: firestore emulator started at http://localhost:8080 i firestore: For testing set FIRESTORE_EMULATOR_HOST=localhost:8080 ✔ emulators: All emulators started, it is now safe to connect.

Would you try running the emulator without firebase.json?

Thanks for sharing the details of your environment. I had 7.0.0 installed. I must have had installed it long ago, and running the install script didn't actually update it. With 7.15.0, it works much better!

Version 7.0.0 required a firebase.json or it would fail (Error: Not in a Firebase app directory (could not locate firebase.json)). It also would fail if I hadn't pre-downloaded the emulator. But the new version downloads the newer emulator automatically when first run.

Beautiful!

flutter: 00:00 +0: Firestore behavior comparison: Document creation by add (Cloud Firestore) VMServiceFlutterDriver: Connected to Flutter application. path: satisfied (Path is satisfied), interface: en1 flutter: 00:01 +1: Firestore behavior comparison: Document creation by add (Firestore Emulator) path: satisfied (Path is satisfied), interface: en1 flutter: 00:01 +2: Firestore behavior comparison: Document creation by add (cloud_firestore_mocks) flutter: 00:01 +3: Firestore behavior comparison: Document creation by setData (Cloud Firestore) flutter: 00:02 +4: Firestore behavior comparison: Document creation by setData (Firestore Emulator) flutter: 00:02 +5: Firestore behavior comparison: Document creation by setData (cloud_firestore_mocks) flutter: 00:02 +6: Firestore behavior comparison: Timestamp field (Cloud Firestore) flutter: 00:03 +7: Firestore behavior comparison: Timestamp field (Firestore Emulator) flutter: 00:03 +8: Firestore behavior comparison: Timestamp field (cloud_firestore_mocks) flutter: 00:03 +9: Firestore behavior comparison: Unsaved documens (Cloud Firestore) flutter: 00:04 +10: Firestore behavior comparison: Unsaved documens (Firestore Emulator) flutter: 00:04 +11: Firestore behavior comparison: Unsaved documens (cloud_firestore_mocks) flutter: 00:04 +12: Firestore behavior comparison: Nested objects creation with updateData (Cloud Firestore) flutter: 00:05 +13: Firestore behavior comparison: Nested objects creation with updateData (Firestore Emulator) flutter: 00:05 +14: Firestore behavior comparison: Nested objects creation with updateData (cloud_firestore_mocks) flutter: 00:05 +15: Firestore behavior comparison: Nested objects update (Cloud Firestore) flutter: 00:06 +16: Firestore behavior comparison: Nested objects update (Firestore Emulator) flutter: 00:06 +17: Firestore behavior comparison: Nested objects update (cloud_firestore_mocks) flutter: 00:06 +18: (tearDownAll) flutter: 00:06 +19: All tests passed! Stopping application instance.

Glad to hear it worked.

atn832

This is a huge accomplishment!

Feel free to tweak the documentation and merge when you're done.

I wonder if eventually, it would be possible to run the full battery of unit tests from cloud_firestore_mocks/test/* on the mock library and emulator/real firebase as part of #19.

suztomo · 2020-03-16T04:19:00Z

@atn832 Updated the document. Regarding PlatformException(Error 7, FIRFirestoreErrorDomain, false for 'create', I suspect your emulator picked up some security rules (that could be from firebase init). Would you try running the emulator without firebase.json?

suztomo · 2020-03-16T04:27:17Z

I don't have write access to the repository. Would you merge current branch?

atn832 · 2020-03-16T13:15:51Z

Merged. Also I've sent you an invite to collaborate. :)

suztomo added 8 commits March 9, 2020 21:18

Updating dependencies

7181855

Updating Podfile

18ef242

Updating iOS files

5b2861d

Excluded test that fails in emulator

a6e745a

Driver test to compare Firestore behaviors

df1df60

Revert unnecessary change

aefc710

format

2c12b61

Added timestamp type check

a8b60a3

suztomo commented Mar 10, 2020

View reviewed changes

suztomo added 3 commits March 10, 2020 09:23

Added iPhone simulator

8643583

Adding README

fc30b6b

Note for Android port

680cfd3

More test cases

c17f805

suztomo mentioned this pull request Mar 10, 2020

UpdateData should throw PlatformException if no existing document #40

Closed

suztomo added 2 commits March 10, 2020 20:04

Simplified test cases

1214278

Merge remote-tracking branch 'origin/master' into i17

78d87d3

suztomo added 3 commits March 13, 2020 09:35

Merge remote-tracking branch 'origin/master' into i17

8b72fd4

updating comment

95f587e

Removed commented line

afd996d

atn832 reviewed Mar 16, 2020

View reviewed changes

atn832 approved these changes Mar 16, 2020

View reviewed changes

Updating document

85a07f3

atn832 merged commit 5d7116b into atn832:master Mar 16, 2020

suztomo deleted the i17 branch March 16, 2020 14:04

suztomo mentioned this pull request Mar 16, 2020

Run tests automatically on new PRs #19

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Driver test to confirm Firestore client behaviors #39

Driver test to confirm Firestore client behaviors #39

suztomo commented Mar 10, 2020 •

edited

suztomo Mar 10, 2020

suztomo Mar 10, 2020

suztomo Mar 10, 2020

suztomo Mar 10, 2020

suztomo Mar 10, 2020

suztomo Mar 10, 2020

suztomo Mar 10, 2020

suztomo commented Mar 10, 2020

atn832 commented Mar 13, 2020

atn832 commented Mar 16, 2020

atn832 Mar 16, 2020

atn832 Mar 16, 2020

suztomo Mar 16, 2020

atn832 Mar 16, 2020

atn832 Mar 16, 2020

suztomo Mar 16, 2020 •

edited

atn832 Mar 16, 2020

atn832 Mar 16, 2020

suztomo Mar 16, 2020

atn832 left a comment

suztomo commented Mar 16, 2020

suztomo commented Mar 16, 2020

atn832 commented Mar 16, 2020

		@@ -1,9 +0,0 @@
		// This is a generated file; do not edit or check into version control.

		@@ -1,10 +0,0 @@
		#!/bin/sh
		# This is a generated file; do not edit or check into version control.

		@@ -15,49 +15,64 @@ def parse_KV_file(file, separator='=')
		if !File.exists? file_abs_path

		@@ -44,6 +44,7 @@
		3B80C3931E831B6300D905FE /* App.framework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; name = App.framework; path = Flutter/App.framework; sourceTree = "<group>"; };

Driver test to confirm Firestore client behaviors #39

Driver test to confirm Firestore client behaviors #39

Conversation

suztomo commented Mar 10, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suztomo commented Mar 10, 2020

atn832 commented Mar 13, 2020

atn832 commented Mar 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suztomo Mar 16, 2020 • edited

Choose a reason for hiding this comment

My environment

Firestore emulator

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

atn832 left a comment

Choose a reason for hiding this comment

suztomo commented Mar 16, 2020

suztomo commented Mar 16, 2020

atn832 commented Mar 16, 2020

suztomo commented Mar 10, 2020 •

edited

suztomo Mar 16, 2020 •

edited