Cross-reference against official functional test suite

[rfk]

Hi @posativ, great work on this minimal server! I'm one of the devs for mozilla's sync server and yeah, it can be a pain to set up a local install. Your work is appreciated.

One thought for increasing compatibility between this and the official version is to run the functional tests from the http://hg.mozilla.org/services/server-storage repo against your server. This will help to highlight any corner-cases where the functionality differs between the two.

Here's how I managed to do it. First, checkout and build the server-storage repo:

hg clone http://hg.mozilla.org/services/server-storage
cd server-storage
make build

Run your minimal server with a test user:

cd ../
git clone https://github.com/posativ/weave-minimal.git
cd weave-minimal
./weave.py --register test:test
./weave.py &

Then execute the functional tests, directing them at the running server:

cd ../server-storage
./bin/python syncstorage/tests/functional/test_storage.py -u test -p test http://127.0.0.1:8080

For me this reports 32 tests run, 9 failures, 9 errors, 4 skipped. That's a pretty impressive success rate already!

Some of the failures may be spurious, e.g. they check implementation details of our server. And frankly, some of them may be bugs in our server that have since been codified into tests...

Anyway, hope you find this helpful.

[posativ]

Hi @rfk thank you for that detailed issue report! I didn't know which repository contains the latest tests, so I took mine from mozilla-services/server-full. Hence, my implementation focused on theses tests (I had to fix first) and they run through with around 85% success rate.

I just cloned server-storage and see, that some functional tests fail. I'll try to fix them as fast as possible! It is a way better testsuite than the old I used before.

[posativ]

I think I have fixed most tests so far. Still two are failing but I am not sure whether this is my issue or the random order of the tests … (especially test_get_item(self) in line 337).

Here's a diff of three minor changes to the testsuite so 30 of 32 tests passes. The first is a bad status code and should be 412 as described in the Storage API 1.1 and the latter two are limitations my implementation does not have :-)

diff -r 63392565d91a syncstorage/tests/functional/test_storage.py
--- a/syncstorage/tests/functional/test_storage.py  Mon Sep 03 12:52:24 2012 +1000
+++ b/syncstorage/tests/functional/test_storage.py  Tue Sep 04 18:09:51 2012 +0200
@@ -583,7 +583,7 @@
         self.assertEquals(res.json["payload"], _PLD)

         # deleting all with no confirmation
-        self.app.delete(self.root + '/storage', status=400)
+        self.app.delete(self.root + '/storage', status=412)

         # deleting all for real now
         res = self.app.delete(self.root + '/storage',
@@ -699,16 +699,16 @@
         wbos = [{'id': str(i), 'payload': _PLD} for i in range(105)]
         res = self.app.post_json(self.root + '/storage/col2', wbos)
         res = res.json
-        self.assertEquals(len(res['success']), 100)
-        self.assertEquals(len(res['failed']), 5)
+        self.assertEquals(len(res['success']), 105)
+        # self.assertEquals(len(res['failed']), 5)
         # The test config has max_bytes=1M.
         # Uploading 5 210MB items should produce one failure.
         wbos = [{'id': str(i), 'payload': "X" * (210 * 1024)}
                 for i in range(5)]
         res = self.app.post_json(self.root + '/storage/col2', wbos)
         res = res.json
-        self.assertEquals(len(res['success']), 4)
-        self.assertEquals(len(res['failed']), 1)
+        self.assertEquals(len(res['success']), 5)
+        # self.assertEquals(len(res['failed']), 1)

     def test_blacklisted_nodes(self):
         # This can't be run against a live server.