New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(#417) Replication #740

Closed
wants to merge 57 commits into
base: vNext
from

Conversation

Projects
None yet
6 participants
@yaronyg
Member

yaronyg commented Jun 24, 2016

@jareksl I know this is huge but it's not going to get any smaller and I need to check it in sometime. It won't pass CI tests because of the Express-PouchDB problem but that will hopefully just be a bug fix. Please ignore all the java files as those are part of #720 and will be reviewed there. So please just pay attention to, well, everything else. A lot of the changes are just linting so in reviewable it should be easy to see that something is just a spacing change and not changing content. Sorry for dumping this monster on you but somebody has to do it. :)


This change is Reviewable

yaronyg and others added some commits Apr 7, 2016

WIP
TeamReadMe - Linting

testThaliNotificationAction - Check that action is in KILLED state after
start returns

thaliNotificationAction - Making sure we go to killed state upon
completion.

thaliNotificationClient - Linting

thaliReplicationPeerAction - Starting on the journey to implementing this.

thaliPeerAction - Linting
WIP
testLocalSeqManager, localSeqManager, thaliPullReplicationFromNotifiction,
thaliReplicationPeerAction - First skeletal work on replication

thaliConfig - Added EC curve definition and some arguments for replication

thaliSendNotificationBasedOnReplication - Exposing APIs to get
public keys and secrets

thaliWiFiInfrastructure - Call our public key api

thaliReplicationManager - Adding in SALTI and hooking things up

package.json - Add in SALTI as well as express-pouchdb

thaliPeerAction - Fixed error in spec

testThaliNotification, testThaliNotificationAction,
testThaliNotificationClient, testThaliNotificationLocal,
testThaliNotificationServer, testThaliPeerDictionary,
thaliNotificationBeacons - Replaced the hard coded calls to our EC curve
with the thaliconfig version
Merge remote-tracking branch 'origin/vNext' into vNext_tompaana_multi…
…ple_issues

# Conflicts:
#	thali/package.json
WIP
testLocalSeqManager - Playing around with how we need to set things so
that our calls will work

testUtils - Let us get a raw level down enabled PouchDB instance

localSeqManager - Clean up, still not done

thaliPullReplicationFromNotification - Fixed typos

thaliReplicationPeerAction - Mostly doc cleanup

thaliReplicationManager - Clean up paths and removing SALTI for now

thaliWifiInfrastructure - Removed unnecessary value
WIP - Getting Android tests to pass
testThaliMobileNativeWrapper.js - Hardened the tests so they would pass
even when the Bluetooth stack is being wonky. Fixed bug #717 to make
us properly handle turning off Bluetooth.

testUtils - Create a hardened HTTP GET that will keep trying even when
there are connection failures.

README - Fixed a syntax error in instructions

testIdentityExchangeEndPoint, testLiveIdentityExchange,
testSmallerHashStateMachine - Lint
Enable final test to exit if run too long
testThaliMobileNative - The last test would hang if anything went wrong,
now it has a time out. Note that the time out isn't perfect and causes
a double call to .end and no I don't care right now.

testThaliMobileNativeWrapper - Removed spurious logger definition

testUtils - Turns out that code that runs before we set Mobile in test
situations use testUtils so I had to delay us loading
thaliMobileNativeWrapper (which we need for one test function) until after
everything else to make sure we didn't load thaliMobileNativeWrapper before
mobile was defined.
WIP
testLocalSeqManager - Added a bunch of tests, the last one in the file is
failing because I don't properly return a promise in the case where
we are waiting for a timer to expire in order to run the next update. Will
fix tomorrow.

httpTester - Added utility to do GETs to a PSK server

localSeqManager - Fixing lots of bugs and adding functionality

utilities - Linting
WIP -
Readme - Add docs on --waitforInstance

testLocalSeqManager - All non-coordinate tests are now done and passing!

testLocalSeqManagerCoordinate - Still debugging why this isn't working

httpTester - Added some PSK tests for getting docs and sequence #s

testUtils - Created utilites to validate combined result objects and
generate a remote facing PSK PouchDB server object

package.json - In both cases I updated lie because I thought there was a
bug in lie but there was actually a bug in my own code. Oh well. Didn't
seem to hurt anything.

thaliNotificationServer - Mostly linting but also explicitly setting the
response to 200

localSeqManager - I hardened this so no matter what we won't have more
than two request promises outstanding.

thaliPeerPoolDefault - Now catch and log errors

testThaliMobile, testThaliNotificationBeacons, testThaliNotificationLocal,
testThaliNotificationServer, testThaliPeerDictionary,
testThaliWifiInfrastructure, thaliMobile - fixing casing and reference
issues around thaliConfig that were caused because I moved our elliptical
curve setting there.

testThaliPeerPoolDefault, httpTester, thaliConfig,
thaliMobileNativeWrapper, thaliWifiInfrastructure - Removed our dependency
on the bogus certs we needed before Shawn fixed a bug in how PKS support
worked in JXcore on TLS.
Fixed our sequence record IDs
I screwed up and for some reason thought that sequence IDs for records
in _local were supposed to use the hashed version of the public key. I
also screwed up and previously thought that the record prefix was
supposed to be thali instead of thali_. And while I was fixing my
screw ups I also moved the thali_ prefix into thaliConfig.

testLocalSeqManagerCoordinated - Linting

testThaliSendNotificationBasedOnReplication - moved the prefix to
thaliConfig

httpTester - Fixed syntax for sequence ID

localSeqManager - Fixed ID generation

thaliSendNotificationBasedOnReplication - Moved id prefix

thaliConfig - Added ID prefix here
Hard code versions
I ran into some issues that may or may not have been caused by the fact
that I am running older versions of some dependencies than a fresh install
would get. This kind of inconsistency leads to madness so for now we'll
hard code specific versions rather than use Semver. I filed an issue,
#266 to specify that we need to use something like greenkeeper so we
can update when new versions of our dependencies come out.
Fixing minor bugs
package.json - Was using the new syntax with an old version of randomstring
so needed to upgrade.

makeIntoCloseAllServer - Fixed an old bug where I screwed up splice
Got localSeqManager tests passing!
testLocalSeqManagerCoordinated - Scoping issues, timing issues, etc.

testUtils - See the notes but there is a bug in PouchDB
(pouchdb/pouchdb#5231) that was causing us
problems that I found a nasty work around for.

thaliNotificationServer - No functional changed, just refactored the code
to make it easier for me to intercept the function's output when I need
to log it.
WIP
testLocalSeqManager - Moved setup server to testUtils so I had to set
	testCloseAllServer on each test.

testLocalSeqManagerCoordinated - Created a timeout so the test can't run
	forever.

testThaliReplicationPeerAction - Starting to rough out and get passing
	some basic tests.

testUtils - Added functions to create PSK enabled test servers, PouchDB
instances for PSK remote servers and create PouchDB instances which will
store their contents in randomly named directories under the current test
directory.

thaliTape - Ran into a problem with a PouchDB error that had a loop in its
Javascript and caused JSON.stringify to blow up so had to get rid of
JSON.stringify. Eventually we might want to use try/catch if this proves
too annoying.

thaliNotificationAction - lint

localSeqManager - lint

thaliReplicationPeerAction - Rough draft

thaliACLLayer - Leaving a note about implementation
WIP -
testthaliReplicationPeerAction - Added time out test and got existing tests
to pass.

thaliReplicationPeerAction - Fixed a silly bug where I was passing live
into the wrong place and not stopping the sequence manager when we
stopped.
ThaliReplicationPeerAction Testing w/out coordinator Done
testThaliReplicationPeerAction - Added in the final tests

httpTester - Added retry capability to validateSeqNumber
Finished thaliReplicationPeerAction testing
testLocalSeqManagerCoordinated - Moved runTestOnAllParticipants to
testUtils so I could re-use with thaliReplicationPeerAction tests.

testThaliReplicationPeerAction - Needed to start at begining of DB not
from most recent chnages.

testThaliReplicationPeerActionCoordinated - A very simple coordinator
test.

testUtils - Added runTestOnAllParticipants

localSeqManager - Making auto detect happy by specifying the argument is
optional.

testthaliNotification, testThaliNotificationClient,
testThaliNotificationLocal, thaliNotificationClient - I moved the
Events object from the object to a prototype because otherwise I had
to require the thaliNotificationClient object which calls thaliMobile
which makes some unhappy calls for unit testing. The change should be fine
as no one really needs the events value unless they are registering for
an event in which case they have an instance of the
thaliNotificationClient.
WIP - Handled consequences of prefix breaking change in PouchDB
identityExchangeTestUtils - Centralized the PouchDB creator function &
linting


testIdentityExchangeEndpoint - Rehooked pouchdb creator function and lint

testIdentityExchangeUtils - Linting

testLiveIdentityExchnage - Hook to central PouchDB creator funcion and
linting

testThaliReplicationManager - Deleted, this is outdated code

testLocalSeqManager - Linting

testThaliNotificationClient - Make sure we stop

testThaliReplicationPeerActionCoordinated - Removed only

httpTester - Extracted sequence doc path function for no useful reason

testUtils - Switched to pouchdb-node (it's smaller) and created new
PouchDB creator function that lets us work around
#738

test/www/jxcore/package.json - Switched from PouchDB to pouchdb-node to
save size and
updated to latest release to get a bug fix we needed. Also updated
express-pouchdb for no good reason. And added in inherits and js-extend
because the code in testUtils needs it

thaliNotificationClient - Fixed a bug where we leaked event listeners.

localSeqManager - Changed the form of the PUT for PouchDB because the old
form is now deprecated

thaliPullReplicationFromNotification - Just starting to work on this.

connectionTable - Linting

thali/package.json - Updating epxress-pouchdb to keep everything consistent

thalireplicationmanager - Deleted, old code
WIP - Starting on thaliPullReplicationFromNotification
testThaliPullReplicationFromNotification - Tested start, stop and the key
function. Defined the work for testing core functionality.

testUtils - Broke out code to generate random pouchdb names to make
replication testing easier

thaliPullReplicationFromNotification - Finished functionality, need to
test
Finished non-coordinated pull replication tests
testThaliPullReplicationFromNotification - Make sure we queue and handle
events properly

thaliNotificationClient - Didn't need the if check because we already
check that the submitted value is an array

thaliPullReplicationFromNotification - Lint
Finished testing replication & added stop to default pool
There is an unhandled connection refused error when running under stress.

README.md - Just clarifying

testThaliPullReplicationFromNotificationCoordinated - A coordinated test
for the pull replication from notification class.

testThaliPeerPoolDefault - Testing the new stop functionality

testUtils - Refactored out a method that I needed to use somewhere else

thaliPeerPoolDefault - Added a stop method to clean up after tests so we
didn't get a bunch of failing actions mucking things up

@yaronyg yaronyg added the in progress label Jun 24, 2016

@ThaliTester

This comment has been minimized.

Show comment
Hide comment
Member

ThaliTester commented Jun 24, 2016

@yaronyg

This comment has been minimized.

Show comment
Hide comment
@yaronyg

yaronyg Jun 24, 2016

Member

We won't actually check this in until #720 is checked in and we can get the known test issues resolved.


Reviewed 8 of 30 files at r4, 23 of 54 files at r5.
Review status: 31 of 62 files reviewed at latest revision, 2 unresolved discussions.


a discussion (no related file):
BTW I went through the files and marked as green the ones that were pure linting or part of #720. That actually got rid of about 1/2 the files in the PR. So you can just focus on the red ones.


Comments from Reviewable

Member

yaronyg commented Jun 24, 2016

We won't actually check this in until #720 is checked in and we can get the known test issues resolved.


Reviewed 8 of 30 files at r4, 23 of 54 files at r5.
Review status: 31 of 62 files reviewed at latest revision, 2 unresolved discussions.


a discussion (no related file):
BTW I went through the files and marked as green the ones that were pure linting or part of #720. That actually got rid of about 1/2 the files in the PR. So you can just focus on the red ones.


Comments from Reviewable

Fixed not handling beacon results with no value for us
testThaliMoble - Moved out function to re-use elsewhere

testThaliNotification - Wasn't calling thaliMobile.stop() in teardown
which screwed up other thaliMobile tests down the line. Also did some
linting and checked thaliMobile.start() result

testThaliNotificationClient - Test to make sure we handle beacons with
no results for us

testThaliPullReplicationFromNotification - Accidentally checked in an only

testUtils - Added another function to validate thaliMobile.start() results
and removed a JSON.stringify that led to an exception when we got an
object with circular references

thaliNotificationClient - Properly handle getting beacon results with
no value for us
@ThaliTester

This comment has been minimized.

Show comment
Hide comment
@ThaliTester

ThaliTester Jun 24, 2016

Member

PR is added to the queue for testing as 1. task. (8130305)

Member

ThaliTester commented Jun 24, 2016

PR is added to the queue for testing as 1. task. (8130305)

@ThaliTester

This comment has been minimized.

Show comment
Hide comment
Member

ThaliTester commented Jun 24, 2016

@jareksl

This comment has been minimized.

Show comment
Hide comment
@jareksl

jareksl Jun 29, 2016

Member

test/www/jxcore/bv_tests/testThaliMobile.js, line 3 [r6] (raw file):

'use strict';

var ThaliMobile = require('thali/NextGeneration/thaliMobile');

By the way as you corrected other lint warnings it would be good to correct the rest of variable names to be consistent with the rest of the code. They should start with a lower case letter. The PascalCase should be used for constructor functions.


Comments from Reviewable

Member

jareksl commented Jun 29, 2016

test/www/jxcore/bv_tests/testThaliMobile.js, line 3 [r6] (raw file):

'use strict';

var ThaliMobile = require('thali/NextGeneration/thaliMobile');

By the way as you corrected other lint warnings it would be good to correct the rest of variable names to be consistent with the rest of the code. They should start with a lower case letter. The PascalCase should be used for constructor functions.


Comments from Reviewable

@yaronyg yaronyg added the 3 - Working label Jul 5, 2016

@yaronyg

This comment has been minimized.

Show comment
Hide comment
@yaronyg

yaronyg Jul 5, 2016

Member

@jareksl When do you think you will have a chance to finish this review? Thanks!

Member

yaronyg commented Jul 5, 2016

@jareksl When do you think you will have a chance to finish this review? Thanks!

@jareksl

This comment has been minimized.

Show comment
Hide comment
@jareksl

jareksl Jul 7, 2016

Member

Reviewed 8 of 30 files at r4, 48 of 54 files at r5, 6 of 6 files at r6.
Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.


Comments from Reviewable

Member

jareksl commented Jul 7, 2016

Reviewed 8 of 30 files at r4, 48 of 54 files at r5, 6 of 6 files at r6.
Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.


Comments from Reviewable

@yaronyg yaronyg assigned yaronyg and unassigned jareksl Jul 7, 2016

@yaronyg yaronyg added this to the New Infra milestone Jul 11, 2016

@yaronyg yaronyg added 2 - Ready and removed 3 - Working labels Jul 22, 2016

andrew-aladev added some commits Jul 25, 2016

Merge pull request #776 from thaliproject/vNext_aaladev_773
Create a custom Thali PouchDB constructor utility
Merge pull request #795 from thaliproject/vNext_aaladev_773
Fixed default adapter and test of pouchdb generator
@ThaliTester

This comment has been minimized.

Show comment
Hide comment
@ThaliTester

ThaliTester Aug 3, 2016

Member

PR is added to the queue for testing as 2. task. (56750c6)

Member

ThaliTester commented Aug 3, 2016

PR is added to the queue for testing as 2. task. (56750c6)

@ThaliTester

This comment has been minimized.

Show comment
Hide comment
@ThaliTester
Member

ThaliTester commented Aug 3, 2016

@artemjackson artemjackson referenced this pull request Aug 24, 2016

Closed

Updating Mux #901

35 of 35 tasks complete
@yaronyg

This comment has been minimized.

Show comment
Hide comment
@yaronyg

yaronyg Aug 26, 2016

Member

Closing in favor of vNext_yarong_417_812

Member

yaronyg commented Aug 26, 2016

Closing in favor of vNext_yarong_417_812

@yaronyg yaronyg closed this Aug 26, 2016

@yaronyg yaronyg removed the 2 - Ready label Aug 26, 2016

@evabishchevich evabishchevich deleted the vNext_yarong_417_1 branch Feb 22, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment