Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP basic cluster communication #35

Open
wants to merge 73 commits into
base: master
Choose a base branch
from
Open

WIP basic cluster communication #35

wants to merge 73 commits into from

Conversation

psFried
Copy link
Owner

@psFried psFried commented Nov 22, 2017

No description provided.

psFried and others added 30 commits November 20, 2017 10:34
@psFried
fix how status readers/writers are created and passed around when ini…
f06424d 
…tializing event streams
@psFried
initialize FloController on startup, though it doesn't actually do an…
d63a2bd 
…ything yet
@psFried
move engine initialization into a separate module
5ff88c0
@psFried
only execute OSX builds with stable rust since they take longer
56d136b
@psFried
set exit code to 1 when running server fails with error
57b11bf
@psFried
increase timeout on expiration test to account for really slow OSX ma…
151bdc8 
…chines on travis
@psFried
WIP on cluster communication
5373124
@psFried
more WIP on cluster communication
0c906d7
@psFried
create separate submodules for each message in flo-protocol
13dd6ae
@psFried
cleanup compiler warnings in protocol and add tests for serializing s…
d291cc1 
…ocket addresses
@psFried
remove unused SetBatchSize protocol message
abce683
@psFried
protocol level implementation of append entries
caee363
@psFried
add request vote messages to protocol
e4c4106
@psFried
keep a reference to the socket address of the current primary server
092a1a9
@psFried
change connection handler to process either protocol or control messages
a35ac27
@psFried
fix produce operation in flo-client cli
7de19a3
@psFried
factor out spawning of connection handler so it can be reused for cre…
b41d765 
…ating outgoing connections

change system stream to use it's own operation type
@psFried
more shuffling things aroung in preparation for outgoing connections
2ea6081
@psFried
WIP on partition controller and managing peer connections, mostly jus…
364e92d 
…t beating around the bush tbh
@psFried
remove peer replication connection module until we are actually ready…
03969e5 
… for it
@psFried
add new fields to peer announce message so members can tell who the p…
0e7ca67 
…rimary is
@psFried
sketch in a real initialization flow for cluster state
06ab54d
@psFried
factor out trait objects for some of the cluster-related state manage…
3d7be05 
…ment, handle peerAnnounce handshake in connectionHandler
@psFried
Send a Tick operation to the system controller at a random interval b…
8383bb9 
…etween 150 and 300 milliseconds
@psFried
remove debug logging from travis builds
7e3b273
@psFried
work basic cluster state and management into initialization
36732c4
@psFried
connections established on startup with known cluster members
2d6223c
@psFried
peer connections are completing the handshake
2f25413
@psFried
change connectionHandler to shutdown on Drop instead or Sink::close
405cb85
@psFried
handle requestVote in connectionHandler
de6e49d
@psFried
change how AppendEntries is sent so that peer offset is maintained by…
a7972cd 
… the connection handler
@psFried
WIP on sending AppendEntries
1aa8d21
@psFried
controller receives appendEntries and sends response, still not actua…
1e04eac 
…lly appending any entries, though
@psFried
primary is elected, seemingly correctly, but still no way to actually…
44cfd28 
… add entries to the system partition
@psFried
tweak travis config to try to speed up builds
b2610ec
@psFried
basic wiring in of commit manager into partitionImpl
6c73000
@psFried
remove unused method from controller state, tell consensus processor …
1c1b121 
…about closed connections
@psFried
start adding ability for partition to notify producers once events ar…
1041e56 
…e committed, only used for system partition right now
@psFried
prevent system stream from being re-initialized as a user stream, add…
5f0c8c6 
… ability for client and cli to use a specific event stream
@psFried
change default stream name to 'user' since the system stream is now l…
3bdf125 
…ocked down
@psFried
backup commit, wip on replication operations
80c2e75
@psFried
add crc function to flo-event
0dc9b02
@psFried
get flo-server compiling again
8aaa098
@psFried
add a CRC to events
56a9bab
@psFried
POC implementation of event replication is sort of working in partiti…
1a49cc4 
…on controller
@psFried
finish the firs basic replication test, make partitionReader explicit…
5424d26 
… about iterating committed vs uncommitted events
@psFried
server handles committed and uncommitted consumers separately and pro…
8148956 
…tocol allows clients to distinguish which one they want
@psFried
add a commitIndexUpdated protocol message
aef29dc
@psFried
change SystemEvent representation to include deserialized data
5f68b1d
@psFried
back out system event additions, change FloInstanceId to be a typedef…
824474d 
… instead of a wrapper type to make serde simpler
@psFried
cleanup compiler warnings
c21e0e5
@psFried
prevent users from producing invalid system events
c09e15f
@psFried
add script to start cluster with multiple members
dbe87d0
@psFried
minor improvements to dev-start-cluster.sh
51b3c58
@psFried
- Make all partitions start as non-writable when in cluster mode (sti…
dd0c681 
…l working on actually setting them to writable again)

- fix blanket impl of FloEvent
- cleanups and foundational work in preparation for actually using system events
@psFried
adjust system event types in prep for cluster membership rewrite
b8264e1
@psFried
start having system events applied to persistent state
84ca31b
@psFried
apply the remaining system event to persistent state
f910402
@psFried
process acknowledgements from peers
ed17aa2
@psFried
backup commit - squash me
f7c94f0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@psFried