setup instructions

[fippo]

are missing. Making a list here.

• what is the primary key of the dynamodb table? 'Item'?
• how is export to redshift done
• what needs to be configured for passport?

[ggarber]

• Primary partition key ConnectionId (String)
• Export is done with Data Pipeline. you can do it manually with this pgsql query:
  copy features from 'dynamodb://Snoop'
  credentials 'aws_access_key_id=key;aws_secret_access_key=secret'
  readratio 50;
• You need to create an application in github developers site. But that's only if you want to use the website that is not very useful yet :(

[fippo]

installation went smoothly. After fighting with AWS security groups. zero effort for integration, just dumped the script with changed path into things.

[fippo]

do you have a definition for data pipeline? I now want the full pain :-)

[fippo]

renaming tables:

alter table features rename to features_old

also: increase the read capacity or dynamodb to 100-200 and use a read ratio of 95

[jmg-syncrtc]

@fippo @ggarber Could you please provide more details in README file in order to configure the server. I have inspected the yaml files in config folder but I still do not understand how works the flow. I am able to run the client, but I do not understand how the client send the stats to the server or how to run the server in order to collect all the data from the different clients.

I think it would be great you guys could bring some light on it, I am really interested in using your application, but I need some extra steps to put all the pieces together.

Thanks in advance guys!

[fippo]

@jmg-syncrtc: unfortunately neither of us has time for documenting this -- and setting up the full server->kinesis->redshift workflow has taken several days the last time we tried.

Maybe @dagingaa can shed some light on this

[jmg-syncrtc]

Thanks for letting me know, @fippo.

@dagingaa , could you, please, provide further steps in order to configure the server? Maybe, not the final setup document, but any help would be great to unlock me.

Thanks so much

[dagingaa]

Err, one day I might have time to open source the terraform config for this, but currently I can't. I agree it's not a lot to go on here, but how I did it was:

• Modify rtcstats to run in elastic beanstalk, I have a specific fork of this to do that, since I use travis to do the deploying
• Create an s3 bucket to dump logs into
• Create a redshift cluster
• Create a firehose and point that to your redshift cluster
• Create the database table using https://github.com/fippo/rtcstats-server/blob/develop/features.sql (it's not completely up to date, so sometimes I scroll through the commit history to find stuff, or look up the source code if I find anything missing that I want)
• Config rtcstats with the firehose stream and s3 bucket
• Now run it and configure the rtcstats-client to connect to it

Reason I'm being so short on these things is that it's surprisingly complex to do, and I don't have the time or energy to sit down and document it currently. At some point some kind soul might contribute patches to this to make it easier and more understandable to get it up and running. I just know this stuff because I spent time figuring out how to deploy it by reading the source and knowing a ton of amazon stuff from before, which certainly made things easier. Sadly (and I honestly regret this), I didn't take the time to document what I did back then.

Confrere might consider donating some time to rtcstats in the future, but it won't be right now.

[jmg-syncrtc]

Thanks so much for this @dagingaa , maybe it is not so documented as you'd like, but I think you have provided very good tips here in order to make the installation easier.

Thanks again guys