Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Missing data in S3 archive? #80

Closed
mrdavidlaing opened this Issue · 6 comments

2 participants

@mrdavidlaing
Collaborator

I'm trying to plot a latency scatterplot from the raw data collected on May 23rd ( when the new TradingAPI was released, and we see the increase in measured latency )

I'm pulling my data from https://s3.amazonaws.com/cityindex.appmetrics/CiapiLatencyCollector/2012-05/*.zip

However, my plot seems to be missing data for a bunch of dates:

Is it possible that S3 is missing some source data?

@fandrei fandrei was assigned
@fandrei
Owner

Yes, it's possible.
Backup sends file to S3 when it's last update is 7 days old, and before this date all records from that file are not present in S3 storage

@mrdavidlaing
Collaborator
@fandrei
Owner

Sure, or even more frequently. But the problem is that file won't be sent to S3 until appropriate user session is finished, and some sessions are many days long. It's made this way because it's impossible to "append" data to S3 object, only completely rewrite it. But I can change this behavior.

@mrdavidlaing
Collaborator
@fandrei
Owner

As "session" here I mean AppMetrics session, not CIAPI

@fandrei
Owner

Getting raw data directly from AppMetrics server:
https://github.com/fandrei/AppMetrics/wiki/Getting-raw-data

@fandrei fandrei closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.