You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Initial idea: everything in S3 with a micro server. Server requests map to simple s3cmd commands
Importing to S3 turned out to be very slow for ~500k objects, each only a few bytes. Bad idea; too much hammer not enough nail.
Parse.com, import a csv, grab a random object.
No random query, would love to use Parse or Firebase to host this though
Current idea: each user gets assigned to a namespace and only grabs the next object from their namespace, hopefully greatly reducing the chance of clashing (ie multiple users editing the same thing at the same time which would happen with a single namespace).
What happens when one namespace runs out of errors? or when they all start running low and everyone is clashing? Assign multiple namespaces (5) to a user and cycle through them? How do we ensure even distribution?
Do we need a checkout system again? Very much want to avoid.
Maybe rotate through namespaces by considering userid + time? Each request from each user would hopefully hit a different namespace every time. Not sure what happens when we approach zero.
I also did a redis thing; generate the csv, import it to redis, deal with it there, but I don't want to maintain an instance and it feels limited.
Fallback: RDS and a micro instance.
Stopgap for today, run directly off the already running Postgres. I hate it, but I need to get away from this for now. I'll figure out namespacing later.
The text was updated successfully, but these errors were encountered:
Have the client download the entire CSV for a particular error on initial load (they're surprisingly small, deadendoneways = 80946 items -> 4mb or we could split them up to some reasonable size), could just put them in a gist
store the status for each item in Parse which sends updates to all other clients
could we use more real-timey stuff like this elsewhere?
More or less figured out using redis for now. The challenging thing was finding a way of randomly selecting items in an efficient matter. Redis sets allows for this. I'm also storing values in redis for now. This will likely change as they can be large. Depending on how many items we items end up having I might have to rethink parts of this as redis is all in memory, but it's fine for ~10M right now and likely good for 5 or 10x.
This has been a mess, some notes:
Stopgap for today, run directly off the already running Postgres. I hate it, but I need to get away from this for now. I'll figure out namespacing later.
The text was updated successfully, but these errors were encountered: