Cache/read local amazon data

[shaiguitar]

Any thoughts on a local amazon cache?

For something like command line applications and accounts with over 1k/2k instances with all resources loaded up, startup time loading basic resources can take a very long time on a low bandwidth connection.

[geemus]

Could be interesting. Did you have thoughts on how we might implement/integrate?

[shaiguitar]

A quick dump/load for fog server collection/instance causes some problems due to fog objects having attrs that are not serializable:

[3] pry(#<AwsConsole>)> s = c.servers.sample; s.id
=> "i-0108b19263edab4b2"
[4] pry(#<AwsConsole>)> x = YAML.dump(s); File.write("foo", x)
=> 64578
[5] pry(#<AwsConsole>)> y = File.read("foo"); YAML.load(y)
TypeError: allocator undefined for Proc
from /Users/shai/.rubies/ruby-2.2.5/lib/ruby/2.2.0/psych/visitors/to_ruby.rb:356:in `allocate'

If I remove the offending proc attr, it will run into some other unserializable data (Digest/other ruby classes). This fails with Marshal as well. Lots of nested objects that probably get in the way. Not to mention other fog objects.

I'm not really sure what the best way to do this would be. Perhaps I'm missing something wrt serialization, and there's some elegant way around this? We could drop down in the stack to adding some replay cache logic at the Excon layer. The json from amazon api is all the same, as long as we could direct fog requests to read from the right place, it could pick up from cache instead of AWS. Would that be a feasible approach? Where would that be/any approaches there that make sense?

Not sure where the best place to hook in caching in the various fog layers would be but for very large aws accounts fog caching would be really great...

[geemus]

I'm not sure how much traction we would get at the request layer or not. That said, I wonder if there might not be some nice, fairly general ways that we could improve serialization across the board. It just isn't something I've needed to do much of directly, so I just haven't spent that much time thinking on it. I feel like if we could figure out something there, it might be easier than full api level caching. What do you think?

[shaiguitar]

So it looks like the part where serialization fails is the service and collection instance variables. If we restore from the simply vanilla attributes, we loose that information, and later interactive api calls fail (ie loaded_from_serialization.reload will bomb out). But if we take care of those, it seems from an initial stab to work:

s = compute.servers.get("i-deadbeef")
File.write("/tmp/server.yml", YAML.dump(s.attributes))

# different session same creds:

service = aws_api.compute
attrs = YAML.load(File.read("/tmp/server.yml"))
x = Fog::Compute::AWS::Server.new(attrs)
x.instance_variable_set(:@service, service)
x.collection = Fog::Compute::AWS::Servers.new({service: service})
x.reload  # this seems to work

Not sure the best way to get/dump/find out the collection & service associated to the resource needed to be required would be. Feedback before going further? Also suggestions on how and the whereabouts in the various fog libs.

[geemus]

Interesting. I guess there might be two parts then, (quite possibly in fog-core?).

1. Offer some serialization/dump thing which omits the shared things (service/collection) if they exist.
2. Make service/collection work in a more lazy-load way? I'm not exactly sure how this part would work though, as you need to have a reference to the service I think. Maybe it's as simple as having a service setter which also sets collection, this could then be used directly (or via some load/undump functionality).

[shaiguitar]

I'm not too familiar with the internals at this level. In terms of robustness and the ease of implementing based on whatever approach taken. @lanej any thoughts on this?

[shaiguitar]

We can move comments to fog/fog-core#204 if that's a better place for this. Basic yml cache dump/load on disk. Would like to use something like this soon.

[geemus]

@shaiguitar sounds good. I'll close this for now. Have onsite/tons of meetings this week, so maybe some delay in responding to the bigger issue, but will try to get to it soon.

[shaiguitar]

@fog if anyone else would like to comment/add feedback on basic cache functionality PR, I'd like to have some cache-usability sooner than later. Thanks!