Add Rackspace Cloud Files input. #772
Conversation
|
@jordansissel Anything I need to do on this one? |
|
Brad, We’re going to have a "merge party” soon and we’ll either merge it or get back to you. Meanwhile, have you signed our CLA? http://www.elasticsearch.org/contributor-agreement/ Thanks! —Aaron On Nov 19, 2013, at 10:40 AM, Brad Gignac notifications@github.com wrote:
|
|
if using fog - why make this plugin rackspace specific? why not make parameters to Fog::Storage.new() configurable so that it could be used with s3 and other fog well supported cloud storage providers that share same cloud file manipulation semantics in fog? |
|
@untergeek Wow! Thanks for the quick response. I just signed the CLA. |
|
@piavlo Good question. The biggest issue with Fog is that the parameters you pass to instantiate a new storage provider are different for each provider, so the number of configuration options would be insane. It might be easier to use and document if there was a base Fog Storage input that could be extended with provider-specific options. I'd love more feedback for the logstash community here though. |
|
I'd use it with a local Ceph cluster (S3 API) if I could configure it to do so. |
|
@stuart-warren Awesome! Right now, this assumes that all log files are GZip'ed text files that can be split on the line. Does this assumption work for you Ceph cluster? |
|
I guess they would likely be standard rolled logfiles so yeah. |
|
@bradgignac - yes a base could storage input would be a good way, so that specific providers inputs would only need to build the parameters hash for Fog::Storage.new() Also I wonder if it's possible/makes sense to have the process_log_files() logic implemented as codec? |
|
|
||
| def process_file(queue, file) | ||
| log_stream = StringIO.new(file.body) | ||
| reader = Zlib::GzipReader.new(log_stream) |
There was a problem hiding this comment.
The lines codec will already split a byte stream up into events.
I also think it might be a good idea to omit the gzip assumptions here. This should be a codec as well so all can benefit regardless of if its compressed or not.
Thoughts?
There was a problem hiding this comment.
@nickethier Sorry for taking so long to get back to you. I'm about to push a change that uses the line codec to read the inflated GZip stream rather than defaulting to the plain codec.
I had planned to extract the GZip assumption to a codec but couldn't find much documentation on codecs. My first attempt was simply a wrapper around the line codec that inflated the GZip stream and passed it off the line codec. However, I wasn't quite sure what the implementation of the encode method should look like. Any suggestions?
The Rackspace Cloud Files provides the ability to pull container and CDN access logs from a Cloud Files container.
|
Can one of the admins verify this patch? |
jordansissel
force-pushed
the
master
branch
from
f86287d
to
a536eef
Compare
|
Hi, this been here for a while now. How do we feel about having this new plugin living/move to logstash-plugins org? makes sense to move the new plugin there? |
|
@bradgignac can you please move this to logstash-plugins repo? See an example here: https://www.elastic.co/guide/en/logstash/current/_how_to_write_a_logstash_output_plugin.html We will not be able to merge this repo in the current form as plugins code have been extracted out of core |
The Rackspace Cloud Files provides the ability to pull container access logs and CDN access logs from a Cloud Files container.