Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

LogParser can accept directories, now #126

Merged
merged 2 commits into from

2 participants

@spikegrobstein

If it encounters a directory in the argument list, it will recursively
go through the files in it and parse them. (no tests for this use-case)

I couldn't figure out how to test this in such a way that fits in with the rest of your test suite, but some quick tests looked like it worked.

I had to process through tens of thousands of S3 logs and didn't have the time or processing power to combine them. This change saved me HOURS.

@spikegrobstein spikegrobstein LogParser can accept directories, now
If it encounters a directory in the argument list, it will recursively
go through the files in it and parse them. (no tests for this use-case)
6279f7a
@barttenbrinke barttenbrinke was assigned
@barttenbrinke
Collaborator

Hey Spike,
thats very nice addition! Thank you for contributing.
But as you said, it does need a test.
You could test it quite simply by adding a subdirectory to the fixtures directory named something like "log_directory" that contains two or three s3 sample files containing two or three lines each.
The test itself can be added to the integration/command_line_usage_spec, something like:

output = run("#{log_directory_fixture} --format s3")
output.any? { |line| /^Parsed requests:\s*12\s/ =~ line }.should be_true

You can add the log_directory_fixture function in helpers.rb and have it point to the created directory.

Greetings, Bart

@spikegrobstein

Cool. Working on it. Thanks for the pointers. This is a lot easier than what I was trying to get working before.

I actually came across a couple of issues with my implementation while writing the tests so I'm working on fixing those and will update the pull request when I get that working.

@spikegrobstein spikegrobstein full support for directories as logfile parameters
 - fixed a bug where it would throw an exception after finishing
   processing a directory.
 - working tests
b34a539
@spikegrobstein

This last commit should tie up all the loose ends.

Let me know if there's anything else I should do. Thanks!

@barttenbrinke barttenbrinke merged commit 3a5f0f7 into from
@barttenbrinke
Collaborator

Hi Spike,

very nice work! Thanks for your contribution!

Greetings, Bart

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Oct 4, 2012
  1. @spikegrobstein

    LogParser can accept directories, now

    spikegrobstein authored
    If it encounters a directory in the argument list, it will recursively
    go through the files in it and parse them. (no tests for this use-case)
Commits on Oct 6, 2012
  1. @spikegrobstein

    full support for directories as logfile parameters

    spikegrobstein authored
     - fixed a bug where it would throw an exception after finishing
       processing a directory.
     - working tests
This page is out of date. Refresh to see the latest.
View
5 lib/request_log_analyzer/source/log_parser.rb
@@ -118,6 +118,11 @@ def decompress_file?(filename)
# <tt>options</tt>:: A Hash of options that will be pased to parse_io.
def parse_file(file, options = {}, &block)
+ if File.directory?(file)
+ parse_files(Dir["#{ file }/*"], options, &block)
+ return
+ end
+
@current_source = File.expand_path(file)
@source_changes_handler.call(:started, @current_source) if @source_changes_handler
View
5 spec/fixtures/s3_logs/2012-10-05-16-18-11-F9AAC5D1A55AEBAD
@@ -0,0 +1,5 @@
+300a66dd319290e36b67767f862a7f39072caa70ba33b8d01c4b79ddd8e6d7ef lamestuff.com [05/Oct/2012:15:24:46 +0000] 174.44.160.70 - CE74FF983317B326 REST.GET.OBJECT public/portfolio/22/thumbnail.jpg "GET /lamestuff.com/public/portfolio/22/thumbnail.jpg HTTP/1.1" 200 - 9515 9515 42 41 "http://spike.grobste.in/portfolio" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" -
+300a66dd319290e36b67767f862a7f39072caa70ba33b8d01c4b79ddd8e6d7ef lamestuff.com [05/Oct/2012:15:25:20 +0000] 174.44.160.70 - D484E1F8E6DE0AAB REST.GET.OBJECT public/projects/46/resume.png "GET /lamestuff.com/public/projects/46/resume.png HTTP/1.1" 200 - 510856 510856 85 74 "http://spike.grobste.in/portfolio/show?name=lamestuff.com-%282007%29" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" -
+300a66dd319290e36b67767f862a7f39072caa70ba33b8d01c4b79ddd8e6d7ef lamestuff.com [05/Oct/2012:15:25:25 +0000] 174.44.160.70 - E5C48446A01539FC REST.GET.OBJECT public/projects/3/gameboy.jpg "GET /lamestuff.com/public/projects/3/gameboy.jpg HTTP/1.1" 200 - 37458 37458 57 55 "http://spike.grobste.in/project/classic-console-art" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" -
+300a66dd319290e36b67767f862a7f39072caa70ba33b8d01c4b79ddd8e6d7ef lamestuff.com [05/Oct/2012:15:25:27 +0000] 174.44.160.70 - CD6CC6713DE2254A REST.GET.OBJECT public/projects/115/xmas-card-2006-d.jpg "GET /lamestuff.com/public/projects/115/xmas-card-2006-d.jpg HTTP/1.1" 200 - 89892 89892 58 55 "http://spike.grobste.in/portfolio/show?name=xmas-2006" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" -
+300a66dd319290e36b67767f862a7f39072caa70ba33b8d01c4b79ddd8e6d7ef lamestuff.com [05/Oct/2012:15:25:31 +0000] 174.44.160.70 - B8B9CBDE2EFE51F3 REST.GET.OBJECT public/projects/34/Tim_Horton-Window-Coffee-ba.jpg "GET /lamestuff.com/public/projects/34/Tim_Horton-Window-Coffee-ba.jpg HTTP/1.1" 200 - 27376 27376 41 40 "http://spike.grobste.in/project/tim-horton's-at-the-nyse" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" -
View
3  spec/fixtures/s3_logs/2012-10-05-16-26-06-15314AF7F0651839
@@ -0,0 +1,3 @@
+300a66dd319290e36b67767f862a7f39072caa70ba33b8d01c4b79ddd8e6d7ef lamestuff.com [05/Oct/2012:15:24:46 +0000] 174.44.160.70 - FC6B8CCFF3510D92 REST.GET.OBJECT public/portfolio/29/thumbnail.jpg "GET /lamestuff.com/public/portfolio/29/thumbnail.jpg HTTP/1.1" 200 - 6240 6240 40 39 "http://spike.grobste.in/portfolio" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" -
+300a66dd319290e36b67767f862a7f39072caa70ba33b8d01c4b79ddd8e6d7ef lamestuff.com [05/Oct/2012:15:25:27 +0000] 174.44.160.70 - 6908D7E8249F8AB0 REST.GET.OBJECT public/projects/114/xmas-card-2006-c.jpg "GET /lamestuff.com/public/projects/114/xmas-card-2006-c.jpg HTTP/1.1" 200 - 99874 99874 51 47 "http://spike.grobste.in/portfolio/show?name=xmas-2006" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" -
+300a66dd319290e36b67767f862a7f39072caa70ba33b8d01c4b79ddd8e6d7ef lamestuff.com [05/Oct/2012:15:25:30 +0000] 174.44.160.70 - 41804BB9A626C674 REST.GET.OBJECT public/projects/28/Tim_Horton_Counter__3x12_.jpg "GET /lamestuff.com/public/projects/28/Tim_Horton_Counter__3x12_.jpg HTTP/1.1" 200 - 22256 22256 93 91 "http://spike.grobste.in/project/tim-horton's-at-the-nyse" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" -
View
5 spec/integration/command_line_usage_spec.rb
@@ -87,4 +87,9 @@
output = run("--format rails - < #{log_fixture(:rails_1x)}")
output.any? { |line| /^Parsed requests\:\s*4\s/ =~ line }.should be_true
end
+
+ it "should accept a directory as a commandline option" do
+ output = run("#{log_directory_fixture("s3_logs")} --format amazon_s3")
+ output.any? { |line| /^Parsed requests:\s*8\s/ =~ line }.should be_true
+ end
end
View
11 spec/lib/helpers.rb
@@ -10,6 +10,11 @@ def log_fixture(name, extention = "log")
File.dirname(__FILE__) + "/../fixtures/#{name}.#{extention}"
end
+ # directory of logs
+ def log_directory_fixture(name)
+ File.dirname(__FILE__) + "/../fixtures/#{name}"
+ end
+
# Creates a log file given some lines
def log_snippet(*lines)
StringIO.new(lines.join("\n") << "\n")
@@ -49,12 +54,12 @@ def cleanup_temp_files!
def temp_output_file(file_type)
File.expand_path("#{File.dirname(__FILE__)}/../../tmp/spec.#{file_type}.tmp")
end
-
+
# Check if a given string can be found in the given file
# Returns the line number if found, nil otherwise
def find_string_in_file(string, file, options = {})
return nil unless File.exists?(file)
-
+
line_counter = 0
File.open( file ) do |io|
@@ -66,7 +71,7 @@ def find_string_in_file(string, file, options = {})
return line_counter if line.include? string
}
end
-
+
return nil
end
end
Something went wrong with that request. Please try again.