wvanbergen / request-log-analyzer

Allmost everything now has rdoc

This is planned for the 1.3.0 release. The source (file) is already stored in a request database, but some cleaning up is still necessary.

This has been resolved in version 1.3.2

Can't we just abuse irb for this ?

Maybe, but it would be nice to get some helper functions that make this easier. The database structure requires a lot of JOINs.

Yeah, but you could just spawn an IRB with some method missing magic, right?

Or offer something like Rails, which starts IRB with some libraries that are loaded when starting. You can specify requires on the irb command line.

That's what I meant :)

I added an IRB console to inspect a database that initializes automatically for the upcoming r-l-a release (version 1.3).

Start the console with:
request-log-analyzer console --database <DATABASE> --format <FORMAT>

The following ActiveRecord classes will be available:

• Request
• Source
• Warning
• A class generated for every line type: :access -> AccessLine, :completed => CompletedLine, etc.

You can emulate behavior like this using the UNIX tail command, and piping the result through RLA:

tail -f log/production.log | request-log-analyzer --database log/requests.db -
(don't forget the dash on the end to indicate that you are parsing the standard input)

However, this will create a new database, so it is not possible to "resume" to a previous database file. Adding that feature would be nice indeed. :-)

Continuous parsing will require to check the database schema, to see whether the old structure is compatible with the file that is being parsed.

I have a rails logger here that just logs everything to the database instead of to file. I'll see if it is easily adaptable

I think directly adding request to the database from Rails is outside the scope of RLA. Moreover, I doubt that this approach will work to make it compatible with other file formats.

IMHO, a better approach is to simply parse the logs and append it to the existing database file. However, before we can add data, we need to check whether the database schema is compatible, which depends on the log format specification. Currently, the old database file is simply deleted at the beginning, wiping all old data.

As a bonus, we could detect if the parsed request data is already inserted into the database to avoid duplicate data. This is a little bit more complex on first sight, as requests do not have a unique identifier.

That sounds like a major performance hog

If you mean the last idea: without some kind of a unique index, indeed. However, maybe a unique index using the timestamp, IP address and controller/action of the request might be acceptable.

Another idea: store the filenames (including last modified date) that have been parsed and iserted into the database in a separate table to keep track of what has been parsed and what not.

Yes to your last comment wvanbergen.

Apparently, danlucraft started working on this

Status update:

• A database will not be cleared when parsing starts (unless --reset-database is given as an option).
• Parsed files are now stored in the sources table, including the filesize and mtime at the moment of parsing.

Still to do:

• Check if the database schema still matches the file format
• Allow resuming by skipping the first n bytes of a file.

I have created new issues for the outstanding TODOs, but basically the continuous parsing works now.

Fixed in current master

Danlucraft's fork has implemented this feature, but as of yet it is untested and cannot be merged for that reason.

I have implemented this feature, which will be included in version 1.3. You can provide connection information by passing a connection string to the --database command line option. The connection string can be in two formats:

• ODBC: adapter=sqlite3;database=requests.db
• URI: mysql://username:password@localhost/database_name

The parameter names for the ODBC query should be the same as the ActiveRecord connection parameters.

The --reset-database command line switch makes sure that the database is cleared before the parsing begins.

Of course, passing --database filename.db still works as expected using an SQLite3 database

Do we really want to do this? The idea of linux tools is that they do 1 thing and they do it good. You could just pipe a tar.gz file trough tar and through rla, right? Maybe add an example of this to the examples.

That approach does not work if you want to parse multiple files at once (production.3000.log.gz, production.3001.log.gz, etc.)

Of course, .bz2 and maybe .zip files could be supported as well.

Sombody agrees with me :) : http://snippets.aktagon.com/snippets/245-How-to-unzip-gunzip-untar-files-with-Ruby

Just calling the system util would be pretty acceptable I guess..

Added basic gz, bz2 and zip support to master. Needs specs + checking :).

The progress bar is not working for gzipped files, but as the decompressors just dump a single string as stdout, this is not an easy fix, so I've closed this ticket. Specs are also added for gz tgz zip and bz2. We need release notes :)

Basic YAML reports have been implemented, using the --dump command line switch. The YAML dump contains all the results, not only the top 20 which are displayed in a normal.

Needs RDOC and specs.

Added RDOC and specs

Maybe we can use Pony (http://adam.blog.heroku.com/past/2008/11/2/pony_the_express_way_to_send_email_from_ruby/)? Pony can send using sendmail or smtp. It does require TMail however, so we would add two dependencies for this feature.

If we would simply use (command line) sendmail for sending the email, we can implement it without dependencies. This will not work on Windows however.

I'd rather do this: simpler and more crossplatform:
http://snippets.dzone.com/posts/show/2362

Is localhost usable as SMTP server on every platform?

Let them specify a host if they want to use a different one. Most server environemnts will have either mailing from localhost or something like a mail.internal host.

def send_email(to, message, host = 'localhost')
msg = <<END_OF_MESSAGE
From: #{from_alias} <#{from}>
To: #{to_alias} <#{to}>
Subject: #{subject}

#{message}
END_OF_MESSAGE

  Net::SMTP.start(host) do |smtp|
    smtp.send_message msg, from, to
  end
end

What about Windows? Moreover, most server environments also have sendmail (or a drop-in replacement) installed. Using the sendmail binary, we would not have to add the NET::SMTP dependency.

NET::STMP works on windows, if you specify the SMTP host. And we could just require Net::STMP when needed. Is it part of default ruby? backticking mail seems a bit ugly when replacing just a single command.

a1f8c26..56c91fb master -> master

Nice work (y)

Maybe such an extension can be created as a separate project?

Extension is now available in /lib/rla_tracker :)

Also a wiki page is now available.
Should I move it to a new project called munin-rails-plugins ?

I am for a separate project. The file in question is buried somewhere in the tree, is is not part of the application itself, so nobody will find this kind of functionality.

Moreover, you may want to add some (rake) scripts to ease the installation, which would be out of place in the current r-l-a repo.

True. However, r-l-a is great for this. I have really cool munin graphs :)
As there allready is a rails-munin app, how should I call it? munin-rails-plugins?

See https://github.com/barttenbrinke/munin-plugins-rails/tree

The exact same syntax works for me. What version of the sqlite3-ruby gem do you have installed? Use "gem list sqlite" to see what versions are available. I am using sqlite3-ruby version 1.2.4.

I installed the newly-released request-log-analyzer-1.2.4, and no longer encounter this error. Thanks!

This was fixed by more robust handling of connections for the table migrations.

http://www.rubycentral.com/pickaxe/tut_threads.html

http://stackoverflow.com/questions/559420/how-to-download-fast-lots-of-web-pages-in-ruby-parallelizing-download

Tried parallelizing parse line. The thread construction used about the same time as the consecutive regexp parsing. Either that or my threads are not working :).

file_format.rb

# Parses a line by trying to parse it using every line definition in this file format
def parse_line(line, &warning_handler)
  request_data = nil

  threadpool = []
  request_data = nil

  count = 0
  self.line_definitions.each do |lt, definition|
    count = count + 1
    threadpool << Thread.new(lt) do |lt|
      match = definition.matches(line, &warning_handler)
      request_data = match if match
    end
  end

  threadpool.each { |t| t.join }

  return request_data
end

Wow, just removed a significant bug in parse_line. RLA is now about 3 times faster :)

I tried to parse multiple lines in parallel. It works fine, but it does not yield any performance gains because of the lousy threading implementation in Ruby. It does increase performance when using JRuby though, but JRuby is way slower to begin with :-)

Because of these findings, parallelization will not be implemented in the master branch, but I saved experimental code in the multithreaded branch if someone is interested.

Parked :D

Another idea: forking instead of threads. This should work better on UNIX platforms.

Issues to keep track of: the amount of processes / processor cores ratio, etc.

After talking with Eloy I think this should be parked until MacRuby, Ruby 1.9 or JRuby because they have basic threading support. You could take a look at the rvm gem. http://rvm.beginrescueend.com/

With we, you mean you :)?

I always use the royal plural. ;)

Excuse me your highness :P

OK, we have now fixed this high-priority issue on master!

Nice work :)

This would be nice as well for reading a configuration from a config file (see issue #16)

This may require rewriting the command line option parsing and handling. In that case, we may switch to optparse, the standard ruby library for this.

Command line now uses Base API.

# TODO:
#   Check if defaults work (Aggregator defaults seem wrong).
#   Refactor :database => options[:database], :dump => options[:dump] away from contoller intialization.

Good enough for now

Available in HEAD

This functionality should first work with the database inserter, afterwards it can be improved to work using the --dump output as well.

The switch is probably going to be called --report-amount.

I will also add a command line option --report-sort mean,sum,stddev to specify what tables should be included in the report and how these tables should be sorted.

• --report-sort mean,stddev,sum: Sort results by mean, sum and standard deviation. This will display 3 different tables.
• --report-amount 20: Only show the top 20 results for every sorting/table.
• --report-amount all: Show all the results for every sorting/table.

These command line switches are now available in HEAD and will show up in the next release.

Standard deviation calculation has been added.

Hmmm, it seems to be working fine for me. Can you maybe send (the first part of) your log file to me? You can send it to willem@railsdoctors.com

It worked after I cleared the logs. Too late! ;) I'll do that if I get the same error again. Thanks.

OK, maybe there was some strange cruft left in the old log files that r-l-a couldn't handle. Happy to hear it works for you again!

How can you test that easily? Spawn a mail server with something like pony and then wrap around?

IMHO, the most important thing to test is that the --mail command line argument actually invokes the mailer component. The actual SMTP stuff is less important. Maybe we can check if the appropriate Net::SMTP method is called?

Can we set up some oneliner SMTP server in ruby to catch the email?

How does rails test actionmailer?

This looks interesting: http://snippets.dzone.com/posts/show/3932

Apparently Mailtrap is designed specifically for this: http://matt.blogs.it/entries/00002655.html http://rubymatt.rubyforge.org/mailtrap/

Wow. that should be damn simple:
Remove mailtrap log
mailtrap start
request-log-analyzer start
mailtrap stop
Check if mail was received in mailtrap log

But it will add a requirement to the test suite.. Which should be pretty ok I Guess

afd95ca..5b2d042 via mailtrap

I fixed this issue. Please install version 1.5.1.

What do you want to have speced? That they actually parse these filetypes?

For example, you can do this:

class MyFileFormat < RequestLogAnalyzer::FileFormat::Rails

  # Add some additional lines here besides the standard Rails lines
  # Also, add some trackers to the report.

  # Hack the request class
  class Request < RequestLogAnalyzer::FileFormat::Rails::Request
    # hack the request class here
  end
end

See http://wiki.github.com/wvanbergen/request-log-analyzer/creating-your-own-file-format-definition for more information. This should behave as expected and currently is completely unspeced. We use this at Floorplanner to parse some of the custom lines we add to our log, like the current user and his account type.

So what do you want to spec? The hacks?

• When subclassing a file format, all line types of the parent file format should be parsed, and the newly added line types should be parsed as well.
• The parent file format should be left unchanged (so it should not parse any of the added line types).

The last bullet seems very logical/simple, but I really struggled to get this working correctly when I was implementing fileformat subclassing.

The same points can be made for the report trackers.

rla:report and rla:report:html are allready there.

Separated out config file to separate issue (see #16)

Wouldn't a skip-lines option be more logical?

Maybe that would be a nice feature as well, but bytes is useful to implement resuming, as you can simply store the filename, mtime and filesize of every logfile (which is what the database inserter already does). By skipping these bytes if the file has become bigger since the last parse, you can resume parsing a log that has new information in it.

def read_backwards_to_timestamp(path, timestamp)
  start = nil
  Elif.open(path) do |elif|
    elif.each do |line|
      if line =~ /\AProcessing .+ at (\d+-\d+-\d+ \d+:\d+:\d+)\)/
        time_of_request = Time.parse($1)
        if time_of_request < timestamp
          break
        else
          start = elif.pos
        end
      end
    end
  end

  file = open(path)
  file.seek(start) if start
  file
end

Maybe a binary search through the file is even faster :-)
However, the above implementation is very Rails specific. I don't know if this can be implemented in a file formated agnostic way.

We could try to parse every line into a date :) Or have each file definition define a date_finder teaser regexp?

We should switch to mongodb. High performance Schema-less Database.

I don't think we should switch to MongoDB because it is a rather non-standard piece of technology, but offering it as an alternative is a good idea. Isn't there a MongoDB ActiveRecord mapper we can use?

I think you should read up on Schemaless Databases. They are fundamentally different from relational databases, therefore a MongoDB ActiveRecord adapter whould be pretty useless. Especially if you are exploiting the specific benefits of a schemaless database. Datamapper is trying to build something that can do both, but a swap in is impossible.

My problem is that something like MongoDB is not that well known with most developers. Moreover, why should we choose MongoDB over CouchDB? Therefore, I don't think we should ditch the current RDBMS implementation.

Offering MongoDB as an alternative besides the current DBMSs is a better option IMHO. This is most easily possible if there is an AR adapter for MongoDB.

Couch is for scaling, Mongo is for performance. The reason I think we should switch is that by using a schemaless DB, all our DB problems will go away. Things like indexes and dynamic columns will work out of the box.

Both should work through the datamapper adapter, but mongomapper is more stable.

I could ask Geoffrey from peepcode. He still ows me one for lending my vga adapter at RailsConf Berlin :)