What do you want to have speced? That they actually parse these filetypes?

For example, you can do this:

class MyFileFormat < RequestLogAnalyzer::FileFormat::Rails

  # Add some additional lines here besides the standard Rails lines
  # Also, add some trackers to the report.

  # Hack the request class
  class Request < RequestLogAnalyzer::FileFormat::Rails::Request
    # hack the request class here
  end
end

See http://wiki.github.com/wvanbergen/request-log-analyzer/creating-your-own-file-format-definition for more information. This should behave as expected and currently is completely unspeced. We use this at Floorplanner to parse some of the custom lines we add to our log, like the current user and his account type.

So what do you want to spec? The hacks?

• When subclassing a file format, all line types of the parent file format should be parsed, and the newly added line types should be parsed as well.
• The parent file format should be left unchanged (so it should not parse any of the added line types).

The last bullet seems very logical/simple, but I really struggled to get this working correctly when I was implementing fileformat subclassing.

The same points can be made for the report trackers.

rla:report and rla:report:html are allready there.

Separated out config file to separate issue (see #16)

Wouldn't a skip-lines option be more logical?

Maybe that would be a nice feature as well, but bytes is useful to implement resuming, as you can simply store the filename, mtime and filesize of every logfile (which is what the database inserter already does). By skipping these bytes if the file has become bigger since the last parse, you can resume parsing a log that has new information in it.

def read_backwards_to_timestamp(path, timestamp)
  start = nil
  Elif.open(path) do |elif|
    elif.each do |line|
      if line =~ /\AProcessing .+ at (\d+-\d+-\d+ \d+:\d+:\d+)\)/
        time_of_request = Time.parse($1)
        if time_of_request < timestamp
          break
        else
          start = elif.pos
        end
      end
    end
  end

  file = open(path)
  file.seek(start) if start
  file
end

Maybe a binary search through the file is even faster :-)
However, the above implementation is very Rails specific. I don't know if this can be implemented in a file formated agnostic way.

We could try to parse every line into a date :) Or have each file definition define a date_finder teaser regexp?

We should switch to mongodb. High performance Schema-less Database.

I don't think we should switch to MongoDB because it is a rather non-standard piece of technology, but offering it as an alternative is a good idea. Isn't there a MongoDB ActiveRecord mapper we can use?

I think you should read up on Schemaless Databases. They are fundamentally different from relational databases, therefore a MongoDB ActiveRecord adapter whould be pretty useless. Especially if you are exploiting the specific benefits of a schemaless database. Datamapper is trying to build something that can do both, but a swap in is impossible.

My problem is that something like MongoDB is not that well known with most developers. Moreover, why should we choose MongoDB over CouchDB? Therefore, I don't think we should ditch the current RDBMS implementation.

Offering MongoDB as an alternative besides the current DBMSs is a better option IMHO. This is most easily possible if there is an AR adapter for MongoDB.

Couch is for scaling, Mongo is for performance. The reason I think we should switch is that by using a schemaless DB, all our DB problems will go away. Things like indexes and dynamic columns will work out of the box.

Both should work through the datamapper adapter, but mongomapper is more stable.

I could ask Geoffrey from peepcode. He still ows me one for lending my vga adapter at RailsConf Berlin :)