improved recordloader action

[mblakele]

I ran into some problems with the recordloader action. Here's an attempt to improve it. This version is a little more robust about filenames, and allows multiple arguments. Arguments can be property files, input files, or Java VM arguments. Anything starting with '-' is assumed to be a VM arg. Anything else is assumed to be a file, which could be an input file or a property file. The only bit that comes from roxy configuration is the CONNECTION_STRING, so the username and password only have to be stored in one place.

Technically RecordLoader doesn't require any args, but I left a check in place to enforce at least one, along with a note to that effect.

Using this code it becomes fairly easy to create app_specific.rb actions that use recordloader. For example:

def load_testdata()
  logger.info "loading test data..."
  ARGV.push "-Xincgc"
  ARGV.push "src/ingestion/test-data.properties"
  ARGV.push "data/test_data.zip"
  recordloader
end

If this patch is acceptable, something very similar could be done for xqsync.

[dmcassel]

Hi Mike. I like the idea of being able to specify JVM arguments and suchlike, but I also like using property files for the connection string and other common recordloader properties. We can do replacements in those files, so that we have a property like this:

CONNECTION_STRING=xcc://${ml.user}:${ml.password}@${ml.server}:${ml.xcc-port}/${ml.content-db}

Likewise, the INPUT_PATH is a normal recordloader property:

INPUT_PATH=${ml.data.dir}/something

The nice thing about that approach is that you can can a normal recordloader config file and use it as-is, or make use of substitutions like the above to make it more generic.

First let me grant that I probably need to document the substitutions above better.

I'm thinking I'll adjust what you put together to use JVM arguments but depend on the properties file for anything recordloader-specific. Thoughts?

[mblakele]

Substitution sounds good. I'm trying to think of a way to have it both ways, though. I like the DRY aspect of building CONNECTION_STRING from the roxy config, without having to make the properties file roxy-specific. Maybe a flag that tells the recordloader action whether or not to build CONNECTION_STRING? I wouldn't really care which way it defaulted.

[dmcassel]

A thought ... if the properties file has a CONNECTION_STRING, use it; otherwise, build one from the properties.

[mblakele]

Yes, that sounds like a good approach.

[paxtonhare]

I'm going to close this pull request because it's become stale.