Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

improved recordloader action #68

Closed
wants to merge 1 commit into from

3 participants

@mblakele

I ran into some problems with the recordloader action. Here's an attempt to improve it. This version is a little more robust about filenames, and allows multiple arguments. Arguments can be property files, input files, or Java VM arguments. Anything starting with '-' is assumed to be a VM arg. Anything else is assumed to be a file, which could be an input file or a property file. The only bit that comes from roxy configuration is the CONNECTION_STRING, so the username and password only have to be stored in one place.

Technically RecordLoader doesn't require any args, but I left a check in place to enforce at least one, along with a note to that effect.

Using this code it becomes fairly easy to create app_specific.rb actions that use recordloader. For example:

def load_testdata()
  logger.info "loading test data..."
  ARGV.push "-Xincgc"
  ARGV.push "src/ingestion/test-data.properties"
  ARGV.push "data/test_data.zip"
  recordloader
end

If this patch is acceptable, something very similar could be done for xqsync.

@dmcassel
Owner

Hi Mike. I like the idea of being able to specify JVM arguments and suchlike, but I also like using property files for the connection string and other common recordloader properties. We can do replacements in those files, so that we have a property like this:

CONNECTION_STRING=xcc://${ml.user}:${ml.password}@${ml.server}:${ml.xcc-port}/${ml.content-db}

Likewise, the INPUT_PATH is a normal recordloader property:

INPUT_PATH=${ml.data.dir}/something

The nice thing about that approach is that you can can a normal recordloader config file and use it as-is, or make use of substitutions like the above to make it more generic.

First let me grant that I probably need to document the substitutions above better.

I'm thinking I'll adjust what you put together to use JVM arguments but depend on the properties file for anything recordloader-specific. Thoughts?

@mblakele

Substitution sounds good. I'm trying to think of a way to have it both ways, though. I like the DRY aspect of building CONNECTION_STRING from the roxy config, without having to make the properties file roxy-specific. Maybe a flag that tells the recordloader action whether or not to build CONNECTION_STRING? I wouldn't really care which way it defaulted.

@dmcassel
Owner

A thought ... if the properties file has a CONNECTION_STRING, use it; otherwise, build one from the properties.

@dmcassel dmcassel was assigned
@mblakele

Yes, that sounds like a good approach.

@paxtonhare
Owner

I'm going to close this pull request because it's become stale.

@paxtonhare paxtonhare closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Dec 12, 2012
  1. @mblakele

    improved recordloader action

    mblakele authored
This page is out of date. Refresh to see the latest.
Showing with 37 additions and 15 deletions.
  1. +37 −15 deploy/lib/server_config.rb
View
52 deploy/lib/server_config.rb
@@ -509,24 +509,46 @@ def is_backup_complete(job)
end
def recordloader
- filename = ARGV.shift
- raise HelpException.new("recordloader", "configfile is required!") unless filename
- properties_file = File.expand_path("../../#{filename}", __FILE__)
- properties = ServerConfig.load_properties(properties_file, "")
- properties = ServerConfig.substitute_properties(properties, @properties, "")
-
- properties.each do |k, v|
- logger.debug "#{k}=#{v}"
- end
-
- prop_string = ""
- properties.each do |k,v|
- prop_string << %Q{-D#{k}="#{v}" }
+ # technically this might not be required - RL can read from STDIN
+ raise HelpException.new("recordloader",
+ "supply property file(s) or input file(s)!") unless ARGV.length > 0
+
+ # build the vmargs and file portions of the command-line
+ # this allows the user to pass in vmargs like -Xincgc or -Xmx1024m
+ # or -Dfoo=bar as well as any number of property files
+ file_args = ""
+ vm_args = ""
+ while ARGV.length > 0
+ a = ARGV.shift
+ if a.start_with?("-")
+ vm_args << " #{a}"
+ else
+ # allow for shell-significant characters - probably not 100% robust
+ file_args << " '"
+ file_args << File.expand_path("../../../#{a}", __FILE__)
+ file_args << "'"
+ end
end
- runme = %Q{java -cp #{File.expand_path("../java/recordloader.jar", __FILE__)}#{path_separator}#{File.expand_path("../java/marklogic-xcc-5.0.2.jar", __FILE__)}#{path_separator}#{File.expand_path("../java/xpp3-1.1.4c.jar", __FILE__)} #{prop_string} com.marklogic.ps.RecordLoader}
+ user = @properties["ml.user"]
+ password = @properties["ml.password"]
+ host = @properties["ml.local-server"]
+ port = @properties["ml.xcc-port"]
+ # TODO might be nice to substitute other databases as desired
+ db = @properties["ml.content-db"]
+ cs = %Q{-DCONNECTION_STRING=xcc://#{user}:#{password}@#{host}:#{port}/#{db}}
+ logger.info cs
+
+ cp = File.expand_path("../java/recordloader.jar", __FILE__)
+ cp << path_separator
+ cp << File.expand_path("../java/marklogic-xcc-5.0.2.jar", __FILE__)
+ cp << path_separator
+ cp << File.expand_path("../java/xpp3-1.1.4c.jar", __FILE__)
+
+ runme = %Q{java -cp #{cp} #{vm_args} #{cs} \
+ com.marklogic.ps.RecordLoader #{file_args}}
logger.info runme
- `#{runme}`
+ system runme
end
def xqsync
Something went wrong with that request. Please try again.