diff --git a/org.eclipse.ice.commands/README.md b/org.eclipse.ice.commands/README.md index b0ec5428d..a2ecf2519 100644 --- a/org.eclipse.ice.commands/README.md +++ b/org.eclipse.ice.commands/README.md @@ -1,6 +1,6 @@ # Commands Package -This README serves as an overview of the commands package, which is a standalone maven package that can be used within or outside of ICE. The package provides the necessary API to set up and run jobs on either one's local computer or a remote host. Additionally, the API includes file transfer and file system browsing capabilities, with the option to move or copy files on the local host or remote host. It is suggested that users encode their file processing logic into a bash/python/powershell script to be run locally/remotely. +This README serves as an overview of the commands package, which is a standalone maven package that can be used within or outside of ICE. The package provides the necessary API to set up and run jobs on either one's local computer or a remote host. Additionally, the API includes file transfer and file system browsing capabilities, with the option to move or copy files on the local host or remote host. It is suggested that users encode their file processing logic into a bash/python/powershell script to be run locally/remotely. For example, a remote job on a remote host B could be run from commands on local host A which executes a remote job on remote host C, assuming the script contains the necessary logic to connect remote host B to remote host C. Examples can be found in either the `src/test/java/org/eclipse/ice/tests/commands` directory or in the standalone package within ICE `org/eclipse/ice/demo/commands/`. @@ -34,7 +34,9 @@ hostname Windows users need to put their ssh credentials into the file located at `C:\Users\Adminstrator\ice-remote-creds.txt` in order for the tests to properly function. -The automated tests will then grab the necessary credentials from this file to run the tests. Any valid ssh connection will work. If you still find that the tests fail, ensure that the ssh connection you are using has been logged into before from your host computer such that there is a key fingerprint associated to that host in your `~/.ssh/known_hosts` file. The Commands package requires that this key exists in order for authentication to proceed, no matter what form of authentication you use. Alternatively, you can set `StrictHostKeyChecking` to false in the `ConnectionManager`, which is not advised as it is inherently unsecure. To do this for the static `ConnectionManager`, just write: +The automated tests will then grab the necessary credentials from this file to run. Any valid ssh connection will work. If you still find that the tests fail, ensure that the ssh connection you are using has been logged into before from your host computer such that there is a key fingerprint associated to that host in your `~/.ssh/known_hosts` file. The Commands package requires that this key exists in order for authentication to proceed, no matter what form of authentication you use. In the event that tests fail on a host that already exists in `known_hosts` (e.g. with the error message `server key did not validate`, try deleting your `known_hosts` file (or the entries in your `known_hosts` that correspond to the host you are trying to run the tests on), logging in again to re-establish a fingerprint, and running the tests again. + +Alternatively, you can set `StrictHostKeyChecking` to false in the `ConnectionManager`, which is in general not advised as it is inherently unsecure. To do this for the static `ConnectionManager`, just write: ```java ConnectionManagerFactory.getConnectionManager().setRequireStrictHostKeyChecking(false); @@ -46,13 +48,15 @@ Note that this is also a way through which ssh validation can be performed in th #### KeyGen Tests and Connections -Connections may be established via a public/private key pair that is generated between the local and remote host. The JSch API only works with RSA keys - Commands can also function with ECDSA, but it is advised to use RSA. You should be sure to generate a key similarly to the following snip of shell code: +Connections may be established via a public/private key pair that is generated between the local and remote host. Commands can function with ECDSA or RSA type keys. To generate an RSA key, for example, use: ```bash $ ssh-keygen -t rsa -m PEM $ ssh-copy-id -i ~/.ssh/keyname.pub username@hostname ``` +Then you should be able to remotely login via `ssh -i /path/to/key username@hostname` without a password requirement. + For the keygen connection tests to pass, you should also create a key to a remote host that the tests expect to find. This can be done with any arbitrary remote server that you have credential access to; however, the key must be named dummyhostkey and must exist in your home `.ssh` directory. In other words, the key must be here: ``` diff --git a/org.eclipse.ice.commands/pom.xml b/org.eclipse.ice.commands/pom.xml index 3e516f5df..c02beb0fb 100644 --- a/org.eclipse.ice.commands/pom.xml +++ b/org.eclipse.ice.commands/pom.xml @@ -25,10 +25,15 @@ log4j 1.2.17 - - com.jcraft - jsch - 0.1.55 + + org.apache.sshd + sshd-core + 2.3.0 + + + org.apache.sshd + sshd-sftp + 2.3.0 - + \ No newline at end of file diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/Command.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/Command.java index eed584270..aa984a08a 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/Command.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/Command.java @@ -161,7 +161,7 @@ public CommandStatus cancel() { * This function is intended to clean up any (possible) remaining loose ends * after the job is finished processing. * - * @return + * @return CommandStatus - indicating that the configuration completed clean up */ protected CommandStatus cleanUp() { @@ -182,7 +182,7 @@ protected CommandStatus cleanUp() { * log/error information. * * @return - CommandStatus indicating that configuration completed and job can - * start running + * start running */ protected CommandStatus setConfiguration() { @@ -330,7 +330,7 @@ protected boolean logOutput(final InputStream output, final InputStream errors) * This function is a simple helper function to check and make sure that the * command status is not set to a flagged error, e.g. failed. * - * @param current_status + * @param current_status to be checked * @return boolean indicating whether or not status is good to continue (true) * or whether or not job has failed (returns false) */ @@ -371,7 +371,7 @@ public void setStatus(CommandStatus status) { * This function returns to the user the configuration that was used to create a * particular command. * - * @return - the particular configuration for this command + * @return - the particular CommandConfiguration for this command */ public CommandConfiguration getCommandConfiguration() { return commandConfig; @@ -380,7 +380,7 @@ public CommandConfiguration getCommandConfiguration() { /** * This function sets the command configuration for a particular command * - * @param config + * @param commandConfig - CommandConfiguration to be set */ public void setCommandConfiguration(CommandConfiguration commandConfig) { this.commandConfig = commandConfig; @@ -400,7 +400,7 @@ public ConnectionConfiguration getConnectionConfiguration() { * This function sets the configuration that is to be used to set up a * particular connection. * - * @param connect + * @param connectionConfig - ConnectionConfiguration to be set */ public void setConnectionConfiguration(ConnectionConfiguration connectionConfig) { this.connectionConfig = connectionConfig; diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/CommandConfiguration.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/CommandConfiguration.java index f36d654ff..22e282b36 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/CommandConfiguration.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/CommandConfiguration.java @@ -73,10 +73,10 @@ public class CommandConfiguration { private HashMap inputFiles = new HashMap(); /** - * This is a list of arguments that the user might want to append to the executable - * name that are _not_ input files. These will not be explicitly checked by the - * file handler for whether or not they exist, as they are presumed to be - * flags/arguments for the job to run. + * This is a list of arguments that the user might want to append to the + * executable name that are _not_ input files. These will not be explicitly + * checked by the file handler for whether or not they exist, as they are + * presumed to be flags/arguments for the job to run. */ private ArrayList argumentList = new ArrayList(); /** @@ -308,16 +308,16 @@ public String getExecutableName() { String separator = "/"; // Add the arguments to the executable name - for(String arg : argumentList) { + for (String arg : argumentList) { fixedExecutableName += " " + arg; } - + // If the input files should be appended, append it if (appendInput) fixedExecutableName += " " + getInputFiles(); fullCommand = fixedExecutableName; - + // Determine the proper separator if (installDirectory != null && installDirectory.contains(":\\")) separator = "\\"; @@ -326,8 +326,6 @@ public String getExecutableName() { if (installDirectory != null && !installDirectory.endsWith(separator)) installDirectory = installDirectory + separator; - - // Search for and replace the ${inputFile} to properly configure the input file for (Map.Entry entry : inputFiles.entrySet()) { if (fixedExecutableName.contains("${" + entry.getKey() + "}") && !appendInput) @@ -374,7 +372,7 @@ public String getExecutableName() { * Setter for CommandId, see * {@link org.eclipse.ice.commands.CommandConfiguration#commandId} * - * @param _commandId + * @param commandId - integer of command ID to be run */ public void setCommandId(int commandId) { this.commandId = commandId; @@ -385,7 +383,7 @@ public void setCommandId(int commandId) { * Getter for CommandId, see * {@link org.eclipse.ice.commands.CommandConfiguration#commandId} * - * @return commandId + * @return commandId - CommandConfiguration ID */ public int getCommandId() { return commandId; @@ -395,7 +393,7 @@ public int getCommandId() { * Setter for executable, see * {@link org.eclipse.ice.commands.CommandConfiguration#executable} * - * @param exec + * @param executable - executable to be run */ public void setExecutable(String executable) { this.executable = executable; @@ -406,17 +404,18 @@ public void setExecutable(String executable) { * Getter for executable, see * {@link org.eclipse.ice.commands.CommandConfiguration#executable} * - * @return executable + * @return executable - executable to be run */ public String getExecutable() { return executable; } /** - * Setter for inputFile, see + * Adder for inputFile, see * {@link org.eclipse.ice.commands.CommandConfiguration#inputFile} * - * @param input + * @param name - name of inputFile to add + * @param path - path to inputFile relative to workingDirectory */ public void addInputFile(String name, String path) { inputFiles.put(name, path); @@ -427,7 +426,7 @@ public void addInputFile(String name, String path) { * Getter for a concatenated string of inputFiles, see * {@link org.eclipse.ice.commands.CommandConfiguration#inputFiles} * - * @return inputFile + * @return String - a string of all the inputFiles concatenated */ public String getInputFiles() { String files = ""; @@ -441,7 +440,7 @@ public String getInputFiles() { * Getter for the inputFile hashmap itself, see * {@link org.eclipse.ice.commands.CommandConfiguration#inputFiles} * - * @return inputFile + * @return inputFile - Hashmap with input file list */ public HashMap getInputFileList() { return inputFiles; @@ -451,7 +450,7 @@ public HashMap getInputFileList() { * Setter for stdErrFileName, see * {@link org.eclipse.ice.commands.CommandConfiguration#stdErrFileName} * - * @param errFile + * @param stdErrFileName - name for StdErr file */ public void setErrFileName(String stdErrFileName) { this.stdErrFileName = stdErrFileName; @@ -462,7 +461,7 @@ public void setErrFileName(String stdErrFileName) { * Getter for stdErrFileName, see * {@link org.eclipse.ice.commands.CommandConfiguration#stdErrFileName} * - * @return stdErrFileName + * @return stdErrFileName - name of StdErr file */ public String getErrFileName() { return stdErrFileName; @@ -472,7 +471,7 @@ public String getErrFileName() { * Setter for stdOutFileName, see * {@link org.eclipse.ice.commands.CommandConfiguration#stdOutFileName} * - * @param outFile + * @param outFile - name of StdOut file */ public void setOutFileName(String stdOutFileName) { this.stdOutFileName = stdOutFileName; @@ -483,7 +482,7 @@ public void setOutFileName(String stdOutFileName) { * Getter for stdOutFileName, see * {@link org.eclipse.ice.commands.CommandConfiguration#stdOutFileName} * - * @return stdOutFileName + * @return stdOutFileName - name of StdOut file */ public String getOutFileName() { return stdOutFileName; @@ -493,7 +492,7 @@ public String getOutFileName() { * Setter for numProcs, see * {@link org.eclipse.ice.commands.CommandConfiguration#numProcs} * - * @param procs + * @param numProcs - number of processes */ public void setNumProcs(String numProcs) { this.numProcs = numProcs; @@ -504,7 +503,7 @@ public void setNumProcs(String numProcs) { * Getter for numProcs, see * {@link org.eclipse.ice.commands.CommandConfiguration#numProcs} * - * @return numProcs + * @return numProcs - number of processes */ public String getNumProcs() { return numProcs; @@ -514,7 +513,7 @@ public String getNumProcs() { * Setter for installDirectory, see * {@link org.eclipse.ice.commands.CommandConfiguration#installDirectory} * - * @param installDir + * @param installDir - String corresponding to path of install directory */ public void setInstallDirectory(String installDirectory) { this.installDirectory = installDirectory; @@ -525,7 +524,7 @@ public void setInstallDirectory(String installDirectory) { * Getter for installDirectory, see * {@link org.eclipse.ice.commands.CommandConfiguration#installDirectory} * - * @return installDirectory + * @return installDirectory - path of install directory */ public String getInstallDirectory() { return installDirectory; @@ -535,7 +534,7 @@ public String getInstallDirectory() { * Getter for os, see {@link org.eclipse.ice.commands.CommandConfiguration#os} * Note that this is set to the default of the local OS * - * @return os + * @return os - operating system that Command is running on */ public String getOS() { return os; @@ -545,7 +544,7 @@ public String getOS() { * Setter for operating system, see * {@link org.eclipse.ice.commands.CommandConfiguration#os} * - * @param OS + * @param os - operating system that Command is running on */ public void setOS(String os) { this.os = os; @@ -555,11 +554,18 @@ public void setOS(String os) { * Setter for workingDirectory, see * {@link org.eclipse.ice.commands.CommandConfiguration#workingDirectory} * - * @param workingDir + * @param workingDir - working directory containing input files/scripts */ public void setWorkingDirectory(String workingDirectory) { // Check to see if the directory ends with a separator - String separator = FileSystems.getDefault().getSeparator(); + // Just get the one that corresponds to the string. In the case of + // commands which are running on *nix but starting on windows, + // or vice versa, this can get things mixed up + String separator = "/"; + // Make a special case for windows + if (workingDirectory.contains("\\")) + separator = "\\"; + if (!workingDirectory.endsWith(separator)) workingDirectory += separator; this.workingDirectory = workingDirectory; @@ -570,7 +576,7 @@ public void setWorkingDirectory(String workingDirectory) { * Getter for workingDirectory, see * {@link org.eclipse.ice.commands.CommandConfiguration#workingDirectory} * - * @return workingDirectory + * @return workingDirectory - working directory containing files/scripts */ public String getWorkingDirectory() { return workingDirectory; @@ -580,7 +586,8 @@ public String getWorkingDirectory() { * Setter for appendInput, see * {@link org.eclipse.ice.commands.CommandConfiguration#appendInput} * - * @param _appendInput + * @param appendInput - boolean of whether or not to append input file names to + * executable */ public void setAppendInput(boolean appendInput) { this.appendInput = appendInput; @@ -591,7 +598,8 @@ public void setAppendInput(boolean appendInput) { * Getter for appendInput, see * {@link org.eclipse.ice.commands.CommandConfiguration#appendInput} * - * @return appendInput + * @return appendInput - boolean of whether or not to append input file names to + * executable */ public boolean getAppendInput() { return appendInput; @@ -603,7 +611,7 @@ public boolean getAppendInput() { * setter protected with the intent that only ConnectionConfiguration can modify * this member variable. * - * @param host + * @param hostname - hostname for command to run on */ protected void setHostname(String hostname) { this.hostname = hostname; @@ -614,7 +622,7 @@ protected void setHostname(String hostname) { * Getter for hostname, see * {@link org.eclipse.ice.commands.CommandConfiguration#hostname}. * - * @return + * @return hostname - hostname for command to run on */ public String getHostname() { return hostname; @@ -624,7 +632,7 @@ public String getHostname() { * Setter for stdErr, see * {@link org.eclipse.ice.commands.CommandConfiguration#stdErr} * - * @param writer + * @param stdErr - BufferedWriter to write error output */ public void setStdErr(BufferedWriter stdErr) { this.stdErr = stdErr; @@ -635,7 +643,7 @@ public void setStdErr(BufferedWriter stdErr) { * Getter for stdErr, see * {@link org.eclipse.ice.commands.CommandConfiguration#stdErr} * - * @return stdErr + * @return stdErr - Buffered writer to write error output */ public BufferedWriter getStdErr() { return stdErr; @@ -645,7 +653,7 @@ public BufferedWriter getStdErr() { * Setter for stdOut, see * {@link org.eclipse.ice.commands.CommandConfiguration#stdOut} * - * @param writer + * @param stdOut - BufferedWriter to write log output */ public void setStdOut(BufferedWriter stdOut) { this.stdOut = stdOut; @@ -656,7 +664,7 @@ public void setStdOut(BufferedWriter stdOut) { * Getter for stdOut, see * {@link org.eclipse.ice.commands.CommandConfiguration#stdOut} * - * @return stdOut + * @return stdOut - BufferedWriter to write log output */ public BufferedWriter getStdOut() { return stdOut; @@ -666,7 +674,7 @@ public BufferedWriter getStdOut() { * Setter for stdOutput, see * {@link org.eclipse.ice.commands.CommandConfiguration#stdOutput} * - * @return stdOutput + * @return stdOutput - String for StdOutput name */ public void setStdOutputString(String stdOutput) { this.stdOutput = stdOutput; @@ -676,7 +684,7 @@ public void setStdOutputString(String stdOutput) { * This function adds the String string to the String * {@link org.eclipse.ice.commands.CommandConfiguration#stdOutput} * - * @param string + * @param string - String to concatenate to the stdOutput string */ public void addToStdOutputString(String string) { stdOutput += "\n" + string; @@ -686,7 +694,7 @@ public void addToStdOutputString(String string) { * Getter for stdOutput, see * {@link org.eclipse.ice.commands.CommandConfiguration#stdOutput} * - * @return stdOutput + * @return stdOutput - String of all stdOutput, separated by '\n' */ public String getStdOutputString() { return stdOutput; @@ -694,9 +702,11 @@ public String getStdOutputString() { /** * Getter for splitCommand, see - * {@link org.eclipse.ice.commands.CommandConfiguration#splitCommand} No setter + * {@link org.eclipse.ice.commands.CommandConfiguration#splitCommand}. No setter * since splitCommand is determined by * {@link org.eclipse.ice.commands.CommandConfiguration#getExecutableName()} + * + * @return splitCommand - ArrayList of split commands */ public ArrayList getSplitCommand() { return splitCommand; @@ -708,9 +718,8 @@ public ArrayList getSplitCommand() { * the setter for full command protected so that it can only be accessed within * the package and not by (e.g.) the user * - * @param command + * @param fullComand - String of the full command to be executed * - * */ protected void setFullCommand(String fullCommand) { this.fullCommand = fullCommand; @@ -721,7 +730,7 @@ protected void setFullCommand(String fullCommand) { * Getter for fullCommand, see * {@link org.eclipse.ice.commands.CommandConfiguration#fullCommand} * - * @return fullCommand + * @return fullCommand - String of the full command to be executed */ public String getFullCommand() { return fullCommand; @@ -731,7 +740,7 @@ public String getFullCommand() { * Add to error message string, see * {@link org.eclipse.ice.commands.CommandConfiguration#errMsg} * - * @param + * @param errMsg - String to add to the error message string */ public void addToErrString(String errMsg) { this.errMsg += errMsg; @@ -741,7 +750,7 @@ public void addToErrString(String errMsg) { * Setter for error message string, see * {@link org.eclipse.ice.commands.CommandConfiguration#errMsg} * - * @param + * @param errMsg - String to add to the error message string */ public void setErrString(String errMsg) { this.errMsg = errMsg; @@ -751,7 +760,7 @@ public void setErrString(String errMsg) { * Getter for error message string, see * {@link org.eclipse.ice.commands.CommandConfiguration#errMsg} * - * @return + * @return - String of error message */ public String getErrString() { return errMsg; @@ -761,7 +770,7 @@ public String getErrString() { * Setter for the remote working directory * {@link org.eclipse.ice.commands.ConnectionConfiguration#workingDirectory} * - * @param dir + * @param remoteWorkingDirectory - path for remote working directory */ public void setRemoteWorkingDirectory(String remoteWorkingDirectory) { this.remoteWorkingDirectory = remoteWorkingDirectory; @@ -771,7 +780,7 @@ public void setRemoteWorkingDirectory(String remoteWorkingDirectory) { * Getter for the remote working directory * {@link org.eclipse.ice.commands.ConnectionConfiguration#workingDirectory} * - * @return + * @return remoteWorkingDirectory - path for remote working directory */ public String getRemoteWorkingDirectory() { return remoteWorkingDirectory; @@ -781,7 +790,7 @@ public String getRemoteWorkingDirectory() { * Setter for interpreter * {@link org.eclipse.ice.commands.CommandConfiguration#interpreter} * - * @param interpreter + * @param interpreter - string of the interpreter to be used for this command */ public void setInterpreter(String interpreter) { this.interpreter = interpreter; @@ -791,26 +800,29 @@ public void setInterpreter(String interpreter) { * Getter for interpreter * {@link org.eclipse.ice.commands.CommandConfiguration#interpreter} * - * @return + * @return - interpreter to be used for this command */ public String getInterpreter() { return interpreter; } - + /** - * Getter for argument list {@link org.eclipse.ice.commands.CommandConfiguration#argumentList} - * @return + * Getter for argument list + * {@link org.eclipse.ice.commands.CommandConfiguration#argumentList} + * + * @return - ArrayList of the argument list */ - public ArrayList getArgumentList(){ + public ArrayList getArgumentList() { return argumentList; } /** * Function that adds an argument to the argument list - * @param argument + * + * @param argument - argument to be added */ public void addArgument(String argument) { argumentList.add(argument); } - + } diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/CommandFactory.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/CommandFactory.java index 47bf709e0..e08a497ba 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/CommandFactory.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/CommandFactory.java @@ -48,16 +48,17 @@ public CommandFactory() { * another host to execute a command. See also * {@link org.eclipse.ice.commands.CommandFactory#getCommand(CommandConfiguration, ConnectionConfiguration, ConnectionConfiguration) * - * @param commandConfig - * @param connectConfig - * @return - * @throws IOException + * @param commandConfig - the CommandConfiguration which holds the particular + * details of a given command. + * @param connectionConfig - the ConnectionConfiguration which holds the details + * on the connection (i.e. local vs. remote) + * @return - The command to be processed */ - public Command getCommand(final CommandConfiguration commandConfig, final ConnectionConfiguration connectConfig) + public Command getCommand(final CommandConfiguration commandConfig, final ConnectionConfiguration connectionConfig) throws IOException { // pass null since the connection configuration connects us from the local host // to the remote host, and we don't have an intermediary host - return getCommand(commandConfig, connectConfig, null); + return getCommand(commandConfig, connectionConfig, null); } /** @@ -65,15 +66,15 @@ public Command getCommand(final CommandConfiguration commandConfig, final Connec * Command class {@link org.eclipse.ice.commands.Command}. * * @param commandConfig - the CommandConfiguration which holds the - * particular details of a given command. + * particular details of a given command. * @param ConnectionConfiguration - the ConnectionConfiguration which holds the - * details on the connection (i.e. local vs. - * remote) + * details on the connection (i.e. local vs. + * remote) * @param extraConnection - An additional connection configuration if - * the user wants to "hop" connections, i.e. log - * into one host and then send a job from that - * host to an additional remote host - * @return + * the user wants to "hop" connections, i.e. log + * into one host and then send a job from that + * host to an additional remote host + * @return - The command to be processed * @throws IOException */ public Command getCommand(final CommandConfiguration commandConfig, final ConnectionConfiguration connectionConfig, @@ -109,9 +110,9 @@ public Command getCommand(final CommandConfiguration commandConfig, final Connec * A function to check whether or not the provided hostname by the user in * CommandFactory is a local hostname or remote hostname. * - * @param host - String of the hostname to be checked + * @param host - String of the hostname to be checked * @return boolean - returns true if the hostname matches that of the local - * hostname, false otherwise. + * hostname, false otherwise. */ private boolean isLocal(String host) { diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/CommandStatus.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/CommandStatus.java index e38709660..966656cfb 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/CommandStatus.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/CommandStatus.java @@ -40,9 +40,6 @@ * */ - - - public enum CommandStatus{ SUCCESS, PROCESSING, RUNNING, INFOERROR, FAILED, CANCELED; } diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/Connection.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/Connection.java index 3153352db..aa388fa79 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/Connection.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/Connection.java @@ -15,11 +15,10 @@ import java.io.OutputStream; import java.util.concurrent.atomic.AtomicReference; -import com.jcraft.jsch.Channel; -import com.jcraft.jsch.ChannelExec; -import com.jcraft.jsch.ChannelSftp; -import com.jcraft.jsch.JSch; -import com.jcraft.jsch.Session; +import org.apache.sshd.client.SshClient; +import org.apache.sshd.client.channel.ChannelExec; +import org.apache.sshd.client.session.ClientSession; +import org.apache.sshd.client.subsystem.sftp.SftpClient; /** * This class represents a connection to a remote system. This could be a system @@ -29,32 +28,32 @@ * */ public class Connection { - /** * An AtomicReference to the ConnectionConfiguration from which connection * information can be gathered */ - private AtomicReference configuration = new AtomicReference(null);; + private AtomicReference configuration = new AtomicReference( + null);; /** - * The secure channel provided by com.jcraft.jsch + * The client entry point */ - private AtomicReference jShell = new AtomicReference(null); + private AtomicReference client = new AtomicReference<>(null); /** - * The JShell session + * The session for this connection */ - private AtomicReference session = new AtomicReference(null); + private AtomicReference clientSession = new AtomicReference<>(null); /** - * The ssh channel for the JSch ssh connection to execute over + * The connection to execute over */ - private AtomicReference execChannel = new AtomicReference(null); + private AtomicReference execChannel = new AtomicReference<>(null); /** - * The ssh channel for the JSch ssh connection to perform sftp transfers over + * The sftp client */ - private AtomicReference sftpChannel = new AtomicReference(null); + private AtomicReference sftpClient = new AtomicReference<>(null); /** * The input stream for the JSch ssh connection @@ -76,7 +75,7 @@ public Connection() { * Constructor which actually sets the connection configuration to a passed * argument * - * @param config + * @param config - the configuration to set the connection information */ public Connection(ConnectionConfiguration config) { configuration = new AtomicReference(config); @@ -85,93 +84,102 @@ public Connection(ConnectionConfiguration config) { /** * Get and return the connection configuration * - * @return + * @return - the Connection's ConnectionConfiguration */ public ConnectionConfiguration getConfiguration() { return configuration.get(); } - public void setConfiguration(ConnectionConfiguration configuration) { - this.configuration.set(configuration); - } /** - * Set the JShell session {@link org.eclipse.ice.commands.Connection#jShell} + * Set the configuration, see + * {@link org.eclipse.ice.commands.Connection#configuration} * - * @param jsch + * @param configuration - ConnectionConfiguration to set for this connection */ - public void setJShellSession(JSch jShell) { - this.jShell = new AtomicReference(jShell); + public void setConfiguration(ConnectionConfiguration configuration) { + this.configuration.set(configuration); } /** - * Get the JShellSession {@link org.eclipse.ice.commands.Connection#jShell} + * Getter to return the SshClient entry point in Mina, see + * {@link org.eclipse.ice.commands.Connection#client} * - * @return + * @return - The Connection's SshClient */ - public JSch getJShellSession() { - return jShell.get(); + public SshClient getClient() { + return client.get(); } /** - * Set the execution channel - * {@link org.eclipse.ice.commands.Connection#execChannel} + * Setter for the SshClient entry point in Mina, see + * {@link org.eclipse.ice.commands.Connection#client} * - * @param execChannel + * @param client - The Connection's SshClient */ - public void setExecChannel(Channel execChannel) { - this.execChannel = new AtomicReference((ChannelExec) execChannel); + public void setClient(SshClient client) { + this.client.set(client); } /** * Get the sftp channel {@link org.eclipse.ice.commands.Connection#sftpChannel} * - * @return + * @return - The Connection's sftp client */ - public ChannelSftp getSftpChannel() { - return sftpChannel.get(); + public SftpClient getSftpChannel() { + return sftpClient.get(); } /** * Set the sftp channel {@link org.eclipse.ice.commands.Connection#sftpChannel} * - * @param sftpChannel + * @param sftpChannel - the Connection's sftp client */ - public void setSftpChannel(Channel sftpChannel) { - this.sftpChannel = new AtomicReference((ChannelSftp) sftpChannel); + public void setSftpChannel(SftpClient sftpClient) { + this.sftpClient.set(sftpClient); } /** * Get the execution channel * {@link org.eclipse.ice.commands.Connection#execChannel} * - * @return + * @return - the execution channel for this Connection */ public ChannelExec getExecChannel() { return execChannel.get(); } + /** + * Set the execution channel + * {@link org.eclipse.ice.commands.Connection#execChannel} + * + * @param execChannel - the execution channel for this Connection + */ + public void setExecChannel(ChannelExec execChannel) { + this.execChannel.set(execChannel); + } + /** * Getter for the session associated to this connection * * @return {@link org.eclipse.ice.commands.Connection#session} */ - public Session getSession() { - return session.get(); + public ClientSession getSession() { + return clientSession.get(); } /** * Set the session {@link org.eclipse.ice.commands.Connection#session} * - * @param _session + * @param session - this connection's session */ - public void setSession(Session session) { - this.session = new AtomicReference(session); + public void setSession(ClientSession session) { + clientSession.set(session); } /** * Set the input stream {@link org.eclipse.ice.commands.Connection#inputStream} * - * @param _stream + * @param inputStream - this Connection's input stream for logging */ public void setInputStream(InputStream inputStream) { this.inputStream = inputStream; @@ -180,7 +188,7 @@ public void setInputStream(InputStream inputStream) { /** * Get the input stream {@link org.eclipse.ice.commands.Connection#inputStream} * - * @return + * @return - this Connection's input stream for logging */ public InputStream getInputStream() { return inputStream; @@ -190,7 +198,7 @@ public InputStream getInputStream() { * Set the output stream * {@link org.eclipse.ice.commands.Connection#outputStream} * - * @param _stream + * @param outputStream - this Connection's output stream for logging */ public void setOutputStream(OutputStream outputStream) { this.outputStream = outputStream; @@ -200,7 +208,7 @@ public void setOutputStream(OutputStream outputStream) { * Get the output stream * {@link org.eclipse.ice.commands.Connection#outputStream} * - * @return + * @return outputStream - this Connection's output stream for logging */ public OutputStream getOutputStream() { return outputStream; diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionAuthorizationHandler.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionAuthorizationHandler.java index b3ce0f3e0..07dea7dca 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionAuthorizationHandler.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionAuthorizationHandler.java @@ -43,20 +43,21 @@ public abstract class ConnectionAuthorizationHandler { String hostname = null; /** - * A char array which can hold a password should the user desire. It is not + * A char array which can hold a password should the user desire. It is not * recommended to use this since this stores the password in memory for some * time. Rather, it is recommended to use any subclass of this class other than - * BasicConnectionAuthorizationHandler, which retrieves the password from the user in some way, or use - * the private key functionality for establishing a connection as in + * BasicConnectionAuthorizationHandler, which retrieves the password from the + * user in some way, or use the private key functionality for establishing a + * connection as in * {@link org.eclipse.ice.commands.ConnectionManagerTest#testOpenConnectionKeyPath}. - * Note that in all other implementations of getPassword other than in - * BasicConnectionAuthorizationHandler, the password is not actually - * stored in this char[] and is rather obtained where it is needed to establish - * a connection and then immediately destroyed. Regardless, after the JSch connection - * is established this member variable is set to null in all cases. + * Note that in all other implementations of getPassword other than in + * BasicConnectionAuthorizationHandler, the password is not actually stored in + * this char[] and is rather obtained where it is needed to establish a + * connection and then immediately destroyed. Regardless, after the JSch + * connection is established this member variable is set to null in all cases. */ char[] password = null; - + /** * This function gets a password for the command authentication. The password is * returned in a char array since Strings are immutable, so it is generally ill @@ -69,21 +70,22 @@ public abstract class ConnectionAuthorizationHandler { protected abstract char[] getPassword(); /** - * This function is intended to be a "jack of all trades" function where an option - * can be passed that may be specific to a particular subclass. For example, for - * text file authorization, a path to the text file can be passed as specified - * by the subclass. The purpose of this function is for ease of setting things - * in the factory method. + * This function is intended to be a "jack of all trades" function where an + * option can be passed that may be specific to a particular subclass. For + * example, for text file authorization, a path to the text file can be passed + * as specified by the subclass. The purpose of this function is for ease of + * setting things in the factory method. * - * @param option + * @param option - a multi-purpose option which depends on the subclass' + * implementation */ public abstract void setOption(String option); - + /** * Getter for authorization hostname * {@link org.eclipse.ice.commands.ConnectionAuthorizationHandler#hostname} * - * @return + * @return hostname - hostname for this authorization */ public String getHostname() { return hostname; @@ -93,7 +95,7 @@ public String getHostname() { * Getter for authorization hostname * {@link org.eclipse.ice.commands.ConnectionAuthorizationHandler#username} * - * @return + * @return username - username for this authorization */ public String getUsername() { return username; @@ -103,7 +105,7 @@ public String getUsername() { * Setter for authorization hostname * {@link org.eclipse.ice.commands.ConnectionAuthorizationHandler#hostname} * - * @param + * @param hostname - hostname for this authorization */ public void setHostname(String hostname) { this.hostname = hostname; @@ -113,29 +115,30 @@ public void setHostname(String hostname) { * Setter for authorization hostname * {@link org.eclipse.ice.commands.ConnectionAuthorizationHandler#username} * - * @param + * @param username - username for this authorization */ public void setUsername(String username) { this.username = username; } /** - * Setter for {@link org.eclipse.ice.commands.ConnectionAuthorizationHandler#password} - * Please see the associated comment to this member variable before using this function. - * It is highly recommended that after a connection is established, this function is - * called again to remove the password information from memory. For example: - * ```java - * ConnectionAuthorizationHandler handler = new BasicConnectionAuthorizationHandler(); - * handler.setPassword("password".toCharArray()); - * // Now establish some connection with the ConnectionManager - * (insert code to establish connection) - * // Now erase the password from memory - * handler.setPassword("".toCharArray()); + * Setter for + * {@link org.eclipse.ice.commands.ConnectionAuthorizationHandler#password} + * Please see the associated comment to this member variable before using this + * function. It is highly recommended that after a connection is established, + * this function is called again to remove the password information from memory. + * For example: + * ```java ConnectionAuthorizationHandler handler = new + * BasicConnectionAuthorizationHandler(); + * handler.setPassword("password".toCharArray()); // Now establish some + * connection with the ConnectionManager (insert code to establish connection) + * // Now erase the password from memory handler.setPassword("".toCharArray()); * ``` - * @param password + * + * @param password - password for this authorization */ public void setPassword(char[] password) { this.password = password; } - + } diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionAuthorizationHandlerFactory.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionAuthorizationHandlerFactory.java index 79d36ebcf..9a41356f0 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionAuthorizationHandlerFactory.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionAuthorizationHandlerFactory.java @@ -34,6 +34,11 @@ public class ConnectionAuthorizationHandlerFactory { */ public final Logger logger = LoggerFactory.getLogger(ConnectionAuthorizationHandlerFactory.class); + /** + * A hashmap which contains the entire list of possible authorization handlers. All + * child classes of ConnectionAuthorizationHandler are included. Instances of each + * class can be obtained by requesting the particular name key from the hash map. + */ private HashMap handlerList = new HashMap(); /** @@ -62,8 +67,8 @@ public ConnectionAuthorizationHandlerFactory() { * and thus can be left as null. Useful when a console or local connection * authorization handler is desired * - * @param type - * @return + * @param type - String corresponding to which handlerList type to get + * @return - The ConnectionAuthorizationHandler requested */ public ConnectionAuthorizationHandler getConnectionAuthorizationHandler(String type) { return getConnectionAuthorizationHandler(type, ""); @@ -73,8 +78,8 @@ public ConnectionAuthorizationHandler getConnectionAuthorizationHandler(String t * Factory method to get a particular connection authorization handler. Returns * a handler to authorize remote connections. * - * @param type - type of authorization handler desired - * @return - authorization handler + * @param type - String corresponding to which handlerList type to get + * @return - The ConnectionAuthorizationHandler requested */ public ConnectionAuthorizationHandler getConnectionAuthorizationHandler(String type, String option) { ConnectionAuthorizationHandler auth = null; @@ -101,8 +106,8 @@ public ConnectionAuthorizationHandler getConnectionAuthorizationHandler(String t * the list and then check for it when determining what type of * ConnectionAuthorizationHandler to return. * - * @param type - * @param auth + * @param type - the name for the type of authorization handler + * @param auth - the instance of a new authorization handler to be added to the hashmap */ public void addConnectionAuthorizationHandlerType(String type, ConnectionAuthorizationHandler auth) { handlerList.put(type, auth); diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionConfiguration.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionConfiguration.java index 499799029..3c0b5eef2 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionConfiguration.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionConfiguration.java @@ -56,7 +56,7 @@ public ConnectionConfiguration() { * A getter to access * {@link org.eclipse.ice.commands.ConnectionConfiguration#name} * - * @return - name + * @return - name of the ConnectionConfiguration */ public String getName() { return name; @@ -66,7 +66,7 @@ public String getName() { * A setter to access * {@link org.eclipse.ice.commands.ConnectionConfiguration#name} * - * @return - name + * @return - name of this ConnectionConfiguration */ public void setName(String name) { this.name = name; @@ -77,7 +77,8 @@ public void setName(String name) { * completion * {@link org.eclipse.ice.commands.ConnectionConfiguration#deleteWorkingDirectory} * - * @return + * @return - boolean indicating whether or not remote working direcctory should + * be deleted upon completion */ public boolean deleteWorkingDirectory() { return deleteWorkingDirectory; @@ -88,7 +89,8 @@ public boolean deleteWorkingDirectory() { * completion * {@link org.eclipse.ice.commands.ConnectionConfiguration#deleteWorkingDirectory} * - * @param + * @param deleteWorkingDirectory - boolean indicating whether or not remote working + * directory should be deleted upon completion */ public void deleteWorkingDirectory(boolean deleteWorkingDirectory) { this.deleteWorkingDirectory = deleteWorkingDirectory; @@ -98,7 +100,7 @@ public void deleteWorkingDirectory(boolean deleteWorkingDirectory) { * Setter for the authorization method, if desired. See * {@link org.eclipse.ice.commands.ConnectionConfiguration#authorization} * - * @param authorization + * @param authorization - authorization for this ConnectionConfiguration */ public void setAuthorization(ConnectionAuthorizationHandler authorization) { this.authorization = new AtomicReference(authorization); @@ -108,7 +110,7 @@ public void setAuthorization(ConnectionAuthorizationHandler authorization) { * Getter for the authorization method, if desired. See * {@link org.eclipse.ice.commands.ConnectionConfiguration#authorization} * - * @param authorization + * @param authorization - authorization for this ConnectionConfiguration */ public ConnectionAuthorizationHandler getAuthorization() { return authorization.get(); diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionManager.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionManager.java index 802613b46..67a4af355 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionManager.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionManager.java @@ -11,18 +11,25 @@ *******************************************************************************/ package org.eclipse.ice.commands; +import java.io.IOException; +import java.nio.file.Paths; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; import java.util.HashMap; +import org.apache.sshd.client.SshClient; +import org.apache.sshd.client.keyverifier.DefaultKnownHostsServerKeyVerifier; +import org.apache.sshd.client.keyverifier.RejectAllServerKeyVerifier; +import org.apache.sshd.client.keyverifier.ServerKeyVerifier; +import org.apache.sshd.client.session.ClientSession; +import org.apache.sshd.common.FactoryManager; +import org.apache.sshd.common.SshConstants; +import org.apache.sshd.common.keyprovider.FileKeyPairProvider; +import org.apache.sshd.common.util.net.SshdSocketAddress; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import com.jcraft.jsch.HostKey; -import com.jcraft.jsch.HostKeyRepository; -import com.jcraft.jsch.JSch; -import com.jcraft.jsch.JSchException; - /** * This class manages remote connections, and as such interfaces with all * classes that are associated with remote connections. @@ -31,6 +38,7 @@ * */ public class ConnectionManager { + private static final int DEFAULT_TIMEOUT = 10000; /** * A HashMap of available Connections to the ConnectionManager, organized by the @@ -52,11 +60,11 @@ public class ConnectionManager { /** * This is a list of authorization types for JSch to allow authentication via. - * The default types added automatically are ssh-rsa and ecdsa-sha2-nistp256. + * The default types added automatically are ssh-rsa and ecdsa-sha2-nistp256. * Clients can add additional types should they need to. */ private ArrayList authTypes = new ArrayList(); - + /** * String containing the path to the known hosts directory. Can be set to * something else if the user has a different default known_host @@ -68,9 +76,10 @@ public class ConnectionManager { */ public ConnectionManager() { // If the OS is windows, then change the known hosts to be windows style - if (System.getProperty("os.name").toLowerCase().contains("win")) + if (System.getProperty("os.name").toLowerCase().contains("win")) { knownHosts = System.getProperty("user.home") + "\\.ssh\\known_hosts"; - + } + // Add the default authorization types authTypes.add("ssh-rsa"); authTypes.add("ecdsa-sha2-nistp256"); @@ -84,15 +93,14 @@ public ConnectionManager() { * @param config - ConnectionConfiguration to be used to open connection * @return Connection - returns connection if successful, null otherwise */ - public Connection openConnection(ConnectionConfiguration config) throws JSchException { + public Connection openConnection(ConnectionConfiguration config) throws IOException { // The new connection to be opened Connection newConnection = new Connection(config); - // Create the shell - JSch jsch = new JSch(); + SshClient client = SshClient.setUpDefaultClient(); + client.start(); - jsch.setKnownHosts(knownHosts); - newConnection.setJShellSession(jsch); + newConnection.setClient(client); logger.info("Trying to open the connection"); @@ -104,34 +112,32 @@ public Connection openConnection(ConnectionConfiguration config) throws JSchExce // Try go get and open the new session try { - newConnection.setSession(newConnection.getJShellSession().getSession(username, hostname)); - } catch (JSchException e) { + ClientSession session = client.connect(username, hostname, SshConstants.DEFAULT_PORT) + .verify(DEFAULT_TIMEOUT).getSession(); + newConnection.setSession(session); + } catch (IOException e) { logger.error("Couldn't open session with given username and hostname. Exiting.", e); - throw new JSchException(); + throw e; } - // Authorize the JSch session with a ConnectionAuthorizationHandler - authorizeSession(newConnection); - - // JSch default requests ssh-rsa host checking, but some keys - // request other types. Loop through the available authorization types - // and add them to the session. - for(String type : authTypes) { - newConnection.getSession().setConfig("server_host_key", type); + // Set the delegate to reject all, then set the key verifier to check the strict + // option + ServerKeyVerifier delegate = RejectAllServerKeyVerifier.INSTANCE; + if (knownHosts != null) { + newConnection.getSession().setServerKeyVerifier(new DefaultKnownHostsServerKeyVerifier(delegate, + requireStrictHostKeyChecking, Paths.get(knownHosts))); + } else { + newConnection.getSession().setServerKeyVerifier( + new DefaultKnownHostsServerKeyVerifier(delegate, requireStrictHostKeyChecking)); } - // If the user wants to disable StrictHostKeyChecking, add it to the - // session configuration - if (!requireStrictHostKeyChecking) - newConnection.getSession().setConfig("StrictHostKeyChecking", "no"); - // Connect the session try { - newConnection.getSession().connect(); - } catch (JSchException e) { + // Authorize the JSch session with a ConnectionAuthorizationHandler + authorizeSession(newConnection); + } catch (IOException e) { logger.error("Couldn't connect to session with given username and/or password/key. Exiting.", e); - - throw new JSchException(); + throw e; } // Add the connection to the list since it was successfully created @@ -157,7 +163,7 @@ public Connection openConnection(ConnectionConfiguration config) throws JSchExce * @param connection * @throws JSchException */ - private void authorizeSession(Connection connection) throws JSchException { + private void authorizeSession(Connection connection) throws IOException { // Get the authorization information ConnectionAuthorizationHandler auth = connection.getConfiguration().getAuthorization(); @@ -169,23 +175,107 @@ private void authorizeSession(Connection connection) throws JSchException { char[] pwd = auth.getPassword(); // Pass it to the session - connection.getSession().setPassword(String.valueOf(pwd)); + connection.getSession().addPasswordIdentity(String.valueOf(pwd)); // Erase contents of pwd and fill with null Arrays.fill(pwd, Character.MIN_VALUE); auth.setPassword(null); - + // Set the authentication requirements - connection.getSession().setConfig("PreferredAuthentications", "publickey,password"); + connection.getSession().setUserAuthFactoriesNames("publickey", "password"); } else { logger.info("Trying to authenticate with a key"); // Otherwise try a key authentication String keyPath = ((KeyPathConnectionAuthorizationHandler) auth).getKeyPath(); - connection.getJShellSession().addIdentity(keyPath); + if (keyPath != null) { + FileKeyPairProvider provider = new FileKeyPairProvider() { + @Override + public String toString() { + return FileKeyPairProvider.class.getSimpleName() + "[clientIdentitiesProvider]"; + } + }; + provider.setPaths(Collections.singleton(Paths.get(keyPath))); + connection.getClient().setKeyIdentityProvider(provider); + } } + + connection.getSession().auth().verify(FactoryManager.DEFAULT_AUTH_TIMEOUT); return; } + /** + * This function opens a forwarding connection between one already established + * connection to a remote system and an additional remote system specified by + * the ConnectionConfiguration. This allows ports to be opened up across + * multiple machines, as in: System A --> System B (intermediateConn) --> System + * C (config). So the connection that is returned from this function connects + * System A to System C. + * + * @param intermediateConn - intermediate jump host + * @param config - ConnectionConfiguration for final host destination + * @return - Connection if successfully established, null otherwise + * @throws IOException + */ + public Connection openForwardingConnection(Connection intermediateConn, ConnectionConfiguration config) + throws IOException { + + // First check that the first connection actually is established + if (!isConnectionOpen(intermediateConn.getConfiguration().getName())) { + logger.error("Can't forward a connection with an unopened intermediate connection! Returning null."); + return null; + } + + // Make the forwarded connection + Connection forwardConnection = new Connection(config); + + // Get the intermediate client and session + SshClient intermClient = intermediateConn.getClient(); + ClientSession intermSesh = intermediateConn.getSession(); + + // Check that the intermediate session is actually connected and open + if (!intermSesh.isOpen()) { + logger.error("Can't forward a connecion with an unopened intermediate session! Returning null."); + return null; + } + + // Set the new connection to have the same ssh client + forwardConnection.setClient(intermClient); + + /** + * To make this work, we will set up a port forwarding from the local host to + * the ssh port of the final destination host through the originally opened + * session. Thus, the connection should go from the local host, to the (below) + * assignedPort, and then to port 22 on the final destination host. + */ + + SshdSocketAddress forwardAdd = new SshdSocketAddress(config.getAuthorization().getHostname(), 22); + SshdSocketAddress localAdd = new SshdSocketAddress("localhost",0); + SshdSocketAddress address = intermSesh.startLocalPortForwarding(localAdd,forwardAdd); + + String username = config.getAuthorization().getUsername(); + String hostname = config.getAuthorization().getHostname(); + + // Connect the session to access 127.0.0.1:address.getPort(), which is where the startLocalPortForwarding + // is now matching the remote servers + ClientSession forwardSession = forwardConnection.getClient().connect(username, "127.0.0.1", address.getPort()).verify(DEFAULT_TIMEOUT) + .getSession(); + + forwardConnection.setSession(forwardSession); + + // Authorize the session with password/key/etc. + authorizeSession(forwardConnection); + + // Add the connection to the list since it was successfully created + connectionList.put(forwardConnection.getConfiguration().getName(), forwardConnection); + + logger.info("Connection at " + username + "@" + hostname + " established successfully through " + + intermediateConn.getConfiguration().getAuthorization().getUsername() + "@" + + intermediateConn.getConfiguration().getAuthorization().getHostname()); + + + return forwardConnection; + } + /** * This function finds the particular connection requested by name in the list * of all connections and returns it. @@ -214,18 +304,30 @@ public void closeConnection(String connectionName) { // Check the channels first if (connection.getExecChannel() != null) { - if (connection.getExecChannel().isConnected()) { - connection.getExecChannel().disconnect(); + if (connection.getExecChannel().isOpen()) { + try { + connection.getExecChannel().close(); + } catch (IOException e) { + logger.error(e.getLocalizedMessage()); + } } } if (connection.getSftpChannel() != null) { - if (connection.getSftpChannel().isConnected()) { - connection.getSftpChannel().disconnect(); + if (connection.getSftpChannel().isOpen()) { + try { + connection.getSftpChannel().close(); + } catch (IOException e) { + logger.error(e.getLocalizedMessage()); + } } } // Disconnect the session. If the session was not connected in the first place, // it does nothing - connection.getSession().disconnect(); + try { + connection.getSession().close(); + } catch (IOException e) { + logger.error(e.getLocalizedMessage()); + } // Confirm with the logger logger.debug("Connection " + connectionName + "@" + connection.getConfiguration().getAuthorization().getHostname() + " closed"); @@ -264,16 +366,28 @@ public void closeAllConnections() { // Iterate over all available connections in the list and disconnect for (Connection connection : connectionList.values()) { if (connection.getExecChannel() != null) { - if (connection.getExecChannel().isConnected()) { - connection.getExecChannel().disconnect(); + if (connection.getExecChannel().isOpen()) { + try { + connection.getExecChannel().close(); + } catch (IOException e) { + logger.error(e.getLocalizedMessage()); + } } } if (connection.getSftpChannel() != null) { - if (connection.getSftpChannel().isConnected()) { - connection.getSftpChannel().disconnect(); + if (connection.getSftpChannel().isOpen()) { + try { + connection.getSftpChannel().close(); + } catch (IOException e) { + logger.error(e.getLocalizedMessage()); + } } } - connection.getSession().disconnect(); + try { + connection.getSession().close(); + } catch (IOException e) { + logger.error(e.getLocalizedMessage()); + } } } @@ -328,7 +442,7 @@ public void listAllConnections() { */ public boolean isConnectionOpen(String connectionName) { Connection connection = getConnection(connectionName); - return connection.getSession().isConnected(); + return connection.getSession().isOpen(); } /** @@ -374,6 +488,7 @@ public void setKnownHosts(String knownHosts) { /** * This function allows clients to add an authorization type for which JSch can * authorize with. ssh-rsa and ecdsa-sha2-nistp256 are added by default. + * * @param type */ public void addAuthorizationType(String type) { diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionManagerFactory.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionManagerFactory.java index fd85cd6de..e9a228cdf 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionManagerFactory.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConnectionManagerFactory.java @@ -35,7 +35,7 @@ public ConnectionManagerFactory() { /** * A function to return the static instance of ConnectionManager * - * @return + * @return - static instance of ConnectionManager */ public static ConnectionManager getConnectionManager() { return manager; diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConsoleEraser.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConsoleEraser.java index f2deb76bd..518ff957b 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConsoleEraser.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/ConsoleEraser.java @@ -40,7 +40,7 @@ public void run() { // Wait for the next character try { - Thread.currentThread().sleep(1); + Thread.sleep(1); } catch (InterruptedException e) { break; } diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/FileHandler.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/FileHandler.java index e8bd75437..99e247bc9 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/FileHandler.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/FileHandler.java @@ -55,7 +55,7 @@ public abstract class FileHandler implements IFileHandler { protected CommandStatus transferStatus; /** - * An enun to determine what the actual handle type is to set for the command. + * An enum to determine what the actual handle type is to set for the command. * Default to null so that the RemoteFileHandler tries to determine it on its * own - however, user can set this explicitly. */ @@ -189,8 +189,8 @@ public Command getCommand() { * Function to determine whether or not a given string is located on the local * machine * - * @param file - * @return + * @return - boolean indicating whether or not the file exists locally (true) + * or not (false) */ protected boolean isLocal(String file) { // Get the path @@ -204,8 +204,8 @@ protected boolean isLocal(String file) { * This operation creates all the directories that are parents of the * destination. * - * @param dest the destination for which parent directories should be created - * @return true if the directories were created + * @param dest - the destination for which parent directories should be created + * @return true if the directories were created, false otherwise * @throws IOException thrown if the dest cannot be created */ protected boolean createDirectories(String dest) throws IOException { @@ -231,7 +231,7 @@ protected boolean createDirectories(String dest) throws IOException { * * @param destination - destination for the file to go to * @return - CommandStatus indicating whether or not the transfer completed - * successfully + * successfully * @throws IOException */ protected CommandStatus executeTransfer(final String destination) throws IOException { @@ -251,7 +251,7 @@ protected CommandStatus executeTransfer(final String destination) throws IOExcep * A setter to set the type of file handle this is. See * {@link org.eclipse.ice.commands.RemoteFileHandler#HANDLE_TYPE} * - * @param HANDLE_TYPE + * @param HANDLE_TYPE - HandleType for a given transfer */ public void setHandleType(HandleType HANDLE_TYPE) { this.HANDLE_TYPE = HANDLE_TYPE; @@ -260,7 +260,7 @@ public void setHandleType(HandleType HANDLE_TYPE) { /** * Get the connection for this file handler * - * @return + * @return - Connection corresponding to this file handler */ public Connection getConnection() { return connection.get(); diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/FileHandlerFactory.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/FileHandlerFactory.java index 1c2a1ea21..e2a61b016 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/FileHandlerFactory.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/FileHandlerFactory.java @@ -69,7 +69,7 @@ public FileHandler getFileHandler(ConnectionConfiguration connectionConfig) thro * * @param host - String of the hostname to be checked * @return boolean - returns true if the hostname matches that of the local - * hostname, false otherwise. + * hostname, false otherwise. */ private boolean isLocal(String host) { diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/IFileHandler.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/IFileHandler.java index 20c0b09b1..2cc4c71c1 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/IFileHandler.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/IFileHandler.java @@ -27,7 +27,7 @@ public interface IFileHandler { * path If the operation fails, an exception is thrown * * @return - CommandStatus - a CommandStatus indicating whether or not the move - * was successful + * was successful * @throws IOException */ public abstract CommandStatus move(final String source, final String destination) throws IOException; @@ -37,7 +37,7 @@ public interface IFileHandler { * path If the operation fails, an exception is thrown * * @return - CommandStatus - a CommandStatus indicating whether or not the copy - * was successful + * was successful * @throws IOException */ public abstract CommandStatus copy(final String source, final String destination) throws IOException; @@ -47,9 +47,9 @@ public interface IFileHandler { * already exists for a given path. * * @param - String - a string with the path for the method to check its - * existence + * existence * @return - boolean indicating whether or not the file exists (returns true) or - * does not exist (returns false) + * does not exist (returns false) * @throws IOException */ public abstract boolean exists(final String file) throws IOException; @@ -59,8 +59,8 @@ public interface IFileHandler { * destination doesn't exist, it tries to make it. If the destination can't be * made, or the source doesn't exist, the method throws an exception. * - * @param source - * @param destination + * @param source - source path to check existence of before file transfer + * @param destination - destination path to check existence of before file transfer * @throws IOException */ public abstract void checkExistence(final String source, final String destination) throws IOException; @@ -68,7 +68,7 @@ public interface IFileHandler { /** * Method that returns a FileBrowser instance * - * @return - FileBrowser + * @return - FileBrowser - a FileBrowser instance * @param - Directory to browse */ public abstract FileBrowser getFileBrowser(final String topDirectory); diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/KeyPathConnectionAuthorizationHandler.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/KeyPathConnectionAuthorizationHandler.java index 48262bb73..5672136ea 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/KeyPathConnectionAuthorizationHandler.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/KeyPathConnectionAuthorizationHandler.java @@ -20,6 +20,9 @@ */ public class KeyPathConnectionAuthorizationHandler extends ConnectionAuthorizationHandler { + /** + * A string that holds the path to the key used for a connection authorization + */ private String keyPath = null; /** @@ -28,6 +31,11 @@ public class KeyPathConnectionAuthorizationHandler extends ConnectionAuthorizati public KeyPathConnectionAuthorizationHandler() { } + /** + * Constructor with a given key path + * + * @param keyPath - String corresponding to the path for the key + */ public KeyPathConnectionAuthorizationHandler(String keyPath) { this.keyPath = keyPath; } @@ -36,7 +44,7 @@ public KeyPathConnectionAuthorizationHandler(String keyPath) { * Setter for * {@link org.eclipse.ice.commands.KeyPathConnectionAuthorizationHandler#keyPath} * - * @param keyPath + * @param keyPath - String corresponding to the path for the key */ @Override public void setOption(String option) { @@ -47,11 +55,12 @@ public void setOption(String option) { * Getter for * {@link org.eclipse.ice.commands.KeyPathConnectionAuthorizationHandler#keyPath} * - * @return + * @return - String corresponding to the path for the key */ public String getKeyPath() { return keyPath; } + /** * See * {@link org.eclipse.ice.commands.ConnectionAuthorizationHandler#getPassword()} diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalCommand.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalCommand.java index 35472c1d3..20af2d255 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalCommand.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalCommand.java @@ -51,14 +51,16 @@ public LocalCommand() { * {@link org.eclipse.ice.commands.LocalCommand#setConfiguration(CommandConfiguration)} * for details about what details are required. * - * @param _configuration + * @param _connection - ConnectionConfiguration corresponding to a local + * connection + * @param _configuration - CommandConfiguration for this particular command */ - public LocalCommand(ConnectionConfiguration _connection, CommandConfiguration _configuration) { + public LocalCommand(ConnectionConfiguration connectionConfig, CommandConfiguration commandConfig) { status = CommandStatus.PROCESSING; // Set the configuration for the command - commandConfig = _configuration; + this.commandConfig = commandConfig; // If commandConfig wasn't set properly, the job can't run if (commandConfig == null) @@ -66,12 +68,12 @@ public LocalCommand(ConnectionConfiguration _connection, CommandConfiguration _c // Set the connection for the local command, which is only relevant for // accessing the hostname of the local computer - connectionConfig = _connection; + this.connectionConfig = connectionConfig; // Make sure both commandConfig and connectionConfig have the same // hostname, since commandConfig needs the hostname for several output // files (e.g. for debugging purposes). - commandConfig.setHostname(connectionConfig.getAuthorization().getHostname()); + this.commandConfig.setHostname(connectionConfig.getAuthorization().getHostname()); } @@ -230,9 +232,6 @@ protected CommandStatus processJob() { * finished successfully according to the ProcessBuilder. * * See also {@link org.eclipse.ice.commands.Command#finishJob()} - * - * @return - CommandStatus indicating whether or not the function processed - * correctly */ @Override protected CommandStatus finishJob() { @@ -310,9 +309,9 @@ protected CommandStatus monitorJob() { // finishes. while (exitValue != 0) { // First make sure the job hasn't been canceled - if(status == CommandStatus.CANCELED) + if (status == CommandStatus.CANCELED) return CommandStatus.CANCELED; - + // Try to get the exit value of the job // If the job completed successfully this will be 0 try { diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalConnectionAuthorizationHandler.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalConnectionAuthorizationHandler.java index 0186e06ff..585e90e6b 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalConnectionAuthorizationHandler.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalConnectionAuthorizationHandler.java @@ -63,7 +63,7 @@ protected static String getLocalHostname() { } /** - * No options required for console authorization See + * No options required for console authorization. See * {@link org.eclipse.ice.commands.ConnectionAuthorizationHandler#setOption(String)} */ @Override diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalCopyFileCommand.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalCopyFileCommand.java index eecb9fffd..95ec66f04 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalCopyFileCommand.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalCopyFileCommand.java @@ -49,7 +49,7 @@ public LocalCopyFileCommand() { * copy was completed successfully. It returns a CommandStatus indicating * whether or not the move was successful. * - * @return CommandStatus + * See also {@link org.eclipse.ice.commands.LocalCommand#execute()} */ @Override public CommandStatus execute() { @@ -65,7 +65,7 @@ public CommandStatus execute() { * CommandStatus indicating that the command is currently running and needs to * be checked that it completed correctly. * - * @return CommandStatus + * See also {@link org.eclipse.ice.commands.LocalCommand#run()} */ @Override protected CommandStatus run() { @@ -98,7 +98,7 @@ public void setConfiguration(String src, String dest) { /** * A function that returns the source path in string form * - * @return - String + * @return - String of source path */ public String getSource() { return source.toString(); @@ -107,7 +107,7 @@ public String getSource() { /** * A function that returns the destination path in string form * - * @return - String + * @return - String of destination path */ public String getDestination() { return destination.toString(); diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalFileBrowser.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalFileBrowser.java index a649ecd0b..fe8565c59 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalFileBrowser.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalFileBrowser.java @@ -49,7 +49,9 @@ public LocalFileBrowser(final String topDirectory) { } - // Add the file path to fileList + /** + * Add the file path to the array list fileList + */ @Override public FileVisitResult visitFile(Path file, BasicFileAttributes attr) { // Add the file, depending on it's attribute @@ -60,7 +62,9 @@ public FileVisitResult visitFile(Path file, BasicFileAttributes attr) { return FileVisitResult.CONTINUE; } - // Add the directory path to the array list directoryList + /** + * Add the directory path to the array list directoryList + */ @Override public FileVisitResult postVisitDirectory(Path dir, IOException exc) { directoryList.add(dir.toString()); @@ -81,7 +85,7 @@ public FileVisitResult visitFileFailed(Path file, IOException exc) { /** * See {@link org.eclipse.ice.commands.FileBrowser#getFileList()} * - * @return + * @return = ArrayList of files */ @Override public ArrayList getFileList() { @@ -91,7 +95,7 @@ public ArrayList getFileList() { /** * See {@link org.eclipse.ice.commands.FileBrowser#getDirectoryList()} * - * @return + * @return - ArrayList of directories */ @Override public ArrayList getDirectoryList() { @@ -103,6 +107,8 @@ public ArrayList getDirectoryList() { * browsing capabilities of LocalFileHandler. It returns a LocalFileWalker so * that the files or directories could be obtained * + * @param topDirectory - top most directory to recursively walk through + * */ private void walkTree(final String topDirectory) { @@ -115,7 +121,7 @@ private void walkTree(final String topDirectory) { try { // Make a dummy path that just gets the return top directory from // Files.walkFileTree - Path path = Files.walkFileTree(topPath, this); + Files.walkFileTree(topPath, this); } catch (IOException e) { logger.error("Unable to walk file tree at path " + topPath.toString(), e); } diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalMoveFileCommand.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalMoveFileCommand.java index 71b26add8..a939d7d86 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalMoveFileCommand.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/LocalMoveFileCommand.java @@ -49,7 +49,7 @@ public LocalMoveFileCommand() { * move command was completed successfully. It returns a CommandStatus * indicating whether or not the move was successful. * - * @return CommandStatus + * See also {@link org.eclipse.ice.commands.LocalCommand#execute()} */ @Override public CommandStatus execute() { @@ -63,8 +63,7 @@ public CommandStatus execute() { * CommandStatus indicating that the command is currently running and needs to * be checked that it completed correctly. * - * @return CommandStatus - indicates whether or not the move was successful or - * not + * See also {@link org.eclipse.ice.commands.LocalCommand#run()} */ @Override protected CommandStatus run() { @@ -185,7 +184,7 @@ public void setConfiguration(String src, String dest) { /** * A function that returns the source path in string form * - * @return - String + * @return - String of the source path */ public String getSource() { return source.toString(); @@ -194,7 +193,7 @@ public String getSource() { /** * A function that returns the destination path in string form * - * @return - String + * @return - String of the destination path */ public String getDestination() { return destination.toString(); diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteCommand.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteCommand.java index e82210c78..807d352ba 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteCommand.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteCommand.java @@ -1,5 +1,4 @@ -/** - * /******************************************************************************* +/******************************************************************************* * Copyright (c) 2019- UT-Battelle, LLC. * All rights reserved. This program and the accompanying materials * are made available under the terms of the Eclipse Public License v1.0 @@ -13,18 +12,25 @@ package org.eclipse.ice.commands; import java.io.BufferedOutputStream; -import java.io.FileNotFoundException; +import java.io.BufferedReader; +import java.io.File; import java.io.FileOutputStream; +import java.io.FileReader; import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; import java.util.ArrayList; import java.util.Collection; +import java.util.EnumSet; import java.util.HashMap; import java.util.Map; import java.util.concurrent.atomic.AtomicReference; -import com.jcraft.jsch.ChannelSftp; -import com.jcraft.jsch.JSchException; -import com.jcraft.jsch.SftpException; +import org.apache.sshd.client.channel.ClientChannelEvent; +import org.apache.sshd.client.subsystem.sftp.SftpClient; +import org.apache.sshd.client.subsystem.sftp.SftpClient.DirEntry; +import org.apache.sshd.client.subsystem.sftp.SftpClientFactory; +import org.apache.sshd.common.subsystem.sftp.SftpException; /** * This class inherits from Command and gives available functionality for remote @@ -92,10 +98,10 @@ public RemoteCommand(CommandConfiguration commandConfig, ConnectionConfiguration /** * Opens and sets a connection based on what was passed in the constructor. This * function first checks if a connection with the same name is already available - * in the connection manager, and if so, grabs it. Otherwise, it opens a new - * connection with the provided information. - * Function is public so that if a user wants to reset the connection for a - * particular command, they have the option to. + * in the connection manager, and if so, grabs it. Otherwise, it opens a new + * connection with the provided information. Function is public so that if a + * user wants to reset the connection for a particular command, they have the + * option to. */ public void openAndSetConnection() { // Open and set the connection(s) @@ -103,14 +109,13 @@ public void openAndSetConnection() { if (manager.getConnection(connectionConfig.getName()) == null) { connection.set(manager.openConnection(connectionConfig)); } else { - if(connection.get().getSession() != null && - connection.get().getSession().isConnected()) { + if (connection.get().getSession() != null && connection.get().getSession().isOpen()) { connection.set(manager.getConnection(connectionConfig.getName())); // Make sure the connections are starting fresh from scratch if (connection.get().getExecChannel() != null) - connection.get().getExecChannel().disconnect(); + connection.get().getExecChannel().close(); if (connection.get().getSftpChannel() != null) - connection.get().getSftpChannel().disconnect(); + connection.get().getSftpChannel().close(); } else { connection.set(manager.openConnection(connectionConfig)); } @@ -122,15 +127,14 @@ public void openAndSetConnection() { // If there is an extra connection so that we are multi-hopping, then open it // too - // TODO - the multi-hop API isn't implemented yet - need to work on it ConnectionConfiguration secondConfig = secondConnection.get().getConfiguration(); if (secondConfig != null) { - secondConnection.set(manager.openConnection(secondConfig)); + secondConnection.set(manager.openForwardingConnection(connection.get(),secondConfig)); // Set the commandConfig hostname to be the extra connection, since this is // really where the job will run commandConfig.setHostname(secondConnection.get().getConfiguration().getAuthorization().getHostname()); } - } catch (JSchException e) { + } catch (IOException e) { // If the connection(s) can't be opened, we can't be expected to execute a job // remotely! status = CommandStatus.INFOERROR; @@ -138,7 +142,8 @@ public void openAndSetConnection() { logger.error("Returning info error", e); return; } - + + return; } /** @@ -152,7 +157,7 @@ protected CommandStatus run() { // error try { status = transferFiles(); - } catch (SftpException | JSchException | IOException e) { + } catch (IOException e) { logger.error("File transfer error, could not complete file transfers to remote host. Exiting."); logger.error("Returning info error", e); return CommandStatus.INFOERROR; @@ -187,8 +192,6 @@ protected CommandStatus run() { * then disconnects the remote channel to finish up the job processing. * * See also {@link org.eclipse.ice.commands.Command#finishJob()} - * - * @return CommandStatus */ @Override protected CommandStatus finishJob() { @@ -198,12 +201,11 @@ protected CommandStatus finishJob() { try { // Open an sftp channel so that we can ls the contents of the path - ChannelSftp channel = (ChannelSftp) connection.get().getSession().openChannel("sftp"); - channel.connect(); + SftpClient client = SftpClientFactory.instance().createSftpClient(connection.get().getSession()); // Delete the directory and all of the contents - deleteRemoteDirectory(channel, commandConfig.getRemoteWorkingDirectory()); - channel.disconnect(); - } catch (JSchException | SftpException e) { + deleteRemoteDirectory(client, commandConfig.getRemoteWorkingDirectory()); + client.close(); + } catch (IOException e) { // This exception just needs to be logged, since it is not harmful to // the job processing in any way logger.warn("Unable to delete remote directory tree."); @@ -211,16 +213,58 @@ protected CommandStatus finishJob() { } // Disconnect the channels and return success - connection.get().getExecChannel().disconnect(); - connection.get().getSftpChannel().disconnect(); + try { + connection.get().getExecChannel().close(); + } catch (IOException e) { + logger.error(e.getLocalizedMessage()); + } + try { + connection.get().getSftpChannel().close(); + } catch (IOException e) { + logger.error(e.getLocalizedMessage()); + } // Don't disconnect the session in the event that a user wants to run multiple - // jobs over the same session. Let session management be handled by the connection manager + // jobs over the same session. Let session management be handled by the + // connection manager /** - * Note that output doesn't have to explicitly be logged - JSch takes care of + * Note that output doesn't have to explicitly be logged - Mina takes care of * this for you in {@link org.eclipse.ice.commands.RemoteCommand#loopCommands} + * However we set the strings in CommandConfiguration here since Mina only logs + * the output to the files */ + String stdOutFileName = commandConfig.getOutFileName(); + String stdErrFileName = commandConfig.getErrFileName(); + + File outfile = new File(stdOutFileName); + File errfile = new File(stdErrFileName); + try { + FileReader outreader = new FileReader(outfile); + BufferedReader outbr = new BufferedReader(outreader); + String line; + // Read in each line + while ((line = outbr.readLine()) != null) { + // Commented lines by definition begin with #, so skip these + if (!line.startsWith("#")) + commandConfig.addToStdOutputString(line); + } + // Close the reader + outbr.close(); + + // Repeat the same process for the error file + FileReader errreader = new FileReader(errfile); + BufferedReader errbr = new BufferedReader(errreader); + line = null; + while ((line = errbr.readLine()) != null) { + if (!line.startsWith("#")) + commandConfig.addToStdOutputString(line); + } + errbr.close(); + + } catch (IOException e) { + logger.error(e.getLocalizedMessage()); + } return CommandStatus.SUCCESS; } @@ -233,33 +277,39 @@ protected CommandStatus monitorJob() { // Poll until the command is complete. If it isn't finished, give it a second // to try and finish up - while (exitValue != 0) { + Collection waitMask = connection.get().getExecChannel() + .waitFor(EnumSet.of(ClientChannelEvent.CLOSED), 1000L); + + // Iterate over the channel being not closed. Only a enum of CLOSED + // indicates that the job is finished processing + while (!waitMask.contains(ClientChannelEvent.CLOSED)) { // First make sure the job hasn't been canceled - if(status == CommandStatus.CANCELED) + if (status == CommandStatus.CANCELED) return CommandStatus.CANCELED; + + // Wait for a new set of return messages from Mina + waitMask = connection.get().getExecChannel() + .waitFor(EnumSet.of(ClientChannelEvent.CLOSED), 1000L); - try { - // Give it a second to finish up - Thread.currentThread().sleep(1000); - } catch (InterruptedException e) { + // If the set contains timeout, then wait another iteration. + // Job is still trying to finish + if (waitMask.contains(ClientChannelEvent.TIMEOUT)) { // Just log this exception, see if thread can wait next iteration - logger.error("Thread couldn't wait for another second while monitoring job...", e); - } - // Query the exit status. 0 is normal completion, everything else is abnormal - exitValue = connection.get().getExecChannel().getExitStatus(); - - // If the connection was closed and the job didn't finish, something bad - // happened... - if (connection.get().getExecChannel().isClosed() && exitValue != 0) { - logger.error("Connection is closed with exit value " + exitValue + ", failed "); - return CommandStatus.FAILED; + logger.error("Thread timed out while monitoring job, waiting..."); + continue; } } + // Query the exit status. 0 is normal completion, everything else is abnormal + exitValue = connection.get().getExecChannel().getExitStatus(); + + // If the job returns anything other than 0, then the job failed. Otherwise // success - if (exitValue != 0) + if (exitValue != 0) { + logger.error("Job finished with abnormal exit status of: " + exitValue); return CommandStatus.FAILED; + } return CommandStatus.SUCCESS; } @@ -293,32 +343,22 @@ protected CommandStatus processJob() { // Now loop over all commands and run them via JSch for (int i = 0; i < completeCommands.size(); i++) { - // Open the channel for the executable to be run on + // Give the command to the channel connection + String thisCommand = completeCommands.get(i); + try { - connection.get().setExecChannel(connection.get().getSession().openChannel("exec")); - } catch (JSchException e) { - logger.error("Execution channel could not be opened over JSch... Returning failed.", e); + connection.get().setExecChannel(connection.get().getSession().createExecChannel(thisCommand)); + } catch (IOException e) { + logger.error("Execution channel could not be opened... Returning failed.", e); // If it can't be opened, fail return CommandStatus.FAILED; } - // Give the command to the channel connection - String thisCommand = completeCommands.get(i); - connection.get().getExecChannel().setCommand(thisCommand); - logger.info("Executing command: " + thisCommand + " remotely in the working directory " + commandConfig.getRemoteWorkingDirectory()); - // Set up the input stream - connection.get().getExecChannel().setInputStream(null); - try { - // Set the input stream for the connection object - connection.get().setInputStream(connection.get().getExecChannel().getInputStream()); - } catch (IOException e) { - logger.error("Input stream could not be set in JSch... Returning failed.", e); - // If we can't set the input stream, fail - return CommandStatus.FAILED; - } + connection.get().getExecChannel().setIn(null); + connection.get().setInputStream(connection.get().getExecChannel().getIn()); // Setup the output streams and pass them to the connection channel try { @@ -326,39 +366,91 @@ protected CommandStatus processJob() { stdOutStream = new FileOutputStream(commandConfig.getOutFileName(), true); BufferedOutputStream stdOutBufferedStream = new BufferedOutputStream(stdOutStream); // Give the streams to the channel now that their names are appropriately set - connection.get().getExecChannel().setOutputStream(stdOutBufferedStream); - connection.get().getExecChannel().setErrStream(stdErrStream); - } catch (FileNotFoundException e) { - logger.error("Logging streams could not be set in JSch... Returning failed.", e); - // If logging streams can't be set, return failed since we won't be - // able to see if job was successful or not + connection.get().getExecChannel().setOut(stdOutBufferedStream); + connection.get().getExecChannel().setErr(stdErrStream); + } catch (IOException e) { + logger.error(e.getLocalizedMessage()); return CommandStatus.FAILED; } - // Make sure the channel is connected try { - //logger.info("Session is connected: " + connection.get().getSession().isConnected()); - //logger.info("Channel is connected: " + connection.get().getExecChannel().isConnected()); - //logger.info("Channel is closed: " + connection.get().getExecChannel().isClosed()); - - // Connect and run the executable - connection.get().getExecChannel().connect(); - - // Log the output and error streams - logOutput(connection.get().getExecChannel().getInputStream(), - connection.get().getExecChannel().getErrStream()); - } catch (JSchException e) { - logger.error("Couldn't connect the channel to run the executable! Returning failed.", e); - return CommandStatus.FAILED; + connection.get().getExecChannel().open().verify(); } catch (IOException e) { - logger.error("Couldn't log the output! Returning failed.", e); - return CommandStatus.FAILED; + e.printStackTrace(); } } return CommandStatus.RUNNING; } + /** + * This function is responsible for setting up the file transfer logic, + * depending on whether or not this is a regular remote command or a jump host + * command. It is called from run and executes the logic in + * {@link org.eclipse.ice.commands.RemoteCommand#transferFiles(ConnectionConfiguration, String, String)} + * + * @return - CommandStatus indicating whether or not the transfer(s) were + * successful + * @throws IOException + * @throws SftpException + * @throws JSchException + */ + protected CommandStatus transferFiles() throws IOException { + /** + * If we have a jump host connection then we need to transfer the files from the + * intermediary host to the local host first, and then transfer them from the + * local host through the forwarded connection. In other words, the file + * transferring goes from (in the case of System A --> System B --> System C + * where B is the jump host and C is the destination host): + * + * System B --> System A - transfer the files to a local temp directory, System A + * --> System C - transfer the files from the local temp directory through the + * forwarded connection + */ + // So first we will transfer from the jump host + if (secondConnection.get().getConfiguration() != null) { + // Create a temporary local directory to move files from the jump host to + // the local host + Path tempLocalDir = Files.createTempDirectory("tempJumpDir"); + + status = transferFiles(connectionConfig, commandConfig.getWorkingDirectory(), tempLocalDir.toString()); + // Check that this transfer completed successfully before moving to + // transfer from A --> C + if (!checkStatus(status)) + return CommandStatus.FAILED; + // Set the working directory to be the temporary local directory that + // was created, so that the rest of the command operates as normal + // from a local host to a remote host with this new temporary directory + status = transferFiles(secondConnection.get().getConfiguration(), + tempLocalDir.toString(), + commandConfig.getRemoteWorkingDirectory()); + + // Delete the local temp file once finished moving to the destination host + File localTempFile = new File(tempLocalDir.toString()); + boolean delete = deleteLocalDirectory(localTempFile); + if (!delete) { + logger.warn("Temporary directory at " + tempLocalDir.toString() + " couldn't be completely deleted."); + } + /** + * In order for the rest of the code base to use the forwarded connection as the + * job processing connection, we need to set the main connection of the class, + * i.e. {@link org.eclipse.ice.commands.Command#connection} as the forwarded + * connection + */ + connection = secondConnection; + + } else { + // If this is not a jump host case, then just move the files from the local + // working directory to the remote host like normal + status = transferFiles(connectionConfig, commandConfig.getWorkingDirectory(), + commandConfig.getRemoteWorkingDirectory()); + } + + return status; + } + + + /** * This function is responsible for transferring the files to the remote host. * It utilizes the file handling API in this package @@ -368,16 +460,21 @@ protected CommandStatus processJob() { * @throws JSchException * @throws IOException */ - protected CommandStatus transferFiles() throws SftpException, JSchException, IOException { + protected CommandStatus transferFiles(ConnectionConfiguration config, String sourceDir, String destDir) throws IOException { + String sourceSep = "/"; + String destSep = "/"; + + // Figure out what the separators are, depending on windows/*nix + if(sourceDir.contains("\\")) + sourceSep = "\\"; + if(destDir.contains("\\")) + destSep = "\\"; + // Set up a remote file handler to transfer the files RemoteFileHandler handler = new RemoteFileHandler(); // Give the handler the same connection as this command - handler.setConnectionConfiguration(connectionConfig); - - // Get the directories where files live/should go - String remoteWorkingDirectory = commandConfig.getRemoteWorkingDirectory(); - String workingDirectory = commandConfig.getWorkingDirectory(); + handler.setConnectionConfiguration(config); // Get the executable to concatenate String shortExecName = commandConfig.getExecutable(); @@ -388,14 +485,21 @@ protected CommandStatus transferFiles() throws SftpException, JSchException, IOE else if (shortExecName.contains("\\")) shortExecName = shortExecName.substring(shortExecName.lastIndexOf("\\") + 1); + // Check that the source directory ends with the right separator + if(!sourceDir.endsWith(sourceSep)) + sourceDir += sourceSep; + // Do the same for the destination - if (!remoteWorkingDirectory.endsWith("/")) - remoteWorkingDirectory += "/"; + if (!destDir.endsWith(destSep)) + destDir += destSep; // Build the source and destination paths - String source = workingDirectory + shortExecName; - String destination = remoteWorkingDirectory + shortExecName; + String source = sourceDir + shortExecName; + String destination = destDir + shortExecName; + logger.info("Trying to transfer " + source + " to " + destination + + " over connection " + config.getName()); + CommandStatus fileTransfer = null; // Check if the source file exists. If the executable is a script, then it will // transfer it to the host. If it is just a command (e.g. ls) then it will skip @@ -431,11 +535,10 @@ else if (shortInputName.contains("\\")) // Now have the full paths, so transfer the files per the logger messages // Put the inputfile to the remote directory. Use a null object for receiving - // notifications about - // the progress of the transfer and use 0 to overwrite the files if they exist - // there already - source = workingDirectory + shortInputName; - destination = remoteWorkingDirectory + shortInputName; + // notifications about the progress of the transfer and use 0 to overwrite + // the files if they exist there already + source = sourceDir + shortInputName; + destination = destDir + shortInputName; fileTransfer = handler.copy(source, destination); if (fileTransfer != CommandStatus.SUCCESS) { logger.error("Couldn't transfer " + source + " to remote host!"); @@ -456,16 +559,13 @@ else if (shortInputName.contains("\\")) * @param path - top directory to delete * @throws SftpException */ - private void deleteRemoteDirectory(ChannelSftp sftpChannel, String path) throws SftpException { - - // Get the path's directory structure - Collection fileList = sftpChannel.ls(path); + private void deleteRemoteDirectory(SftpClient sftpChannel, String path) throws IOException { // Iterate through the list to get the file/directory names - for (ChannelSftp.LsEntry file : fileList) { + for (DirEntry file : sftpChannel.readDir(path)) { // If it isn't a directory delete it - if (!file.getAttrs().isDir()) { - sftpChannel.rm(path + "/" + file.getFilename()); + if (!file.getAttributes().isDirectory()) { + sftpChannel.remove(path + "/" + file.getFilename()); } else if (!(".".equals(file.getFilename()) || "..".equals(file.getFilename()))) { // If it is a subdir. // Otherwise its a subdirectory, so try deleting it try { @@ -481,6 +581,27 @@ private void deleteRemoteDirectory(ChannelSftp sftpChannel, String path) throws sftpChannel.rmdir(path); // delete the parent directory after empty } + /** + * Recursive function that deletes a local directory and its contents + * + * @param directory - directory to be deleted + * @return boolean - true if directory was deleted, false otherwise + */ + private boolean deleteLocalDirectory(File directory) { + // Get the file list of the directory + File[] contents = directory.listFiles(); + // If there are files/subdirectories in the directory, recursively iterate over + // them to + // delete them so that the directory can be deleted + if (contents != null) { + for (File file : contents) { + deleteLocalDirectory(file); + } + } + // Return whether or not the directory was deleted + return directory.delete(); + } + /** * Set a particular connection for a particular RemoteCommand * diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteCopyFileCommand.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteCopyFileCommand.java index 5121bfe9a..ab33976cc 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteCopyFileCommand.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteCopyFileCommand.java @@ -88,7 +88,7 @@ protected CommandStatus run() { /** * Get the source file string * - * @return + * @return - String of the source path */ public String getSource() { return source; @@ -97,7 +97,7 @@ public String getSource() { /** * Get the destination file string * - * @return + * @return - String of the destination path */ public String getDestination() { return destination; @@ -106,7 +106,7 @@ public String getDestination() { /** * Set the copy type variable * - * @param type + * @param copyType - HandleType for this file transfer */ public void setCopyType(HandleType copyType) { this.copyType = copyType; @@ -115,7 +115,7 @@ public void setCopyType(HandleType copyType) { /** * Set the permissions for a chmod during file transfer * - * @param permissions + * @param permissions - file permissions to be set upon file transfer */ protected void setPermissions(int permissions) { this.permissions = permissions; diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteFileBrowser.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteFileBrowser.java index 8413447ad..897aa26a6 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteFileBrowser.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteFileBrowser.java @@ -11,12 +11,11 @@ *******************************************************************************/ package org.eclipse.ice.commands; -import java.io.File; +import java.io.IOException; import java.util.ArrayList; -import java.util.Collection; -import com.jcraft.jsch.ChannelSftp; -import com.jcraft.jsch.SftpException; +import org.apache.sshd.client.subsystem.sftp.SftpClient; +import org.apache.sshd.client.subsystem.sftp.SftpClient.DirEntry; /** * This class allows for remote file and directory browsing on a remote host @@ -25,30 +24,26 @@ * */ public class RemoteFileBrowser implements FileBrowser { - /** - * A connection with which to browse files - */ - private Connection connection; - - /** - * Default constructor + * Default constructor */ public RemoteFileBrowser() { + // Clear the arrays to start fresh for every browser fileList.clear(); directoryList.clear(); - this.connection = null; } - + /** * Default constructor with connection and top directory name + * + * @param connection - Connection over which to file browse + * @param topDirectory - top most directory to recursively walk through */ public RemoteFileBrowser(Connection connection, final String topDirectory) { // Make sure we start with a fresh list every time the browser is called fileList.clear(); directoryList.clear(); - this.connection = connection; - + // Fill the arrays with the relevant file information fillArrays(topDirectory, connection.getSftpChannel()); } @@ -58,28 +53,25 @@ public RemoteFileBrowser(Connection connection, final String topDirectory) { * remote host and fill the member variable arrays with the files and * directories * - * @param channel - * @param topDirectory + * @param channel - sftp channel to walk the file tree with + * @param topDirectory - top most directory under which to browse */ - protected void fillArrays(String topDirectory, ChannelSftp channel) { - + protected void fillArrays(String topDirectory, SftpClient channel) { try { // Make sure the top directory ends with the appropriate separator String separator = "/"; - // If the remote file system returns a home directory with \, then it - // must be windows - if (channel.getHome().contains("\\")) + // If the remote directory to search contains a \\, then it must be windows + // and thus we should set the separator as such + if (topDirectory.contains("\\")) separator = "\\"; // Now check the path name if (!topDirectory.endsWith(separator)) topDirectory += separator; - // Get the path's directory structure - Collection directoryStructure = channel.ls(topDirectory); // Iterate through the structure - for (ChannelSftp.LsEntry file : directoryStructure) { - if (!file.getAttrs().isDir()) { + for (DirEntry file : channel.readDir(topDirectory)) { + if (!file.getAttributes().isDirectory()) { fileList.add(topDirectory + file.getFilename()); // Else if it is a subdirectory and not '.' or '..' } else if (!(".".equals(file.getFilename()) || "..".equals(file.getFilename()))) { @@ -91,7 +83,7 @@ protected void fillArrays(String topDirectory, ChannelSftp channel) { } - } catch (SftpException e) { + } catch (IOException e) { logger.error("Could not use channel to connect to browse directories!", e); } @@ -99,8 +91,6 @@ protected void fillArrays(String topDirectory, ChannelSftp channel) { /** * See {@link org.eclipse.ice.commands.FileBrowser#getDirectoryList()} - * - * @return */ @Override public ArrayList getDirectoryList() { @@ -109,8 +99,6 @@ public ArrayList getDirectoryList() { /** * See {@link org.eclipse.ice.commands.FileBrowser#getFileList()} - * - * @return */ @Override public ArrayList getFileList() { diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteFileHandler.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteFileHandler.java index 22a95098d..403927099 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteFileHandler.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteFileHandler.java @@ -13,10 +13,8 @@ import java.io.IOException; -import com.jcraft.jsch.ChannelSftp; -import com.jcraft.jsch.JSchException; -import com.jcraft.jsch.SftpATTRS; -import com.jcraft.jsch.SftpException; +import org.apache.sshd.client.subsystem.sftp.SftpClient; +import org.apache.sshd.client.subsystem.sftp.SftpClientFactory; /** * This class inherits from FileHandler and handles the processing of remote @@ -27,8 +25,6 @@ * */ - - public class RemoteFileHandler extends FileHandler { /** @@ -50,7 +46,7 @@ public RemoteFileHandler() { * checks if the connection is already open, and if it is not it calls the * connection manager to open the connection * - * @param config + * @param config - ConnectionConfiguration for file transferring */ public void setConnectionConfiguration(ConnectionConfiguration config) { // Get the connection manager and open the connection in constructor so that it @@ -63,7 +59,7 @@ public void setConnectionConfiguration(ConnectionConfiguration config) { try { logger.info("Manager is opening a connection"); connection.set(manager.openConnection(config)); - } catch (JSchException e) { + } catch (IOException e) { logger.error("Connection could not be established. Transfer will fail.", e); } } else { @@ -73,10 +69,9 @@ public void setConnectionConfiguration(ConnectionConfiguration config) { // Open an sftp channel for this remote file handler to use try { // Set it for the connection - connection.get().setSftpChannel(connection.get().getSession().openChannel("sftp")); - connection.get().getSftpChannel().connect(); - - } catch (JSchException e) { + SftpClientFactory factory = SftpClientFactory.instance(); + connection.get().setSftpChannel(factory.createSftpClient(connection.get().getSession())); + } catch (IOException e) { logger.error( "Connection seems to have an unopened channel, but there was a failure when trying to open the channel.", e); @@ -89,19 +84,16 @@ public void setConnectionConfiguration(ConnectionConfiguration config) { @Override public boolean exists(String file) throws IOException { - ChannelSftp sftpChannel = null; try { // Get the sftp channel to check existence - sftpChannel = connection.get().getSftpChannel(); - + SftpClient sftpChannel = connection.get().getSftpChannel(); // Try to lstat the path. If an exception is thrown, it means it does not exist - SftpATTRS attrs = sftpChannel.lstat(file); - } catch (SftpException e) { + sftpChannel.lstat(file); + } catch (IOException e) { if (isLocal(file)) { // If the file can be found locally, return true since we found it. // Up to checkExistence to determine what kind of move this is (e.g. - // local->remote - // or vice versa) + // local->remote or vice versa) return true; } else { return false; @@ -124,11 +116,10 @@ public boolean exists(String file) throws IOException { */ private boolean makeRemoteDirectory(String file) { logger.warn("Path doesn't exist on the remote host, trying to make it."); - ChannelSftp sftpChannel = null; try { // Get the sftp channel to check existence - sftpChannel = connection.get().getSftpChannel(); + SftpClient sftpChannel = connection.get().getSftpChannel(); // Try to make the directory on the remote host // Could be many directories, so we need to iterate over each piece @@ -138,17 +129,15 @@ private boolean makeRemoteDirectory(String file) { for (int i = 0; i < directories.length; i++) { // Add the next directory to the full path name directory += "/" + directories[i]; - // Try to ls the directory. If it throws an exception, then - // it doesn't exist, so we should make it try { - SftpATTRS attrs = sftpChannel.lstat(directory); - } catch (SftpException e) { + sftpChannel.lstat(directory); + } catch (IOException e) { sftpChannel.mkdir(directory); } } logger.info("Made new remote directory"); - } catch (SftpException e) { + } catch (IOException e) { logger.error("Couldn't make nonexistent remote directory, exiting.", e); return false; } @@ -164,6 +153,9 @@ private boolean makeRemoteDirectory(String file) { * determines what kind of remote move it is, i.e. which direction the move is * going (local --> remote, remote --> local, remote --> remote, etc.) * + * Alternatively, the move can be set by the client using + * {@link org.eclipse.ice.commands.RemoteFileHandler#setHandleType(HandleType)} + * * @throws JSchException */ @Override @@ -186,19 +178,19 @@ public void checkExistence(String source, String destination) throws IOException // If the user set the handle type explicitly, check their existence // and return if they are confirmed to exist - if(HANDLE_TYPE != null) { - if(HANDLE_TYPE == HandleType.localRemote) { - if(isLocal(source) && exists(destination)) + if (HANDLE_TYPE != null) { + if (HANDLE_TYPE == HandleType.localRemote) { + if (isLocal(source) && exists(destination)) return; } else if (HANDLE_TYPE == HandleType.remoteLocal) { - if(isLocal(destination) && exists(source)) + if (isLocal(destination) && exists(source)) return; } else { - if(exists(destination) && exists(source)) + if (exists(destination) && exists(source)) return; } } - + // Otherwise try and figure out what kind of file transfer is // If the source is local, then try a local --> remote handle if (isLocal(source)) { @@ -260,8 +252,8 @@ public void checkExistence(String source, String destination) throws IOException } // Print out the determined handle type for informational purposes - logger.info( - "FileHandler is moving/copying " + source + " to " + destination + " with the handle type " + HANDLE_TYPE); + logger.info("FileHandler is moving/copying " + source + " to " + destination + " with the handle type " + + HANDLE_TYPE); } @@ -317,7 +309,8 @@ protected void configureCopyCommand(String source, String destination) { * normal chmod, and changes it to decimal. * * - * @param permissions + * @param permissions - String of the permissions to set. Taken as a string + * so that it can be converted to decimal from octal */ public void setPermissions(String permissions) { this.permissions = Integer.parseInt(permissions, 8); diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteMoveFileCommand.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteMoveFileCommand.java index 7dcc6e8cf..0a04d610c 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteMoveFileCommand.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteMoveFileCommand.java @@ -89,7 +89,7 @@ protected CommandStatus run() { /** * Get the source file string * - * @return + * @return - string of the source path */ public String getSource() { return source; @@ -98,7 +98,7 @@ public String getSource() { /** * Get the destination file string * - * @return + * @return - string of the destination path */ public String getDestination() { return destination; @@ -107,7 +107,7 @@ public String getDestination() { /** * Set the move type variable * - * @param type + * @param moveType - HandleType for this file transfer */ public void setMoveType(HandleType moveType) { this.moveType = moveType; @@ -116,7 +116,7 @@ public void setMoveType(HandleType moveType) { /** * Set the permissions for a chmod during file transfer * - * @param permissions + * @param permissions - file permissions for new file upon transfer */ protected void setPermissions(int permissions) { this.permissions = permissions; diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteTransferExecution.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteTransferExecution.java index 73e5faa0a..f15b6e882 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteTransferExecution.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/RemoteTransferExecution.java @@ -13,14 +13,22 @@ package org.eclipse.ice.commands; +import java.io.FileInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.nio.file.FileSystems; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.apache.sshd.client.subsystem.sftp.SftpClient; +import org.apache.sshd.client.subsystem.sftp.SftpClient.Attributes; +import org.apache.sshd.client.subsystem.sftp.SftpClient.OpenMode; +import org.apache.sshd.common.subsystem.sftp.SftpConstants; +import org.apache.sshd.common.subsystem.sftp.SftpException; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import com.jcraft.jsch.ChannelExec; -import com.jcraft.jsch.ChannelSftp; -import com.jcraft.jsch.JSchException; -import com.jcraft.jsch.SftpException; - /** * This class performs the remote transfer execution for a remote copy or move * command It is called from RemoteCopyFileCommand or RemoteMoveFileCommand, @@ -52,17 +60,27 @@ public class RemoteTransferExecution { public RemoteTransferExecution() { } + /** + * This function actually performs the logic to transfer a file over an sftp + * channel to or from a remote host, given a particular handle type. + * + * @param connection - Connection over which to transfer files + * @param source - source path + * @param destination - destination path + * @param permissions - permissions for the remote file (if the file + * destination is remote) + * @param transferType - the HandleType indicating what kind of transfer it is + * @return - CommandStatus indicating whether or not the transfer was successful + */ protected CommandStatus executeTransfer(Connection connection, String source, String destination, int permissions, HandleType transferType) { - ChannelSftp channel = null; try { - channel = connection.getSftpChannel(); // Determine how to proceed given what kind of copy it is - if (transferType == HandleType.localRemote) { // If move type is local -> remote, use put - channel.put(source, destination); - } else if (transferType == HandleType.remoteLocal) { // if move type is remote -> local, use get - channel.get(source, destination); - } else if (transferType == HandleType.remoteRemote) { // if move type is remote -> remote, call function + if (transferType == HandleType.localRemote) { + transferLocalToRemote(connection, source, destination); + } else if (transferType == HandleType.remoteLocal) { + transferRemoteToLocal(connection, source, destination); + } else if (transferType == HandleType.remoteRemote) { transferRemoteToRemote(connection, source, destination); } else { logger.info("Unknown handle type..."); @@ -72,10 +90,13 @@ protected CommandStatus executeTransfer(Connection connection, String source, St // If permissions was actually instantiated and isn't the default, then perform // a chmod - if (permissions != -999) - channel.chmod(permissions, destination); - - } catch (JSchException | SftpException e) { + if (permissions != -999 && transferType != HandleType.remoteLocal) { + SftpClient channel = connection.getSftpChannel(); + Attributes attrs = channel.stat(destination); + attrs.setPermissions(permissions); + channel.setStat(destination, attrs); + } + } catch (IOException e) { logger.error("Failed to connect or obtain file to/from remote host. Returning failed.", e); status = CommandStatus.FAILED; return status; @@ -91,11 +112,12 @@ protected CommandStatus executeTransfer(Connection connection, String source, St * copy a file from one location on the remote host to another location on the * remote host. * - * @throws JSchException + * @param connection - Connection with which to transfer + * @param source - source path + * @param destination - destination path + * @throws IOException */ - private void transferRemoteToRemote(Connection connection, String source, String destination) throws JSchException { - // Open an execution channel - ChannelExec execChannel = (ChannelExec) connection.getSession().openChannel("exec"); + private void transferRemoteToRemote(Connection connection, String source, String destination) throws IOException { // Make a transfer command to execute // First check if it is a move or copy String moveType = "cp "; @@ -105,23 +127,76 @@ private void transferRemoteToRemote(Connection connection, String source, String // Build the command to execute String command = moveType + source + " " + destination; // Set the command for the JSch connection - execChannel.setCommand(command); - // If the channel isn't connected, connect and run the command + connection.getSession().executeRemoteCommand(command); + } + + /** + * Transfers a local file to the remote host. If the remote destination is a + * directory, a file with the same name as the local file will be created in the + * directory. + * + * @param connection - Connection with which to transfer + * @param source - source path + * @param destination - destination path + * @throws IOException + */ + private void transferLocalToRemote(Connection connection, String source, String destination) throws IOException { + SftpClient client = connection.getSftpChannel(); try { - execChannel.connect(); - } catch (JSchException e) { - logger.error("Channel isn't connected and can't copy remote to remote...", e); - throw e; + if (client.stat(destination).isDirectory()) { + Path path = FileSystems.getDefault().getPath(source); + String sep = ""; + if (!destination.endsWith("/")) { + sep = "/"; + } + destination += sep + path.getFileName(); + } + } catch (SftpException e) { + if (!(e.getStatus() == SftpConstants.SSH_FX_NO_SUCH_FILE)) { + throw e; + } + } + try (OutputStream dstStream = client.write(destination, OpenMode.Create, OpenMode.Write, OpenMode.Truncate)) { + try (InputStream srcStream = new FileInputStream(source)) { + byte[] buf = new byte[32 * 1024]; + while (srcStream.read(buf) > 0) { + dstStream.write(buf); + } + } + } + } + + /** + * Transfers a remote file to the local host. If the destination is a directory, + * a file with the same name as the remote file will be created in the + * directory. + * + * @param connection - Connection with which to transfer + * @param source - source path + * @param destination - destination path + * @throws IOException + */ + private void transferRemoteToLocal(Connection connection, String source, String destination) throws IOException { + Path dstPath = FileSystems.getDefault().getPath(destination); + if (dstPath.toFile().isDirectory()) { + String[] tokens = source.split("/"); + dstPath = dstPath.resolve(tokens[tokens.length - 1]); + } + try (OutputStream dstStream = Files.newOutputStream(dstPath)) { + try (InputStream srcStream = connection.getSftpChannel().read(source, OpenMode.Read)) { + byte[] buf = new byte[32 * 1024]; + while (srcStream.read(buf) > 0) { + dstStream.write(buf); + } + } } - // Disconnect extra channel when finished - execChannel.disconnect(); } /** * Setter function to tell the class whether or not the transfer is a move or a * copy * - * @param isMove + * @param isMove - true if it is a move, false if it is a copy */ protected void isMove(boolean isMove) { this.isMove = isMove; diff --git a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/TxtFileConnectionAuthorizationHandler.java b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/TxtFileConnectionAuthorizationHandler.java index 0a271e404..8bed77f8f 100644 --- a/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/TxtFileConnectionAuthorizationHandler.java +++ b/org.eclipse.ice.commands/src/main/java/org/eclipse/ice/commands/TxtFileConnectionAuthorizationHandler.java @@ -42,14 +42,15 @@ public TxtFileConnectionAuthorizationHandler() { } /** - * No options required for console authorization - * See {@link org.eclipse.ice.commands.ConnectionAuthorizationHandler#setOption(String)} + * No options required for console authorization See + * {@link org.eclipse.ice.commands.ConnectionAuthorizationHandler#setOption(String)} */ @Override public void setOption(String option) { this.path = option; setUsernameHostname(); } + /** * See * {@link org.eclipse.ice.commands.ConnectionAuthorizationHandler#getPassword()} @@ -59,20 +60,16 @@ protected char[] getPassword() { File credFile = new File(path); - Scanner scanner = null; - try { - scanner = new Scanner(credFile); + try (Scanner scanner = new Scanner(credFile)) { + // Skip the username since it has already been set + username = scanner.next(); + // Get the password + char[] pwd = scanner.next().toCharArray(); + return pwd; } catch (FileNotFoundException e) { logger.error("A path was given where the ssh credentials live, but that path doesn't exist!", e); - return null; } - // Skip the username since it has already been set - username = scanner.next(); - // Get the password - char[] pwd = scanner.next().toCharArray(); - - return pwd; - + return null; } /** @@ -86,17 +83,18 @@ private void setUsernameHostname() { Scanner scanner = null; try { scanner = new Scanner(credFile); + + username = scanner.next(); + // Get the next line, and then set it to null so that it is picked up by the + // garbage collector and erased + char[] password = scanner.next().toCharArray(); + // Erase contents of pwd and fill with null + Arrays.fill(password, Character.MIN_VALUE); + + hostname = scanner.next(); } catch (FileNotFoundException e) { logger.error("A path was given where the ssh credentials live, but that path doesn't exist!", e); } - username = scanner.next(); - // Get the next line, and then set it to null so that it is picked up by the - // garbage collector and erased - char[] password = scanner.next().toCharArray(); - // Erase contents of pwd and fill with null - Arrays.fill(password, Character.MIN_VALUE); - - hostname = scanner.next(); } @@ -104,7 +102,7 @@ private void setUsernameHostname() { * Getter for * {@link org.eclipse.ice.commands.TxtFileConnectionAuthorizationHandler#path} * - * @return + * @return path - string of path to text file */ public String getPath() { return path; @@ -114,7 +112,8 @@ public String getPath() { * Setter for * {@link org.eclipse.ice.commands.TxtFileConnectionAuthorizationHandler#path} * Also then sets the username and hostname to that defined by the new path - * @param + * + * @param path - string of path to set */ public void setPath(String path) { this.path = path; diff --git a/org.eclipse.ice.commands/src/main/resources/log4j.properties b/org.eclipse.ice.commands/src/main/resources/log4j.properties index e4fcb0130..d20b3927d 100644 --- a/org.eclipse.ice.commands/src/main/resources/log4j.properties +++ b/org.eclipse.ice.commands/src/main/resources/log4j.properties @@ -1,9 +1,7 @@ -# Set root logger level to DEBUG and its only appender to A1. -log4j.rootLogger=DEBUG, A1 - -# A1 is set to be a ConsoleAppender. -log4j.appender.A1=org.apache.log4j.ConsoleAppender - -# A1 uses PatternLayout. -log4j.appender.A1.layout=org.apache.log4j.PatternLayout -log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n \ No newline at end of file +# Root logger +log4j.rootLogger=INFO, console + +log4j.appender.console=org.apache.log4j.ConsoleAppender +#log4j.appender.console.Target=System.out +log4j.appender.console.layout=org.apache.log4j.PatternLayout +log4j.appender.consoleAppender.layout.ConversionPattern=[%t] %-5p %c %x - %m%n diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/CommandConfigurationTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/CommandConfigurationTest.java index 64f403c45..038d5a7fe 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/CommandConfigurationTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/CommandConfigurationTest.java @@ -11,6 +11,10 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; + import java.util.ArrayList; import org.eclipse.ice.commands.CommandConfiguration; @@ -42,18 +46,18 @@ public void testCommandConfiguration() { config.setOutFileName("outFile.txt"); // Assert whether or not things are/aren't set - assert (config.getOutFileName() != null); - assert (config.getOS() != null); + assertNotNull(config.getOutFileName()); + assertNotNull(config.getOS()); // Didn't set working directory - assert (config.getWorkingDirectory() == null); + assertNull(config.getWorkingDirectory()); // Assert that the default local OS is set - assert (config.getOS().equals(System.getProperty("os.name"))); + assertEquals(System.getProperty("os.name"), config.getOS()); config.setOS("JoeOsbornOS"); - assert (config.getOS().equals("JoeOsbornOS")); + assertEquals("JoeOsbornOS", config.getOS()); } @@ -73,14 +77,14 @@ public void testGetExecutableName() { config.setAppendInput(true); config.setOS(System.getProperty("os.name")); String executable = config.getExecutableName(); - assert (executable.equals("bash ./test_code_execution.sh someInputFile.txt")); + assertEquals("bash ./test_code_execution.sh someInputFile.txt", executable); // Test that if num processes is more than 1, mpi options are added config.setNumProcs("4"); // arbitrary number // We can test append input as well when it is false config.setAppendInput(false); executable = config.getExecutableName(); - assert (executable.equals("mpirun -np 4 bash ./test_code_execution.sh")); + assertEquals("mpirun -np 4 bash ./test_code_execution.sh", executable); } @@ -103,7 +107,7 @@ public void testArgumentAndInputFileConfiguration() { String executable = configuration.getExecutableName(); - assert (executable.equals("python random_python_script.py some_arg some_other_arg someInputFile.txt")); + assertEquals("python random_python_script.py some_arg some_other_arg someInputFile.txt", executable); } /** @@ -126,8 +130,9 @@ public void testGetExecutableNameSplitCommand() { splitConfig.setInstallDirectory("~/install_dir"); splitConfig.setOS(System.getProperty("os.name")); String executable = splitConfig.getExecutableName(); - assert (executable.equals( - "./dummy.sh inputfile.txt; ./next_file.sh /some/dummy/path/to/an/inputfile.txt; ./other_file.sh ~/install_dir/")); + assertEquals( + "./dummy.sh inputfile.txt; ./next_file.sh /some/dummy/path/to/an/inputfile.txt; ./other_file.sh ~/install_dir/", + executable); ArrayList split = new ArrayList(); split = splitConfig.getSplitCommand(); @@ -139,7 +144,7 @@ public void testGetExecutableNameSplitCommand() { checkSplit.add("./other_file.sh ~/install_dir/"); for (int i = 0; i < split.size(); i++) { - assert (split.get(i).equals(checkSplit.get(i))); + assertEquals(checkSplit.get(i), split.get(i)); } } diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/CommandFactoryTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/CommandFactoryTest.java index 1f3536de8..11103e280 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/CommandFactoryTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/CommandFactoryTest.java @@ -11,6 +11,7 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertEquals; import static org.junit.Assert.fail; import java.io.File; @@ -19,6 +20,8 @@ import java.net.UnknownHostException; import java.util.ArrayList; +import org.apache.sshd.client.subsystem.sftp.SftpClient; +import org.apache.sshd.client.subsystem.sftp.SftpClientFactory; import org.eclipse.ice.commands.Command; import org.eclipse.ice.commands.CommandConfiguration; import org.eclipse.ice.commands.CommandFactory; @@ -36,12 +39,9 @@ import org.junit.After; import org.junit.AfterClass; import org.junit.BeforeClass; +import org.junit.Ignore; import org.junit.Test; -import com.jcraft.jsch.ChannelSftp; -import com.jcraft.jsch.JSchException; -import com.jcraft.jsch.SftpException; - /** * This class tests {@link org.eclipse.ice.commands.CommandFactory}. * @@ -124,7 +124,7 @@ public static void tearDownAfterClass() throws IOException, InterruptedException rm += " someLsOutFile.txt someLsErrFile.txt someMultRemoteOutFile.txt someMultRemoteErrFile.txt"; rm += " somePythOutFile.txt somePythErrFile.txt someLsRemoteErrFile.txt someLsRemoteOutFile.txt"; rm += " src/test/java/org/eclipse/ice/tests/someInputFile.txt src/test/java/org/eclipse/ice/tests/someOtherInputFile.txt"; - rm += " pythOutFile.txt pythErrFile.txt"; + rm += " pythOutFile.txt pythErrFile.txt hopRemoteOutFile.txt hopRemoteErrFile.txt"; ArrayList command = new ArrayList(); // Build a command if (System.getProperty("os.name").toLowerCase().contains("win")) { @@ -152,25 +152,27 @@ public static void tearDownAfterClass() throws IOException, InterruptedException /** * This function tests a multi-hop remote command, where the command logs into a - * remote host and then executes on a different remote host. TODO - Commented - * out for now since multi-hop command source code is not implemented and can be - * for future development. + * remote host and then executes on a different remote host. */ - // @Test + @Test + @Ignore // ignore until second host is setup public void testMultiHopRemoteCommand() { - fail("src not implemented"); System.out.println("\n\n\nTesting a multi-hop remote command"); // Set the CommandConfiguration class CommandConfiguration commandConfig = setupDefaultCommandConfig(); commandConfig.setExecutable("./test_code_execution.sh"); commandConfig.addInputFile("someInputFile", "someInputFile.txt"); + commandConfig.addInputFile("someOtherFile", "someOtherInputFile.txt"); commandConfig.setAppendInput(true); commandConfig.setCommandId(99); commandConfig.setErrFileName("hopRemoteErrFile.txt"); commandConfig.setOutFileName("hopRemoteOutFile.txt"); + // This is the directory to run the job on the destination system, i.e. + // system C commandConfig.setRemoteWorkingDirectory("/tmp/remoteCommandTestDirectory"); - // Just put in a dummy directory for now - commandConfig.setWorkingDirectory("/home/user/somedirectory"); + // This is the directory on the jump host which contains the job information + // and files, e.g. the script, input files, etc. + commandConfig.setWorkingDirectory("/home/4jo/remoteCommandDirectory"); // Set the connection configuration to a dummy remote connection // This is the connection where the job will be executed @@ -178,26 +180,34 @@ public void testMultiHopRemoteCommand() { ConnectionAuthorizationHandlerFactory authFactory = new ConnectionAuthorizationHandlerFactory(); // Request a ConnectionAuthorization of type text file which contains the // credentials - ConnectionAuthorizationHandler auth = authFactory.getConnectionAuthorizationHandler("text", - "/tmp/ice-remote-creds.txt"); + String keyPath = System.getProperty("user.home") + "/.ssh/somekey"; + ConnectionAuthorizationHandler auth = authFactory.getConnectionAuthorizationHandler("keypath", + keyPath); + auth.setHostname("hostname"); + auth.setUsername("password"); // Set it - connectionConfig.setAuthorization(auth); + ConnectionConfiguration firstConn = new ConnectionConfiguration(); + firstConn.setAuthorization(auth); // Note the password can be input at the console by not setting the // the password explicitly in the connection configuration - connectionConfig.setName("executeConnection"); - connectionConfig.deleteWorkingDirectory(true); + firstConn.setName("hopConnection"); + firstConn.deleteWorkingDirectory(false); - ConnectionConfiguration intermConnection = new ConnectionConfiguration(); - // TODO - this will have to be changed to some other remote connection - intermConnection.setAuthorization(auth); - intermConnection.setName("intermediateConnection"); - intermConnection.deleteWorkingDirectory(false); + ConnectionConfiguration secondConn = new ConnectionConfiguration(); + String credFile = "/tmp/ice-remote-creds.txt"; + if(System.getProperty("os.name").toLowerCase().contains("win")) + credFile = "C:\\Users\\Administrator\\ice-remote-creds.txt"; + + ConnectionAuthorizationHandler intermAuth = authFactory.getConnectionAuthorizationHandler("text",credFile); + secondConn.setAuthorization(intermAuth); + secondConn.setName("executeConnection"); + secondConn.deleteWorkingDirectory(false); // Get the command Command remoteCommand = null; try { - remoteCommand = factory.getCommand(commandConfig, intermConnection, connectionConfig); + remoteCommand = factory.getCommand(commandConfig, firstConn, secondConn); } catch (IOException e) { e.printStackTrace(); } @@ -206,7 +216,7 @@ public void testMultiHopRemoteCommand() { CommandStatus status = remoteCommand.execute(); // assert that it was successful - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); } @@ -253,7 +263,7 @@ public void testBoringCommandLocally() { CommandStatus status = cmd.execute(); // Check that it properly finished - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); System.out.println("Finished boring command locally"); } @@ -287,7 +297,7 @@ public void testBoringCommandRemotely() { CommandStatus status = cmd.execute(); // Assert that it finished correctly - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); } /** @@ -327,7 +337,7 @@ public void testFunctionalLocalCommand() { // Run it CommandStatus status = localCommand.execute(); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); System.out.println("Finished functional local command"); } @@ -364,7 +374,7 @@ public void testNonFunctionalLocalCommand() { // Run it and expect that it fails CommandStatus status = localCommand.execute(); - assert (status == CommandStatus.INFOERROR); + assertEquals(CommandStatus.INFOERROR, status); } @@ -409,7 +419,7 @@ public void testIncorrectWorkingDirectory() { // Run it and expect that it fails CommandStatus status2 = localCommand2.execute(); - assert (status2 == CommandStatus.FAILED); + assertEquals(CommandStatus.FAILED, status2); } /** @@ -447,7 +457,7 @@ public void testFunctionalRemoteCommand() { CommandStatus status = remoteCommand.execute(); // assert that it was successful - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); System.out.println("Finished remote functional command"); @@ -486,7 +496,7 @@ public void testMultipleInputFilesRemotely() { CommandStatus status = remoteCommand.execute(); // assert that it was successful - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); System.out.println("Finished multiple input files remotely"); } @@ -529,7 +539,7 @@ public void testMultipleInputFilesLocally() { // Run it CommandStatus status = localCommand.execute(); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); System.out.println("Finished testing multiple input files locally"); } @@ -573,8 +583,7 @@ public void testPythonScript() { CommandStatus status = command.execute(); - assert (status == CommandStatus.SUCCESS); - System.out.println("finished python script test"); + assertEquals(CommandStatus.SUCCESS, status); } /** @@ -584,11 +593,10 @@ public void testPythonScript() { @Test public void testRemoteExecutableLocalInputFiles() { System.out.println("Testing command where files live on different hosts."); - + // Get the dummy connection configuration ConnectionConfiguration connectionConfig = setupDummyConnectionConfiguration(); - // Get the present working directory String pwd = System.getProperty("user.dir"); @@ -606,12 +614,11 @@ public void testRemoteExecutableLocalInputFiles() { RemoteFileHandler remoteHandler = new RemoteFileHandler(); remoteHandler.setConnectionConfiguration(connectionConfig); remoteHandler.copy(inputFileDir + "commands/test_python_script.py", "/tmp/test_python_script.py"); - + // Create a command configuration corresponding to a python script CommandConfiguration configuration = setupDefaultCommandConfig(); // This path exists on the dummy host server - configuration.setExecutable( - "/tmp/test_python_script.py"); + configuration.setExecutable("/tmp/test_python_script.py"); configuration.setInterpreter("python"); configuration.setCommandId(9); configuration.setErrFileName("pythErrFile.txt"); @@ -622,7 +629,6 @@ public void testRemoteExecutableLocalInputFiles() { configuration.addInputFile("inputfile2", "someOtherInputFile.txt"); configuration.setRemoteWorkingDirectory("/tmp/pythonTest"); - // Get the command and run it Command command = null; try { @@ -633,24 +639,24 @@ public void testRemoteExecutableLocalInputFiles() { CommandStatus status = command.execute(); - assert (status == CommandStatus.SUCCESS); - + assertEquals(CommandStatus.SUCCESS, status); + // Delete the moved python script on the remote server once finished // Get the connection and channel to delete - Connection connection = ((RemoteCommand)command).getConnection(); + Connection connection = ((RemoteCommand) command).getConnection(); // Delete the script try { + SftpClient client = SftpClientFactory.instance().createSftpClient(connection.getSession()); // open the channel - connection.setSftpChannel(connection.getSession().openChannel("sftp")); - ChannelSftp channel = connection.getSftpChannel(); + connection.setSftpChannel(client); + SftpClient channel = connection.getSftpChannel(); // connect and delete the script - channel.connect(); - channel.rm("/tmp/test_python_script.py"); - channel.disconnect(); - } catch (JSchException | SftpException e1) { + channel.remove("/tmp/test_python_script.py"); + channel.close(); + } catch (IOException e1) { e1.printStackTrace(); } - + System.out.println("finished python script test"); } diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/CommandTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/CommandTest.java index 112e5607b..dd4daeb8d 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/CommandTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/CommandTest.java @@ -12,6 +12,9 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; + import java.io.File; import java.io.IOException; import java.util.ArrayList; @@ -31,8 +34,6 @@ import org.junit.BeforeClass; import org.junit.Test; -import com.jcraft.jsch.JSchException; - /** * Test for class {@link org.eclipse.ice.commands.Command}. * @@ -142,7 +143,7 @@ public void testRemoteCommand() { CommandStatus status = remoteCommand.execute(); // Assert that it finished correctly - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // You can get the output in string form if desired String output = remoteCommand.getCommandConfiguration().getStdOutputString(); @@ -182,8 +183,7 @@ public void testLocalCommand() { // Use the function already defined in the command factory to get the // local host name - CommandFactoryTest factory = new CommandFactoryTest(); - String hostname = factory.getLocalHostname(); + String hostname = CommandFactoryTest.getLocalHostname(); // Create a connectionConfiguration to indicate a local command ConnectionConfiguration connection = new ConnectionConfiguration(); @@ -197,7 +197,7 @@ public void testLocalCommand() { // Check that it completed correctly System.out.println("Command finished with: " + status); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); } @@ -215,7 +215,7 @@ public void testCheckStatus() { // Check to make sure that an exception is thrown for a bad status when the // status is checked - assert (!command.checkStatus(status)); + assertFalse(command.checkStatus(status)); } /** @@ -243,7 +243,7 @@ public void testMultipleRemoteCommands() { // Open the connection try { ConnectionManagerFactory.getConnectionManager().openConnection(connectConfig); - } catch (JSchException e) { + } catch (IOException e) { e.printStackTrace(); } @@ -266,7 +266,7 @@ public void testMultipleRemoteCommands() { CommandStatus status = remoteCommand.execute(); // Assert that it finished correctly - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Try a second command with the same instance of command configuration and // remote command @@ -280,7 +280,7 @@ public void testMultipleRemoteCommands() { status = remoteCommand.execute(); // Assert successful completion - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); } @@ -309,7 +309,7 @@ public void testArgumentCommand() { // Open the connection try { ConnectionManagerFactory.getConnectionManager().openConnection(connectConfig); - } catch (JSchException e) { + } catch (IOException e) { e.printStackTrace(); } @@ -334,7 +334,7 @@ public void testArgumentCommand() { CommandStatus status = remoteCommand.execute(); // Assert that it finished correctly - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); } diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionAuthorizationHandlerFactoryTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionAuthorizationHandlerFactoryTest.java index 709aae7c6..311568c73 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionAuthorizationHandlerFactoryTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionAuthorizationHandlerFactoryTest.java @@ -11,8 +11,11 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + import java.io.File; -import java.io.FileNotFoundException; +import java.io.IOException; import java.net.InetAddress; import java.net.UnknownHostException; import java.util.Arrays; @@ -24,8 +27,6 @@ import org.eclipse.ice.commands.ConnectionManagerFactory; import org.junit.Test; -import com.jcraft.jsch.JSchException; - /** * This class tests * {@link org.eclipse.ice.commands.ConnectionAuthorizationHandlerFactory} @@ -53,8 +54,8 @@ public void testLocalAuthorization() { e.printStackTrace(); } // Assert that the username and hostname are that of the local computer - assert (local.getHostname() == addr.getHostName()); - assert (local.getUsername() == System.getProperty("user.name")); + assertEquals(addr.getHostName(), local.getHostname()); + assertEquals(local.getUsername(), System.getProperty("user.name")); } @@ -65,7 +66,7 @@ public void testLocalAuthorization() { * @throws JSchException */ @Test - public void testTextAuthorization() throws JSchException { + public void testTextAuthorization() throws IOException { String credFile = "/tmp/ice-remote-creds.txt"; if (System.getProperty("os.name").toLowerCase().contains("win")) credFile = "C:\\Users\\Administrator\\ice-remote-creds.txt"; @@ -75,32 +76,28 @@ public void testTextAuthorization() throws JSchException { // Assert that the hostname and username are whatever was put in File file = new File(credFile); - Scanner scanner = null; - try { - scanner = new Scanner(file); - } catch (FileNotFoundException e){ - e.printStackTrace(); + try (Scanner scanner = new Scanner(file)) { + String username = scanner.next(); + char[] pwd = scanner.next().toCharArray(); + // delete the password since we don't need it here + Arrays.fill(pwd, Character.MIN_VALUE); + String hostname = scanner.next(); + + assertEquals(username, text.getUsername()); + assertEquals(hostname, text.getHostname()); + + // Create a connection configuration to actually try and open the connection + ConnectionConfiguration config = new ConnectionConfiguration(); + config.setName("Text"); + config.setAuthorization(text); + + // Try to open the connection + ConnectionManagerFactory.getConnectionManager().openConnection(config); + // Assert that it was correctly opened + assertTrue(ConnectionManagerFactory.getConnectionManager().isConnectionOpen("Text")); + // Close it since we are done with it + ConnectionManagerFactory.getConnectionManager().removeAllConnections(); } - String username = scanner.next(); - char[] pwd = scanner.next().toCharArray(); - // delete the password since we don't need it here - Arrays.fill(pwd, Character.MIN_VALUE); - String hostname = scanner.next(); - - assert(text.getUsername().equals(username)); - assert(text.getHostname().equals(hostname)); - - // Create a connection configuration to actually try and open the connection - ConnectionConfiguration config = new ConnectionConfiguration(); - config.setName("Text"); - config.setAuthorization(text); - - // Try to open the connection - ConnectionManagerFactory.getConnectionManager().openConnection(config); - // Assert that it was correctly opened - assert (ConnectionManagerFactory.getConnectionManager().isConnectionOpen("Text")); - // Close it since we are done with it - ConnectionManagerFactory.getConnectionManager().removeAllConnections(); } /** @@ -110,7 +107,7 @@ public void testTextAuthorization() throws JSchException { * @throws JSchException */ @Test - public void testKeyPathAuthorization() throws JSchException { + public void testKeyPathAuthorization() throws IOException { // Filepath to the dummy host key String keyPath = System.getProperty("user.home") + "/.ssh/dummyhostkey"; @@ -136,7 +133,7 @@ public void testKeyPathAuthorization() throws JSchException { // Try to open the connection ConnectionManagerFactory.getConnectionManager().openConnection(config); // Assert that it was correctly opened - assert (ConnectionManagerFactory.getConnectionManager().isConnectionOpen("keyPath")); + assertTrue(ConnectionManagerFactory.getConnectionManager().isConnectionOpen("keyPath")); // Close it since we are done with it ConnectionManagerFactory.getConnectionManager().removeAllConnections(); diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionConfigurationTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionConfigurationTest.java index 60629b76e..109de1080 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionConfigurationTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionConfigurationTest.java @@ -11,6 +11,9 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; + import org.eclipse.ice.commands.ConnectionConfiguration; import org.junit.Test; @@ -32,10 +35,10 @@ public void testConstructor() { ConnectionConfiguration config = new ConnectionConfiguration(); // Check that things are set to blank by default - assert (config.getName().equals("")); + assertEquals("", config.getName()); // Check that this defaults to false - assert (config.deleteWorkingDirectory() == false); + assertFalse(config.deleteWorkingDirectory()); } } diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionManagerTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionManagerTest.java index 980138abd..60b392091 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionManagerTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionManagerTest.java @@ -11,12 +11,15 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; -import java.io.File; -import java.io.FileNotFoundException; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; import java.util.HashMap; -import java.util.Scanner; -import org.eclipse.ice.commands.BasicConnectionAuthorizationHandler; import org.eclipse.ice.commands.Connection; import org.eclipse.ice.commands.ConnectionAuthorizationHandler; import org.eclipse.ice.commands.ConnectionAuthorizationHandlerFactory; @@ -27,10 +30,9 @@ import org.junit.After; import org.junit.AfterClass; import org.junit.BeforeClass; +import org.junit.Ignore; import org.junit.Test; -import com.jcraft.jsch.JSchException; - /** * Test for class {@link org.eclipse.ice.commands.ConnectionManager}. Note that * as currently implemented, the ssh connection is a dummy account set up within @@ -84,21 +86,22 @@ public static void setUpBeforeClass() throws Exception { } /** - * Clear out the connections formed after each test so that each test starts fresh - * with a clean slated connection manager + * Clear out the connections formed after each test so that each test starts + * fresh with a clean slated connection manager + * * @throws Exception */ @After public void tearDown() throws Exception { // Clear out the connection manager so we start fresh with each test ConnectionManagerFactory.getConnectionManager().removeAllConnections(); - // Reset the known hosts directory, for after the test with the + // Reset the known hosts directory, for after the test with the // expected JSch exception due to nonexistent known_hosts ConnectionManagerFactory.getConnectionManager() - .setKnownHosts(System.getProperty("user.home") + "/.ssh/known_hosts"); + .setKnownHosts(System.getProperty("user.home") + "/.ssh/known_hosts"); - } + /** * This function deletes all of the connections in the connection manager once * the tests have run and completed. @@ -112,23 +115,23 @@ public static void tearDownAfterClass() throws Exception { manager.removeAllConnections(); // Assert that there are no more connections in the list - assert (manager.getConnectionList().size() == 0); + assertEquals(0, manager.getConnectionList().size()); // Make sure the known hosts are reset to the default directory manager.setKnownHosts(System.getProperty("user.home") + "/.ssh/known_hosts"); - + } - - + /** * This function tests opening a connection with an already generated key path - * @throws JSchException + * + * @throws JSchException */ @Test - public void testOpenConnectionKeyPath() throws JSchException { + public void testOpenConnectionKeyPath() throws IOException { ConnectionManager manager = ConnectionManagerFactory.getConnectionManager(); System.out.println("Testing keypath open connection"); - + // Make a connection configuration for using a key path ConnectionConfiguration keyConfiguration = new ConnectionConfiguration(); keyConfiguration.setName("keypath"); @@ -143,16 +146,15 @@ public void testOpenConnectionKeyPath() throws JSchException { keyConfiguration.setAuthorization(auth); // Open the connection manager.openConnection(keyConfiguration); - + // assert that it was properly opened - assert(manager.isConnectionOpen("keypath")); + assert (manager.isConnectionOpen("keypath")); ConnectionManagerFactory.getConnectionManager() - .setKnownHosts(System.getProperty("user.home") + "/.ssh/known_hosts"); + .setKnownHosts(System.getProperty("user.home") + "/.ssh/known_hosts"); } - /** * Test method for * {@link org.eclipse.ice.commands.ConnectionManager#OpenConnection(ConnectionConfiguration)} @@ -164,12 +166,12 @@ public void testOpenConnection() { // Try to open a connection try { connect = manager.openConnection(configuration); - } catch (JSchException e) { + } catch (IOException e) { e.printStackTrace(); } // If connect is not null and there was no error thrown, it was established // correctly - assert (connect != null); + assertNotNull(connect); } @@ -181,11 +183,11 @@ public void testGetConnection() { ConnectionManager manager = ConnectionManagerFactory.getConnectionManager(); Connection testConnection = null; testConnection = manager.getConnection(connectionName); - assert (testConnection != null); + assertNotNull(testConnection); // test a bad connection that doesn't exist in the connection list Connection badConnection = manager.getConnection("nonexistent_connection"); - assert (badConnection == null); + assertNull(badConnection); } /** @@ -197,14 +199,14 @@ public void testCloseConnection() { // disconnect the session manager.closeConnection(connectionName); - assert (!manager.getConnection(connectionName).getSession().isConnected()); + assertFalse(manager.getConnection(connectionName).getSession().isOpen()); // Remove the connection from the connection manager manager.removeConnection(connectionName); // Assert that this connection no longer exists, so when you try to get it it is // null - assert (manager.getConnection(connectionName) == null); + assertNull(manager.getConnection(connectionName)); } /** @@ -214,7 +216,7 @@ public void testCloseConnection() { public void testMultipleConnections() { ConnectionManager manager = ConnectionManagerFactory.getConnectionManager(); - + // Read in a dummy configuration file that contains credentials String credFile = "/tmp/ice-remote-creds.txt"; if (System.getProperty("os.name").toLowerCase().contains("win")) @@ -237,13 +239,12 @@ public void testMultipleConnections() { conf2.setAuthorization(auth2); conf2.setName("someOtherConnection"); - Connection conn1 = null, conn2 = null; // Open some configurations try { - conn1 = manager.openConnection(configuration); - conn2 = manager.openConnection(conf2); + manager.openConnection(configuration); + manager.openConnection(conf2); - } catch (JSchException e) { + } catch (IOException e) { e.printStackTrace(); } @@ -253,24 +254,24 @@ public void testMultipleConnections() { // Expect only two connections since one of the connections is not good (i.e. // conn3 has a bad password, therefore it isn't added to the list) - - assert (connections.size() == 2); + + assertEquals(2, connections.size()); // List all available connections to the console screen manager.listAllConnections(); // Check that the name returns the appropriate connection from ConnectionManager - assert (manager.getConnection("FirstConnection").getConfiguration().getName().equals("FirstConnection")); + assertEquals("FirstConnection", manager.getConnection("FirstConnection").getConfiguration().getName()); // Check that the connection is actually open, since FirstConnection is a good // connection - assert (manager.isConnectionOpen("FirstConnection")); + assertTrue(manager.isConnectionOpen("FirstConnection")); // Test closing all of the connections manager.closeAllConnections(); // Check that the connection is closed - assert (!manager.isConnectionOpen("FirstConnection")); + assertFalse(manager.isConnectionOpen("FirstConnection")); } @@ -283,7 +284,7 @@ public void testMultipleConnections() { @Test public void testValidConnection() { System.out.println("Testing valid connection"); - + testOpenConnection(); testGetConnection(); @@ -297,19 +298,64 @@ public void testValidConnection() { * * @throws JSchException */ - @Test(expected = JSchException.class) - public void testNoKnownHost() throws JSchException { + @Test(expected = IOException.class) + public void testNoKnownHost() throws IOException { // Set the known hosts to something random, where we know the ssh fingerprint // doesn't exist System.out.println("Testing with no known host file, exception expected"); ConnectionManagerFactory.getConnectionManager().setKnownHosts("/tmp/knownhosts"); // Try to open a connection - // Should throw a JSchException since the host fingerprint won't match + // Should throw a IOException since the host fingerprint won't match connect = ConnectionManagerFactory.getConnectionManager().openConnection(configuration); } + /** + * This tests the method openForwardingConnection, which creates a connection + * between three systems by porting the connection through an intermediary host + * @throws IOException + */ + @Test + @Ignore // ignore until second dummy host is setup + public void testForwardConnection() throws IOException { + + ConnectionManager manager = ConnectionManagerFactory.getConnectionManager(); + + // Read in a dummy configuration file that contains credentials + String credFile = "/tmp/ice-remote-creds.txt"; + if (System.getProperty("os.name").toLowerCase().contains("win")) + credFile = "C:\\Users\\Administrator\\ice-remote-creds.txt"; - - + ConnectionAuthorizationHandlerFactory authFactory = new ConnectionAuthorizationHandlerFactory(); + // Request a ConnectionAuthorization of type text file which contains the + // credentials + String keyPath = System.getProperty("user.home") + "/.ssh/somekey"; + ConnectionAuthorizationHandler auth = authFactory.getConnectionAuthorizationHandler("keypath", + keyPath); + auth.setHostname("hostname"); + auth.setUsername("password"); + ConnectionConfiguration config = new ConnectionConfiguration(); + config.setAuthorization(auth); + config.setName("forwardConnection"); + Connection firstConnection = manager.openConnection(config); + + // Make sure the connection was opened properly + assert (manager.isConnectionOpen(firstConnection.getConfiguration().getName())); + + // Now get the final host authorization + ConnectionAuthorizationHandler intermauth = authFactory.getConnectionAuthorizationHandler("text", + credFile); + + // Setup the configuration + ConnectionConfiguration secondConn = new ConnectionConfiguration(); + secondConn.setAuthorization(intermauth); + secondConn.setName("executeConnection"); + secondConn.deleteWorkingDirectory(false); + System.out.println("\n\n\n\nOpening forwarding connection"); + // Try to open it + Connection forwardConnection = manager.openForwardingConnection(firstConnection, secondConn); + + // Assert that it is open + assert(manager.isConnectionOpen(forwardConnection.getConfiguration().getName())); + } } diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionTest.java index 8aa57d0fe..9831d824c 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/ConnectionTest.java @@ -11,6 +11,8 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertNull; + import org.eclipse.ice.commands.Connection; import org.junit.Test; @@ -31,9 +33,9 @@ public void testConnection() { Connection connection = new Connection(); // Check that the default connection leaves the configuration empty - assert (connection.getConfiguration() == null); + assertNull(connection.getConfiguration()); - assert (connection.getSession() == null); + assertNull(connection.getSession()); } } diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/IFileHandlerFactoryTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/IFileHandlerFactoryTest.java index 29826112e..ef7755ae2 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/IFileHandlerFactoryTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/IFileHandlerFactoryTest.java @@ -11,6 +11,9 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + import java.io.File; import java.net.InetAddress; import java.net.UnknownHostException; @@ -113,11 +116,11 @@ public void testLocalFileHandlerFactoryCopyCommand() throws Exception { // Now try to copy the file CommandStatus status = handler.copy(theSource, theDestination); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Check that the file exists now boolean exist = handler.exists(theDestination); - assert (exist == true); + assertTrue(exist); // Delete the test file/directory now that the test is finished fileCreator.deleteLocalSource(); @@ -156,11 +159,11 @@ public void testLocalFileHandlerFactoryMoveCommand() throws Exception { // Now try to move the file CommandStatus status = handler.move(theSource, theDestination); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Check that the file exists now boolean exist = handler.exists(theDestination); - assert (exist == true); + assertTrue(exist); // Delete the test file/directory now that the test is finished fileCreator.deleteLocalSource(); @@ -200,11 +203,11 @@ public void testLocalFileHandlerFactoryDestinationNonexistent() throws Exception // Now try to move the file CommandStatus status = handler.move(theSource, theDestination + newDirectory); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Check that the file exists now boolean exist = handler.exists(theDestination + newDirectory); - assert (exist == true); + assertTrue(exist); // Delete the test file/directory now that the test is finished fileCreator.deleteLocalSource(); @@ -247,11 +250,11 @@ public void testLocalFileHandlerFactoryChangeName() throws Exception { handler = factory.getFileHandler(config); CommandStatus status = handler.move(theSource, localNewName); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Check that the file exists now boolean exist = handler.exists(localNewName); - assert (exist == true); + assertTrue(exist); // If the file was successfully created, delete it here // Needs a special delete since the filename was created in this function @@ -280,15 +283,14 @@ public void testRemoteFileHandlerCopyCommand() throws Exception { // Get the file transfer handler IFileHandler handler = factory.getFileHandler(fileCreator.getConnection().getConfiguration()); String separator = FileSystems.getDefault().getSeparator(); - String filename = theSource.substring(theSource.lastIndexOf(separator)); + String filename = theSource.substring(theSource.lastIndexOf(separator)+1); // Now try to copy the file CommandStatus status = handler.copy(theSource, theDestination); - assert (status == CommandStatus.SUCCESS); - + assertEquals(CommandStatus.SUCCESS, status); // Check that the file exists now boolean exist = handler.exists(theDestination + filename); - assert (exist == true); + assertTrue(exist); // Delete the test file/directory now that the test is finished fileCreator.deleteLocalSource(); @@ -310,16 +312,16 @@ public void testSetHandleType() throws Exception { theSource = fileCreator.getSource(); theDestination = fileCreator.getDestination(); String separator = FileSystems.getDefault().getSeparator(); - String filename = theSource.substring(theSource.lastIndexOf(separator)); + String filename = theSource.substring(theSource.lastIndexOf(separator)+1); FileHandler handler = factory.getFileHandler(fileCreator.getConnection().getConfiguration()); handler.setHandleType(HandleType.localRemote); CommandStatus status = handler.copy(theSource, theDestination); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); - assert (handler.exists(theDestination + filename)); + assertTrue(handler.exists(theDestination + filename)); fileCreator.deleteLocalSource(); fileCreator.deleteRemoteDestination(); @@ -341,14 +343,14 @@ public void testRemoteFileHandlerMoveCommand() throws Exception { // Get the file transfer handler IFileHandler handler = factory.getFileHandler(fileCreator.getConnection().getConfiguration()); String separator = FileSystems.getDefault().getSeparator(); - String filename = theSource.substring(theSource.lastIndexOf(separator)); + String filename = theSource.substring(theSource.lastIndexOf(separator)+1); // Now try to move the file CommandStatus status = handler.move(theSource, theDestination); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Check that the file exists now boolean exist = handler.exists(theDestination + filename); - assert (exist == true); + assertTrue(exist); // Delete the test file/directory now that the test is finished fileCreator.deleteLocalSource(); @@ -375,11 +377,11 @@ public void testRemoteFileHandlerFactoryDestinationNonexistent() throws Exceptio String newDirectory = "some/other/dir/"; // theDestination = theDestination + newDirectory; CommandStatus status = handler.move(theSource, theDestination + newDirectory); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Check that the file exists now boolean exist = handler.exists(theDestination + newDirectory); - assert (exist == true); + assertTrue(exist); // Delete the test file/directory now that the test is finished fileCreator.deleteLocalSource(); diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalCommandTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalCommandTest.java index 9ce12bf6c..4421bf573 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalCommandTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalCommandTest.java @@ -11,6 +11,8 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertEquals; + import java.io.File; import java.io.IOException; import java.util.ArrayList; @@ -63,11 +65,6 @@ public class LocalCommandTest { @Before public void setUp() throws Exception { - // Use the function already defined in the command factory to get the - // local host name - CommandFactoryTest factory = new CommandFactoryTest(); - String hostname = factory.getLocalHostname(); - connection = new ConnectionConfiguration(); ConnectionAuthorizationHandlerFactory authFactory = new ConnectionAuthorizationHandlerFactory(); connection.setAuthorization(authFactory.getConnectionAuthorizationHandler("local")); @@ -128,7 +125,7 @@ public void testLocalCommand() { LocalCommand realCommand = new LocalCommand(connection, commandConfig); CommandStatus testStatus = realCommand.getStatus(); System.out.println(testStatus); - assert (testStatus == CommandStatus.PROCESSING); + assertEquals(CommandStatus.PROCESSING, testStatus); System.out.println("Finished testConfiguration\n"); } @@ -167,7 +164,7 @@ public void testExecute() { // Check that it completed correctly System.out.println("Command finished with: " + status); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); } diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalCopyFileCommandTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalCopyFileCommandTest.java index 8e5f1af0e..1558ca50a 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalCopyFileCommandTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalCopyFileCommandTest.java @@ -11,6 +11,8 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertTrue; + import java.io.IOException; import java.nio.file.DirectoryNotEmptyException; import java.nio.file.FileSystems; @@ -160,7 +162,7 @@ public void testLocalCopyFileCommand() { command.execute(); // Check if the path exists now Path path = Paths.get(dest); - assert (Files.exists(path)); + assertTrue(Files.exists(path)); } } diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalFileBrowserTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalFileBrowserTest.java index 8fd70ad77..59dabb443 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalFileBrowserTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalFileBrowserTest.java @@ -12,6 +12,9 @@ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + import java.io.File; import java.io.IOException; import java.nio.file.DirectoryNotEmptyException; @@ -23,10 +26,6 @@ import org.eclipse.ice.commands.LocalFileBrowser; import org.eclipse.ice.commands.LocalFileHandler; -import org.junit.After; -import org.junit.AfterClass; -import org.junit.Before; -import org.junit.BeforeClass; import org.junit.Test; /** @@ -70,17 +69,17 @@ public void testLocalFileBrowsing(Path topPath) throws IOException { ArrayList files = browser.getFileList(); // four files were created - assert (files.size() == 4); + assertEquals(4, files.size()); // Let's assert that each file we created is in the list for (int i = 0; i < files.size(); i++) { String file = files.get(i); // assert that the file is actually there - assert (handler.exists(file)); + assertTrue(handler.exists(file)); Path path = Paths.get(file); // assert that it is a file - assert (Files.isRegularFile(path)); + assertTrue(Files.isRegularFile(path)); } @@ -98,17 +97,17 @@ public void testLocalDirectoryBrowsing(Path topPath) throws IOException { System.out.println(files.get(i)); } // Four directories total, including the top directory - assert (files.size() == 4); + assertEquals(4, files.size()); // Let's assert that each directory we created is in the list for (int i = 0; i < files.size(); i++) { String file = files.get(i); // assert that it exists - assert (handler.exists(file)); + assertTrue(handler.exists(file)); // assert that it is a directory Path path = Paths.get(file); - assert (Files.isDirectory(path)); + assertTrue(Files.isDirectory(path)); } } diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalFileHandlerTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalFileHandlerTest.java index 9ec530017..814b9bc5d 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalFileHandlerTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalFileHandlerTest.java @@ -11,6 +11,10 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + import java.io.File; import java.io.IOException; import java.nio.file.FileSystems; @@ -104,12 +108,12 @@ public void testLocalCopy() throws Exception { FileHandler handler = null; handler = new LocalFileHandler(); CommandStatus status = handler.copy(localSource, localDestination); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Check that it exists try { boolean exist = handler.exists(localDestination + filename); - assert (exist == true); + assertTrue(exist); } catch (IOException e) { e.printStackTrace(); } @@ -132,14 +136,9 @@ public void testLocalMove() throws Exception { // Try to make a local temp file to play with createLocalTempFile(); - // Get the filename for testing exists later - String separator = FileSystems.getDefault().getSeparator(); - String filename = localSource.substring(localSource.lastIndexOf(separator)); - FileHandler handler = null; - handler = new LocalFileHandler(); - + FileHandler handler = new LocalFileHandler(); CommandStatus status = handler.move(localSource, localDestination); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Delete the local dummy file deleteLocalTempFile(); @@ -165,10 +164,10 @@ public void testLocalExists() throws Exception { localSource = fileCreator.getSource(); FileHandler handler = new LocalFileHandler(); - assert (handler.exists(localSource)); + assertTrue(handler.exists(localSource)); System.out.println("Testing testExists() with a non existing file"); - assert (!handler.exists("/usr/file_that_definitely_doesnot_exist.txt")); + assertFalse(handler.exists("/usr/file_that_definitely_doesnot_exist.txt")); // delete the dummy file fileCreator.deleteLocalSource(); @@ -184,6 +183,7 @@ public void testLocalExists() throws Exception { * underneath * @return - boolean - true if everything deleted, false if not */ + @SuppressWarnings("unused") private boolean deleteLocalDirectory(File directory) { File[] contents = directory.listFiles(); if (contents != null) { diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalMoveFileCommandTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalMoveFileCommandTest.java index b8c69f293..dcef55102 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalMoveFileCommandTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/LocalMoveFileCommandTest.java @@ -11,6 +11,9 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + import java.io.IOException; import java.nio.file.FileSystems; import java.nio.file.Files; @@ -137,7 +140,7 @@ public void testLocalMoveFileCommand() throws Exception { // Make the command LocalMoveFileCommand command = new LocalMoveFileCommand(); command.setConfiguration(source, dest); - CommandStatus status = command.execute(); + command.execute(); String filename = ""; if (System.getProperty("os.name").toLowerCase().contains("win")) { filename = source.substring(source.lastIndexOf("\\")); @@ -146,7 +149,7 @@ public void testLocalMoveFileCommand() throws Exception { } // Check if the path exists now Path path = Paths.get(dest + filename); - assert (Files.exists(path)); + assertTrue(Files.exists(path)); deleteFiles(); @@ -177,11 +180,11 @@ public void testLocalMoveSameDir() throws Exception { command.setConfiguration(source, dest); CommandStatus status = command.execute(); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Check that the file exists with the new name Path path = Paths.get(dest); - assert (Files.exists(path)); + assertTrue(Files.exists(path)); deleteFiles(); } @@ -203,11 +206,11 @@ public void testLocalMoveNewName() throws Exception { command.setConfiguration(source, dest); CommandStatus status = command.execute(); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Check if the path exists now Path path = Paths.get(dest); - assert (Files.exists(path)); + assertTrue(Files.exists(path)); deleteFiles(); // Delete the extra directory diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteCommandTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteCommandTest.java index 74e30aa27..b3b93d8cc 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteCommandTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteCommandTest.java @@ -11,6 +11,8 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertEquals; + import java.io.File; import java.io.IOException; import java.util.ArrayList; @@ -29,8 +31,6 @@ import org.junit.BeforeClass; import org.junit.Test; -import com.jcraft.jsch.JSchException; - /** * Test for class {@link org.eclipse.ice.commands.RemoteCommand}. * @@ -167,7 +167,7 @@ public void testRemoteCommand() { CommandStatus status = command.getStatus(); // Check that the return is processing, i.e. we are ready to execute - assert (status == CommandStatus.PROCESSING); + assertEquals(CommandStatus.PROCESSING, status); System.out.println("Finished remote command configuration test."); } @@ -177,7 +177,7 @@ public void testRemoteCommand() { * established. Expect an exception since the connection will not be able to be * established. */ - @Test(expected = NullPointerException.class) + @Test(expected = IllegalArgumentException.class) public void testFailedConnectionRemoteCommand() { System.out.println("Testing remote command with a bad connection"); @@ -196,7 +196,7 @@ public void testFailedConnectionRemoteCommand() { RemoteCommand command = new RemoteCommand(commandConfig, cfg, null); // Check that the command gives an error in its status due to poor connection - assert (command.getStatus() == CommandStatus.INFOERROR); + assertEquals(CommandStatus.INFOERROR, command.getStatus()); } /** @@ -208,7 +208,7 @@ public void testFailedConnectionRemoteCommand() { * @throws JSchException */ @Test(expected = NullPointerException.class) - public void testBadExecute() throws JSchException { + public void testBadExecute() throws IOException { CommandConfiguration badConfig = new CommandConfiguration(); badConfig.setCommandId(24); @@ -238,7 +238,7 @@ public void testBadExecute() throws JSchException { RemoteCommand testCommand = new RemoteCommand(badConfig, cfg, null); - CommandStatus testStatus = testCommand.execute(); + testCommand.execute(); } @@ -255,7 +255,7 @@ public void testExecute() { CommandStatus status = command.execute(); // Check that the command was successfully completed - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); System.out.println("Finished testing remote command execute"); } @@ -284,7 +284,7 @@ public void testLongRemoteJob() { RemoteCommand testCommand = new RemoteCommand(longConfig, connectConfig, null); - CommandStatus testStatus = testCommand.execute(); + testCommand.execute(); } diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteCopyFileCommandTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteCopyFileCommandTest.java index f5cab53ea..01f82fc85 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteCopyFileCommandTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteCopyFileCommandTest.java @@ -11,10 +11,15 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import org.apache.sshd.client.subsystem.sftp.SftpClient; import org.eclipse.ice.commands.CommandStatus; import org.eclipse.ice.commands.ConnectionManager; import org.eclipse.ice.commands.ConnectionManagerFactory; @@ -24,9 +29,6 @@ import org.junit.BeforeClass; import org.junit.Test; -import com.jcraft.jsch.ChannelSftp; -import com.jcraft.jsch.SftpException; - /** * Test for class {@link org.eclipse.ice.commands.RemoteCopyFileCommand}. * @@ -96,11 +98,11 @@ public void testRemoteCopyFileCommand() throws Exception { CommandStatus status = command.execute(); // Assert that the command status was properly configured - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Assert that the command was actually successful and that command status // wasn't inadvertently set to successful - assert (remotePathExists()); + assertTrue(remotePathExists()); // Delete the temporary files that were created to test handlerTest.deleteRemoteSource(); @@ -133,11 +135,11 @@ public void testRemoteCopyFileCommandDownload() throws Exception { CommandStatus status = command.execute(); // Assert that the command status was properly configured - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Assert that the command was actually successful and that command status // wasn't inadvertently set to successful - assert (localPathExists()); + assertTrue(localPathExists()); // Delete the temporary files that were created to test handlerTest.deleteRemoteSource(); @@ -170,11 +172,11 @@ public void testRemoteCopyFileCommandUpload() throws Exception { // execute the command CommandStatus status = command.execute(); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Check that the path exists try { - assert (remotePathExists()); + assertTrue(remotePathExists()); } catch (Exception e) { e.printStackTrace(); } @@ -202,11 +204,11 @@ public boolean localPathExists() { public boolean remotePathExists() throws Exception { // Get the connected channel from the connection - ChannelSftp sftpChannel = handlerTest.getConnection().getSftpChannel(); + SftpClient sftpChannel = handlerTest.getConnection().getSftpChannel(); try { sftpChannel.lstat(dest); - } catch (SftpException e) { + } catch (IOException e) { // If an exception is caught, this means the file was not there return false; } diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteFileBrowserTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteFileBrowserTest.java index 1cab312c7..a2c619e90 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteFileBrowserTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteFileBrowserTest.java @@ -12,11 +12,23 @@ package org.eclipse.ice.tests.commands; -import java.io.File; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.FileInputStream; import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.nio.file.FileSystems; +import java.nio.file.Path; import java.util.ArrayList; -import java.util.Collection; +import org.apache.sshd.client.subsystem.sftp.SftpClient; +import org.apache.sshd.client.subsystem.sftp.SftpClient.DirEntry; +import org.apache.sshd.client.subsystem.sftp.SftpClient.OpenMode; +import org.apache.sshd.client.subsystem.sftp.SftpClientFactory; +import org.apache.sshd.common.subsystem.sftp.SftpConstants; +import org.apache.sshd.common.subsystem.sftp.SftpException; import org.eclipse.ice.commands.Connection; import org.eclipse.ice.commands.ConnectionAuthorizationHandler; import org.eclipse.ice.commands.ConnectionAuthorizationHandlerFactory; @@ -25,18 +37,10 @@ import org.eclipse.ice.commands.ConnectionManagerFactory; import org.eclipse.ice.commands.RemoteFileBrowser; import org.eclipse.ice.commands.RemoteFileHandler; -import org.junit.After; import org.junit.AfterClass; -import org.junit.Before; import org.junit.BeforeClass; import org.junit.Test; -import com.jcraft.jsch.ChannelExec; -import com.jcraft.jsch.ChannelSftp; -import com.jcraft.jsch.JSchException; -import com.jcraft.jsch.SftpATTRS; -import com.jcraft.jsch.SftpException; - /** * This class tests the RemoteFileBrowser and associated functionality * @@ -67,8 +71,7 @@ public static void setUpBeforeClass() throws Exception { fileTransferConn = manager.openConnection(config); - fileTransferConn.setSftpChannel(fileTransferConn.getSession().openChannel("sftp")); - fileTransferConn.getSftpChannel().connect(); + fileTransferConn.setSftpChannel(SftpClientFactory.instance().createSftpClient(fileTransferConn.getSession())); // To set up the file creator information for use later RemoteFileHandlerTest.setUpBeforeClass(); @@ -115,9 +118,8 @@ public void testRemoteBrowsing() throws Exception { * Test for file browsing on remote system * * @throws IOException - * @throws SftpException */ - public void testRemoteFileBrowsing(String topDirectory) throws IOException, SftpException { + public void testRemoteFileBrowsing(String topDirectory) throws IOException { RemoteFileHandler handler = new RemoteFileHandler(); RemoteFileBrowser browser = new RemoteFileBrowser(fileTransferConn, topDirectory); @@ -128,19 +130,15 @@ public void testRemoteFileBrowsing(String topDirectory) throws IOException, Sftp // files should only be 4 entries since there are only 4 files in the tree // structure we created - assert (files.size() == 4); + assertEquals(4, files.size()); + SftpClient channel = fileTransferConn.getSftpChannel(); for (int i = 0; i < files.size(); i++) { // Assert that the file exists - assert (handler.exists(files.get(i))); + assertTrue(handler.exists(files.get(i))); - ChannelSftp channel = fileTransferConn.getSftpChannel(); - Collection structure = channel.ls(files.get(i)); // This should just be a one file vector, since we are ls-ing the filename - for (ChannelSftp.LsEntry filename : structure) { - // Assert that the file is indeed a regular file - assert (filename.getAttrs().isReg()); - } + assertTrue(channel.stat(files.get(i)).isRegularFile()); } } @@ -148,9 +146,8 @@ public void testRemoteFileBrowsing(String topDirectory) throws IOException, Sftp * Test for directory browsing on remote system * * @throws IOException - * @throws SftpException */ - public void testRemoteDirectoryBrowsing(String topDirectory) throws IOException, SftpException { + public void testRemoteDirectoryBrowsing(String topDirectory) throws IOException { RemoteFileHandler handler = new RemoteFileHandler(); handler.setConnectionConfiguration(fileTransferConn.getConfiguration()); @@ -160,25 +157,23 @@ public void testRemoteDirectoryBrowsing(String topDirectory) throws IOException, // directories should only be 3 entries since there are only 3 directories in // the tree structure we created - assert (files.size() == 3); + assertEquals(3, files.size()); // Use the default separator of the remote dummy system String separator = "/"; for (int i = 0; i < files.size(); i++) { - assert (handler.exists(files.get(i))); - ChannelSftp channel = fileTransferConn.getSftpChannel(); + assertTrue(handler.exists(files.get(i))); + SftpClient channel = fileTransferConn.getSftpChannel(); // ls the previous directory so that we can look at the subdirectories - Collection structure = channel - .ls(files.get(i).substring(0, files.get(i).lastIndexOf(separator))); // Iterate through the ls - for (ChannelSftp.LsEntry filename : structure) { + for (DirEntry filename : channel.readDir(files.get(i).substring(0, files.get(i).lastIndexOf(separator)))) { // Check that the ls filename is actually in the directory list (and thus is a // directory) if (files.get(i).contains(filename.getFilename())) { // assert that it is a directory - assert (filename.getAttrs().isDir()); + assertTrue(filename.getAttributes().isDirectory()); } } } @@ -191,42 +186,63 @@ public void testRemoteDirectoryBrowsing(String topDirectory) throws IOException, * * @param - topDirectory - refers to top directory whose contents will hold the * dummy directories/files - * @throws SftpException + * @throws Exception */ - protected void createRemoteFileStructure(String topDirectory) throws Exception, SftpException { + protected void createRemoteFileStructure(String topDirectory) throws Exception { - ChannelSftp sftpChannel = fileTransferConn.getSftpChannel(); + SftpClient sftpChannel = fileTransferConn.getSftpChannel(); // Check if the directory already exists - SftpATTRS attrs = null; try { - attrs = sftpChannel.lstat(topDirectory); + sftpChannel.lstat(topDirectory); } catch (Exception e) { System.out.println("Remote directory not found, trying to make it"); - } - if (attrs == null) { // Remote directory doesn't exist, so make it // Create a remote source directory sftpChannel.mkdir(topDirectory); } // make another directory in top directory - sftpChannel.cd(topDirectory); - sftpChannel.mkdir("dir1"); - sftpChannel.mkdir("dir2"); - sftpChannel.mkdir("dir3"); + sftpChannel.mkdir(topDirectory + "/dir1"); + sftpChannel.mkdir(topDirectory + "/dir2"); + sftpChannel.mkdir(topDirectory + "/dir3"); // create a local file to put there fileCreator.createLocalSource(); // Get the filename that was just created String filename = fileCreator.getSource(); // put it in a few places in the directory structure - sftpChannel.put(filename, topDirectory); - sftpChannel.put(filename, topDirectory + "/dir1/"); - sftpChannel.put(filename, topDirectory + "/dir3/"); - sftpChannel.put(filename, topDirectory + "/dir3/newfile.txt"); + putFile(sftpChannel, filename, topDirectory); + putFile(sftpChannel, filename, topDirectory + "/dir1/"); + putFile(sftpChannel, filename, topDirectory + "/dir3/"); + putFile(sftpChannel, filename, topDirectory + "/dir3/newfile.txt"); } + + private void putFile(SftpClient client, String src, String dest) throws IOException { + try { + if (client.stat(dest).isDirectory()) { + Path path = FileSystems.getDefault().getPath(src); + String sep = ""; + if (!dest.endsWith("/")) { + sep = "/"; + } + dest += sep + path.getFileName(); + } + } catch (SftpException e) { + if (!(e.getStatus() == SftpConstants.SSH_FX_NO_SUCH_FILE)) { + throw e; + } + } + try (OutputStream dstStream = client.write(dest, OpenMode.Create, OpenMode.Write, OpenMode.Truncate)) { + try (InputStream srcStream = new FileInputStream(src)) { + byte[] buf = new byte[32*1024]; + while (srcStream.read(buf) > 0) { + dstStream.write(buf); + } + } + } + } /** * Function that deletes the remote file structure tree created for testing file @@ -234,12 +250,11 @@ protected void createRemoteFileStructure(String topDirectory) throws Exception, * * @param - topDirectory - refers to top directory whose contents hold the dummy * directories/files - * @throws JSchException - * @throws SftpException + * @throws IOException */ - protected void deleteRemoteFileStructure(String topDirectory) throws SftpException, JSchException { + protected void deleteRemoteFileStructure(String topDirectory) throws IOException { // Connect the channel from the connection - ChannelSftp sftpChannel = fileTransferConn.getSftpChannel(); + SftpClient sftpChannel = fileTransferConn.getSftpChannel(); System.out.println("Deleting remote destination at : " + topDirectory); // Recursively delete the directory and its contents @@ -252,20 +267,16 @@ protected void deleteRemoteFileStructure(String topDirectory) throws SftpExcepti * * @param sftpChannel * @param path - * @throws SftpException - * @throws JSchException + * @throws IOException */ - private void deleteRemoteDirectory(ChannelSftp sftpChannel, String path) throws SftpException, JSchException { - - // Get the path's directory structure - Collection fileList = sftpChannel.ls(path); + private void deleteRemoteDirectory(SftpClient sftpChannel, String path) throws IOException { // Iterate through the list to get the file/directory names - for (ChannelSftp.LsEntry file : fileList) { + for (DirEntry file : sftpChannel.readDir(path)) { // If it isn't a directory delete it - if (!file.getAttrs().isDir()) { + if (!file.getAttributes().isDirectory()) { // Can use / here because we know the dummy directory is on linux, not windows - sftpChannel.rm(path + "/" + file.getFilename()); + sftpChannel.remove(path + "/" + file.getFilename()); } else if (!(".".equals(file.getFilename()) || "..".equals(file.getFilename()))) { // If it is a subdir. // Otherwise its a subdirectory, so try deleting it try { @@ -280,13 +291,10 @@ private void deleteRemoteDirectory(ChannelSftp sftpChannel, String path) throws } try { sftpChannel.rmdir(path); // delete the parent directory after empty - } catch (SftpException e) { + } catch (IOException e) { // If the sftp channel can't delete it, just manually rm it with a execution // channel - ChannelExec execChannel = (ChannelExec) fileTransferConn.getSession().openChannel("exec"); - execChannel.setCommand("rm -r " + path); - execChannel.connect(); - execChannel.disconnect(); + fileTransferConn.getSession().executeRemoteCommand("rm -r " + path); } } diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteFileHandlerTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteFileHandlerTest.java index ae51b6c4e..6a9041b83 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteFileHandlerTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteFileHandlerTest.java @@ -11,16 +11,26 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + import java.io.File; +import java.io.FileInputStream; import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; import java.nio.file.DirectoryNotEmptyException; import java.nio.file.FileSystems; import java.nio.file.Files; import java.nio.file.NoSuchFileException; import java.nio.file.Path; import java.nio.file.Paths; -import java.util.Collection; +import org.apache.sshd.client.subsystem.sftp.SftpClient; +import org.apache.sshd.client.subsystem.sftp.SftpClient.DirEntry; +import org.apache.sshd.client.subsystem.sftp.SftpClient.OpenMode; +import org.apache.sshd.client.subsystem.sftp.SftpClientFactory; import org.eclipse.ice.commands.CommandStatus; import org.eclipse.ice.commands.Connection; import org.eclipse.ice.commands.ConnectionAuthorizationHandler; @@ -33,12 +43,6 @@ import org.junit.BeforeClass; import org.junit.Test; -import com.jcraft.jsch.ChannelExec; -import com.jcraft.jsch.ChannelSftp; -import com.jcraft.jsch.JSchException; -import com.jcraft.jsch.SftpATTRS; -import com.jcraft.jsch.SftpException; - /** * This class tests remote file handling capabilities * @@ -76,9 +80,7 @@ public static void setUpBeforeClass() throws Exception { fileTransferConn = manager.openConnection(config); - fileTransferConn.setSftpChannel(fileTransferConn.getSession().openChannel("sftp")); - fileTransferConn.getSftpChannel().connect(); - + fileTransferConn.setSftpChannel(SftpClientFactory.instance().createSftpClient(fileTransferConn.getSession())); } /** @@ -120,9 +122,9 @@ public void testSetConnection() throws Exception { handler.setConnectionConfiguration(config); - assert (handler.getConnection().getSession().isConnected()); + assertTrue(handler.getConnection().getSession().isOpen()); - assert (handler.getConnection().getSftpChannel().isConnected()); + assertTrue(handler.getConnection().getSftpChannel().isOpen()); System.out.println("all finished testing set connection"); } @@ -133,8 +135,8 @@ public void testSetNewConnection() throws Exception { RemoteFileHandler handler = new RemoteFileHandler(); handler.setConnectionConfiguration(config); - assert (handler.getConnection().getSession().isConnected()); - assert (handler.getConnection().getSftpChannel().isConnected()); + assertTrue(handler.getConnection().getSession().isOpen()); + assertTrue(handler.getConnection().getSftpChannel().isOpen()); ConnectionManagerFactory.getConnectionManager().removeConnection(config.getName()); } @@ -155,9 +157,9 @@ public void testRemoteExists() throws Exception { // Check two asserts - that the created file exists, and that // some other nonexistent file throws an error - assert (handler.exists(theSource)); + assertTrue(handler.exists(theSource)); - assert (!handler.exists("/some/nonexistent/path/file.txt")); + assertFalse(handler.exists("/some/nonexistent/path/file.txt")); // Done with the remote source, delete it deleteRemoteSource(); @@ -166,7 +168,7 @@ public void testRemoteExists() throws Exception { // returns true createLocalSource(); - assert (handler.exists(theSource)); + assertTrue(handler.exists(theSource)); // Done with the local source, delete it deleteLocalSource(); @@ -193,7 +195,7 @@ public void testRemoteToLocalMove() throws Exception { // Now try to move the file CommandStatus status = handler.move(src, theDestination); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); deleteLocalDestination(); deleteRemoteSource(); @@ -221,14 +223,14 @@ public void testLocalToRemoteMove() throws Exception { // Now try to move the file CommandStatus status = handler.move(theSource, theDestination); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Lets try a file move also where we change the name of the file // and add an additional directory String dest = theDestination + "newDirectory/newFileName.txt"; status = handler.move(theSource, dest); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Delete the test file/directory now that the test is finished deleteLocalSource(); @@ -250,13 +252,10 @@ public void testRemoteToRemoteCopy() throws Exception { handler.setConnectionConfiguration(fileTransferConn.getConfiguration()); String src = theSource; - // Get the filename to check for existence - String filename = src.substring(src.lastIndexOf("/")); - // Now try to copy the file CommandStatus status = handler.copy(src, theDestination); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); deleteRemoteDestination(); deleteRemoteSource(); @@ -279,7 +278,7 @@ public void testRemoteToRemoteMove() throws Exception { // Now try to move the file CommandStatus status = handler.move(src, theDestination); - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); deleteRemoteDestination(); deleteRemoteSource(); @@ -427,18 +426,14 @@ public void deleteLocalDestination() throws Exception { */ public void createRemoteSource() throws Exception { - ChannelSftp sftpChannel = fileTransferConn.getSftpChannel(); + SftpClient sftpChannel = fileTransferConn.getSftpChannel(); String remoteDest = "/tmp/remoteFileHandlerSource/"; - // Check if the directory already exists - SftpATTRS attrs = null; try { - attrs = sftpChannel.lstat(remoteDest); + sftpChannel.lstat(remoteDest); } catch (Exception e) { System.out.println("Remote directory not found, trying to make it"); - } - if (attrs == null) { // Remote directory doesn't exist, so make it // Create a remote source directory sftpChannel.mkdir(remoteDest); @@ -458,7 +453,7 @@ public void createRemoteSource() throws Exception { String filename = tokens[tokens.length - 1]; // Move it to the remote host - sftpChannel.put(theSource, remoteDest); + putFile(sftpChannel, theSource, remoteDest); // Delete the local directory that was created since it is no longer needed Path path = Paths.get(theSource); @@ -475,6 +470,24 @@ public void createRemoteSource() throws Exception { System.out.println("Moved source file to new remote source destination " + theSource); } + + private void putFile(SftpClient client, String src, String dest) throws IOException { + if (dest.endsWith("/")) { + String separator = FileSystems.getDefault().getSeparator(); + if (System.getProperty("os.name").toLowerCase().contains("win")) + separator += "\\"; + String[] tokens = src.split(separator); + dest += tokens[tokens.length - 1]; + } + try (OutputStream dstStream = client.write(dest, OpenMode.Create, OpenMode.Write, OpenMode.Truncate)) { + try (InputStream srcStream = new FileInputStream(src)) { + byte[] buf = new byte[32*1024]; + while (srcStream.read(buf) > 0) { + dstStream.write(buf); + } + } + } + } /** * This function deletes the remote source that was created for testing @@ -483,7 +496,7 @@ public void createRemoteSource() throws Exception { */ public void deleteRemoteSource() throws Exception { // Connect the channel from the connection - ChannelSftp sftpChannel = fileTransferConn.getSftpChannel(); + SftpClient sftpChannel = fileTransferConn.getSftpChannel(); // Get the path to the source file // Leave this as unix command since the remote system is unix @@ -508,18 +521,15 @@ public void deleteRemoteSource() throws Exception { */ public void createRemoteDestination() throws Exception { // Connect the channel from the connection - ChannelSftp sftpChannel = fileTransferConn.getSftpChannel(); + SftpClient sftpChannel = fileTransferConn.getSftpChannel(); String remoteDest = "/tmp/remoteFileHandlerDestination/"; // Check if the directory already exists - SftpATTRS attrs = null; try { - attrs = sftpChannel.lstat(remoteDest); + sftpChannel.lstat(remoteDest); } catch (Exception e) { System.out.println("Remote directory not found, trying to make it"); - } - if (attrs == null) { // Remote directory doesn't exist, so make it // Create a remote source directory sftpChannel.mkdir(remoteDest); @@ -539,7 +549,7 @@ public void createRemoteDestination() throws Exception { */ public void deleteRemoteDestination() throws Exception { // Connect the channel from the connection - ChannelSftp sftpChannel = fileTransferConn.getSftpChannel(); + SftpClient sftpChannel = fileTransferConn.getSftpChannel(); System.out.println("Deleting remote destination at : " + theDestination); // Recursively delete the directory and its contents @@ -570,20 +580,16 @@ private boolean deleteLocalDirectory(File directory) { * * @param sftpChannel * @param path - * @throws SftpException - * @throws JSchException + * @throws IOException */ - private void deleteRemoteDirectory(ChannelSftp sftpChannel, String path) throws SftpException, JSchException { - - // Get the path's directory structure - Collection fileList = sftpChannel.ls(path); + private void deleteRemoteDirectory(SftpClient sftpChannel, String path) throws IOException { // Iterate through the list to get the file/directory names - for (ChannelSftp.LsEntry file : fileList) { + for (DirEntry file : sftpChannel.readDir(path)) { // If it isn't a directory delete it - if (!file.getAttrs().isDir()) { + if (!file.getAttributes().isDirectory()) { // Can use / here because we know the dummy directory is on linux, not windows - sftpChannel.rm(path + "/" + file.getFilename()); + sftpChannel.remove(path + "/" + file.getFilename()); } else if (!(".".equals(file.getFilename()) || "..".equals(file.getFilename()))) { // If it is a subdir. // Otherwise its a subdirectory, so try deleting it try { @@ -598,13 +604,10 @@ private void deleteRemoteDirectory(ChannelSftp sftpChannel, String path) throws } try { sftpChannel.rmdir(path); // delete the parent directory after empty - } catch (SftpException e) { + } catch (IOException e) { // If the sftp channel can't delete it, just manually rm it with a execution // channel - ChannelExec execChannel = (ChannelExec) fileTransferConn.getSession().openChannel("exec"); - execChannel.setCommand("rm -r " + path); - execChannel.connect(); - execChannel.disconnect(); + fileTransferConn.getSession().executeRemoteCommand("rm -r " + path); } } diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteMoveFileCommandTest.java b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteMoveFileCommandTest.java index 27b56c756..33f8e91e2 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteMoveFileCommandTest.java +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/RemoteMoveFileCommandTest.java @@ -11,10 +11,15 @@ *******************************************************************************/ package org.eclipse.ice.tests.commands; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import org.apache.sshd.client.subsystem.sftp.SftpClient; import org.eclipse.ice.commands.CommandStatus; import org.eclipse.ice.commands.ConnectionManager; import org.eclipse.ice.commands.ConnectionManagerFactory; @@ -24,9 +29,6 @@ import org.junit.BeforeClass; import org.junit.Test; -import com.jcraft.jsch.ChannelSftp; -import com.jcraft.jsch.SftpException; - /** * Test for class {@link org.eclipse.ice.commands.RemoteMoveFileCommand}. * @@ -97,11 +99,11 @@ public void testRemoteMoveFileCommand() throws Exception { CommandStatus status = command.execute(); // Assert that the command status was properly configured - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); // Assert that the command was actually successful and that command status // wasn't inadvertently set to successful - assert (remotePathExists()); + assertTrue(remotePathExists()); // Delete the temporary files that were created to test // Don't need to delete the source since it was moved @@ -138,7 +140,7 @@ public void testRemoteMoveFileCommandDownload() throws Exception { // Assert that the command was actually successful and that command status // wasn't inadvertently set to successful - assert (localPathExists()); + assertTrue(localPathExists()); // Delete the temporary files that were created to test handlerTest.deleteRemoteSource(); @@ -171,15 +173,11 @@ public void testRemoteMoveFileCommandUpload() throws Exception { CommandStatus status = command.execute(); // Assert that the command status was properly configured - assert (status == CommandStatus.SUCCESS); + assertEquals(CommandStatus.SUCCESS, status); - try { - // Assert that the command was actually successful and that command status - // wasn't inadvertently set to successful - assert (remotePathExists()); - } catch (Exception e) { - e.printStackTrace(); - } + // Assert that the command was actually successful and that command status + // wasn't inadvertently set to successful + assertTrue(remotePathExists()); // Delete the temporary files that were created to test handlerTest.deleteLocalSource(); @@ -196,11 +194,11 @@ public void testRemoteMoveFileCommandUpload() throws Exception { public boolean remotePathExists() throws Exception { // Connect the channel from the connection - ChannelSftp sftpChannel = handlerTest.getConnection().getSftpChannel(); + SftpClient sftpChannel = handlerTest.getConnection().getSftpChannel(); try { sftpChannel.lstat(dest); - } catch (SftpException e) { + } catch (IOException e) { // If an exception is caught, this means the file was not there. return false; } diff --git a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/test_python_script.py b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/test_python_script.py index 32cd36caa..044a0632c 100644 --- a/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/test_python_script.py +++ b/org.eclipse.ice.commands/src/test/java/org/eclipse/ice/tests/commands/test_python_script.py @@ -21,9 +21,9 @@ print("This is a hello world python script") f = open(filename1,'r') +print (f.read()) +f.close() f2 = open(filename2,'r'); - -print(f.read()) - print(f2.read()) +f2.close() diff --git a/org.eclipse.ice.demo/META-INF/MANIFEST.MF b/org.eclipse.ice.demo/META-INF/MANIFEST.MF index 5d220fc50..d585add49 100644 --- a/org.eclipse.ice.demo/META-INF/MANIFEST.MF +++ b/org.eclipse.ice.demo/META-INF/MANIFEST.MF @@ -7,7 +7,8 @@ Bundle-Vendor: Oak Ridge National Laboratory Require-Bundle: org.eclipse.ui, org.eclipse.january.geometry.model Bundle-RequiredExecutionEnvironment: JavaSE-1.8 -Import-Package: org.eclipse.core.resources, +Import-Package: org.apache.sshd.client.subsystem.sftp;version="2.0.0", + org.eclipse.core.resources, org.eclipse.core.runtime, org.eclipse.core.runtime.jobs, org.eclipse.e4.core.contexts, @@ -15,8 +16,8 @@ Import-Package: org.eclipse.core.resources, org.eclipse.e4.core.di.annotations, org.eclipse.e4.ui.model.application, org.eclipse.e4.ui.model.application.ui, - org.eclipse.eavp.viz.datastructures.ui, org.eclipse.eavp.viz.datastructures.VizObject, + org.eclipse.eavp.viz.datastructures.ui, org.eclipse.eavp.viz.modeling, org.eclipse.eavp.viz.modeling.base, org.eclipse.eavp.viz.service, diff --git a/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/CommandFactoryExample.java b/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/CommandFactoryExample.java index a445f8d6d..0b7750fad 100644 --- a/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/CommandFactoryExample.java +++ b/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/CommandFactoryExample.java @@ -111,7 +111,7 @@ private static void runRemoteCommand() { CommandStatus status = command.execute(); // Ensure it finished properly - assert (status == CommandStatus.SUCCESS); + assert (status.equals(CommandStatus.SUCCESS)); // Clear out the remaining connections to start fresh next example ConnectionManagerFactory.getConnectionManager().removeAllConnections(); @@ -147,7 +147,7 @@ private static void runRemoteShellCommand() { CommandStatus status = command.execute(); - assert (status == CommandStatus.SUCCESS); + assert (status.equals(CommandStatus.SUCCESS)); String output = config.getStdOutputString(); @@ -182,8 +182,12 @@ private static ConnectionConfiguration makeDumConnectionConfig() { ConnectionAuthorizationHandlerFactory authFactory = new ConnectionAuthorizationHandlerFactory(); // Request a ConnectionAuthorization of type text file which contains the // dummy remote host credentials + String credPath = "/tmp/ice-remote-creds.txt"; + if(System.getProperty("os.name").toLowerCase().contains("win")) + credPath = "C:\\Users\\Administrator\\ice-remote-creds.txt"; ConnectionAuthorizationHandler auth = authFactory.getConnectionAuthorizationHandler("text", - "/tmp/ice-remote-creds.txt"); + credPath); + /** * Alternatively, one can authorize with their password at the console line by * performing the following set of code @@ -191,6 +195,9 @@ private static ConnectionConfiguration makeDumConnectionConfig() { * ConnectionAuthorizationHandler auth = * authFactory.getConnectionAuthorizationHandler("console"); * auth.setHostname("hostname"); auth.setUsername("username"); + * + * One can also use a keypath authorization, see ConnectionAuthorizationHandlerFactory + * for more details */ // Set it so that the connection can authorize itself connectionConfig.setAuthorization(auth); @@ -246,6 +253,8 @@ private static void runLocalCommand() { commandConfig.setWorkingDirectory(scriptDir); commandConfig.setAppendInput(true); commandConfig.setOS(System.getProperty("os.name")); + if(System.getProperty("os.name").toLowerCase().contains("win")) + commandConfig.setInterpreter("powershell.exe"); // Make a ConnectionConfiguration to indicate that we want to run locally ConnectionConfiguration connectionConfig = new ConnectionConfiguration(); @@ -269,7 +278,7 @@ private static void runLocalCommand() { // Run it CommandStatus status = localCommand.execute(); - assert (status == CommandStatus.SUCCESS); + assert (status.equals(CommandStatus.SUCCESS)); // Get a string of the output that is produced from the job String output = commandConfig.getStdOutputString(); @@ -317,8 +326,11 @@ private static void runPythonScript() { // Get the authorization type. In this case, local, which is basically // equivalent to // "no authorization" + String credPath = "/tmp/ice-remote-creds.txt"; + if(System.getProperty("os.name").toLowerCase().contains("win")) + credPath = "C:\\Users\\Administrator\\ice-remote-creds.txt"; ConnectionAuthorizationHandler auth = authFactory.getConnectionAuthorizationHandler("text", - "/tmp/ice-remote-creds.txt"); + credPath); // Set the connectionConfig to have access to e.g. the hostname connectionConfig.setAuthorization(auth); @@ -335,7 +347,7 @@ private static void runPythonScript() { CommandStatus status = command.execute(); - assert (status == CommandStatus.SUCCESS); + assert (status.equals(CommandStatus.SUCCESS)); // Clear out the remaining connections to start fresh next example ConnectionManagerFactory.getConnectionManager().removeAllConnections(); diff --git a/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/FileHandlerExample.java b/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/FileHandlerExample.java index a7596d712..6af2bb87b 100644 --- a/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/FileHandlerExample.java +++ b/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/FileHandlerExample.java @@ -13,6 +13,7 @@ import java.io.File; import java.io.IOException; +import java.nio.file.FileSystems; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; @@ -189,7 +190,8 @@ public static void copyFileRemotely() { ConnectionConfiguration configuration = makeConnectionConfiguration(); // Get the filename of the dummy file - String filename = localSource.substring(localSource.lastIndexOf("/")); + String separator = FileSystems.getDefault().getSeparator(); + String filename = localSource.substring(localSource.lastIndexOf(separator)); // Get the file handler factory to create the transfer FileHandlerFactory factory = new FileHandlerFactory(); @@ -284,8 +286,9 @@ public static void moveFileLocally() { if (status != CommandStatus.SUCCESS) { System.out.println("Move file failed! Check console for error messages"); } + String separator = FileSystems.getDefault().getSeparator(); // Check that it exists - String filename = localSource.substring(localSource.lastIndexOf("/")); + String filename = localSource.substring(localSource.lastIndexOf(separator)); boolean exist = handler.exists(localDestination + filename); } catch (IOException e) { @@ -388,8 +391,11 @@ private static ConnectionConfiguration makeConnectionConfiguration() { ConnectionAuthorizationHandlerFactory authFactory = new ConnectionAuthorizationHandlerFactory(); // Request a ConnectionAuthorization of type text file which contains the // credentials + String credPath = "/tmp/ice-remote-creds.txt"; + if(System.getProperty("os.name").toLowerCase().contains("win")) + credPath = "C:\\Users\\Administrator\\ice-remote-creds.txt"; ConnectionAuthorizationHandler auth = authFactory.getConnectionAuthorizationHandler("text", - "/tmp/ice-remote-creds.txt"); + credPath); // Set it config.setAuthorization(auth); diff --git a/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/MultiRemoteHostCommandExample.java b/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/MultiRemoteHostCommandExample.java new file mode 100644 index 000000000..b566b399d --- /dev/null +++ b/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/MultiRemoteHostCommandExample.java @@ -0,0 +1,168 @@ +/******************************************************************************* + * Copyright (c) 2019- UT-Battelle, LLC. + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Initial API and implementation and/or initial documentation - + * Jay Jay Billings, Joe Osborn + *******************************************************************************/ +package org.eclipse.ice.demo.commands; + +import java.io.IOException; + +import org.eclipse.ice.commands.Command; +import org.eclipse.ice.commands.CommandConfiguration; +import org.eclipse.ice.commands.CommandFactory; +import org.eclipse.ice.commands.CommandStatus; +import org.eclipse.ice.commands.ConnectionAuthorizationHandler; +import org.eclipse.ice.commands.ConnectionAuthorizationHandlerFactory; +import org.eclipse.ice.commands.ConnectionConfiguration; +import org.eclipse.ice.commands.FileHandlerFactory; +import org.eclipse.ice.commands.IFileHandler; + +/** + * This class demonstrates an example of executing a job on a remote host which + * executes a job on another remote host. For example, if Commands is running on + * host A, and the user desires to orchestrate the execution of a Command on + * host B to run on host C while operating the Commands package on host A. + * + * The idea here is to run a RemoteCommand from host A to host B, where the + * command is a script in which the script contains the logic to run a remote + * command from host B to host C. + * + * Note that one needs to set the host keys below to hosts that they actually have + * keys to in order for this example to work. + * + * @author Joe Osborn + * + */ +public class MultiRemoteHostCommandExample { + + /** + * Set these parameters to make the job run to hosts of your choosing + */ + static String hostB = "host"; + static String userB = "user"; + static String hostBKeyPath = "/some/path/to/key"; + static String hostC = "dummy@osbornjd-ice-host.ornl.gov"; + // This key exists on host B, connecting B to C + static String hostCKeyPath = "~/.ssh/dummyhostkey"; + + /** + * @param args + * @throws IOException + */ + public static void main(String[] args) throws IOException { + + /** + * We will run this command from host A (your local computer) such that it + * executes a Command on host B to be remotely executed on host C. In this case, + * host B will be osbornjd-ice-host.ornl.gov and host C will be Denisovan. + * + * Since the example assumes that you are using an arbitrary remote host, we + * must first create the files on host A and move them to host B. Nominally, + * these files would already exist on host B, but because of the arbitrary + * remote host assumption we put them there for you + */ + + // Now we will actually execute the remote command on remote host B, where in + // turn the script will execute a command on remote host C. + runRemoteCommand(); + + } + + /** + * This function sets up host B in the assumed configuration that it would + * actually be in. Since this example assumes that you are using an arbitrary + * remote host, the files to run this dummy job would not nominally be on that + * remote host. So we move them there in this function. + * + * @throws IOException + */ + private static void runRemoteCommand() throws IOException { + // Setup the connection to the host + ConnectionConfiguration connectionConfig = createConnection(); + + // Setup some strings of files to move + String pwd = System.getProperty("user.dir"); + String workingDir = pwd + "/src/org/eclipse/ice/demo/commands/"; + // This is the script to be run on host B. Need to move it from host A to B + // so that the environment is set to run the script which will run scriptName + // through Commands + String scriptName = "test_code_execution.sh"; + String inputFileName = "someInputFile.txt"; + String remoteWorkingDir = "/tmp/remoteRemoteCommand/"; + + // We need to explicitly move the script and input file to host B since we don't + // want it appended as an argument to the script running on remote host B. The + // script remoteCommand.sh will take care of that + FileHandlerFactory FHFactory = new FileHandlerFactory(); + IFileHandler handler = FHFactory.getFileHandler(connectionConfig); + + CommandStatus fileStatus = handler.move(workingDir + inputFileName, remoteWorkingDir); + + assert (fileStatus == CommandStatus.SUCCESS); + + fileStatus = handler.move(workingDir + scriptName, remoteWorkingDir); + + assert (fileStatus == CommandStatus.SUCCESS); + + CommandConfiguration commandConfig = new CommandConfiguration(); + commandConfig.setNumProcs("1"); + commandConfig.setInstallDirectory(""); + commandConfig.setWorkingDirectory(workingDir); + commandConfig.setOS(System.getProperty("os.name")); + commandConfig.setExecutable("./remoteCommand.sh"); + commandConfig.setAppendInput(true); + commandConfig.setCommandId(1); + commandConfig.setErrFileName("RemoteRemoteErr.txt"); + commandConfig.setOutFileName("RemoteRemoteOut.txt"); + commandConfig.setRemoteWorkingDirectory(remoteWorkingDir); + + // add arguments that remoteCommand.sh needs to run on host C + commandConfig.addArgument(hostC + " /tmp/remoteDir " + hostCKeyPath); + CommandFactory factory = new CommandFactory(); + Command remoteCommand = null; + try { + remoteCommand = factory.getCommand(commandConfig, connectionConfig); + } catch (IOException e) { + e.printStackTrace(); + } + + CommandStatus status = remoteCommand.execute(); + + assert (status == CommandStatus.SUCCESS); + + + } + + /** + * Create a connection to the host B + */ + private static ConnectionConfiguration createConnection() { + + ConnectionConfiguration connectionConfig = new ConnectionConfiguration(); + // Set the connection configuration to a dummy remote connection + // Get a factory which determines the type of authorization + ConnectionAuthorizationHandlerFactory authFactory = new ConnectionAuthorizationHandlerFactory(); + // Request a ConnectionAuthorization of type text file which contains the + // dummy remote host credentials + ConnectionAuthorizationHandler auth = authFactory.getConnectionAuthorizationHandler("keypath", + hostBKeyPath); + auth.setHostname(hostB); + auth.setUsername(userB); + // Set it so that the connection can authorize itself + connectionConfig.setAuthorization(auth); + // Give the connection a name + connectionConfig.setName("PCConnection"); + + // Delete the remote working directory once we are finished running the job + connectionConfig.deleteWorkingDirectory(true); + + return connectionConfig; + } + +} diff --git a/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/remoteCommand.sh b/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/remoteCommand.sh new file mode 100755 index 000000000..32a382bad --- /dev/null +++ b/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/remoteCommand.sh @@ -0,0 +1,33 @@ +#/******************************************************************************* +# * Copyright (c) 2019- UT-Battelle, LLC. +# * All rights reserved. This program and the accompanying materials +# * are made available under the terms of the Eclipse Public License v1.0 +# * which accompanies this distribution, and is available at +# * http://www.eclipse.org/legal/epl-v10.html +# * +# * Contributors: +# * Initial API and implementation and/or initial documentation - +# * Jay Jay Billings, Joe Osborn +# *******************************************************************************/ + +#!/bin/bash + +script="test_code_execution.sh" +inputFile="someInputFile.txt" +userhostname=$1 +hostCdirectory=$2 +keypath=$3 + +# Make a directory to work in on the server +ssh -i $keypath $userhostname "mkdir $2" + +# scp the files to that directory +scp -i $keypath $inputFile $userhostname:$2 +scp -i $keypath $script $userhostname:$2 + + +# Run the script which executes the job +ssh -i $keypath $userhostname "cd $2; chmod 700 $script; ./$script $inputFile;" + +# Delete the directory when you are finished running it +ssh -i $keypath $userhostname "rm -rf $2" diff --git a/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/someInputFile.txt b/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/someInputFile.txt new file mode 100644 index 000000000..46c6ce01c --- /dev/null +++ b/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/someInputFile.txt @@ -0,0 +1,3 @@ +1 2 3 4 5 +2 3 4 5 6 +3 4 5 6 7 diff --git a/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/test_code_execution.sh b/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/test_code_execution.sh new file mode 100755 index 000000000..ea1bfba55 --- /dev/null +++ b/org.eclipse.ice.demo/src/org/eclipse/ice/demo/commands/test_code_execution.sh @@ -0,0 +1,49 @@ +#!/bin/bash +#******************************************************************************* +# * Copyright (c) 2019- UT-Battelle, LLC. +# * All rights reserved. This program and the accompanying materials +# * are made available under the terms of the Eclipse Public License v1.0 +# * which accompanies this distribution, and is available at +# * http://www.eclipse.org/legal/epl-v10.html +# * +# * Contributors: +# * Initial API and implementation and initial documentation - +# * Joe Osborn +# ***************************************************************************** + + + +# This is a test executable shell script for running a job locally. +# It is for testing the LocalCommand chain of execution. +# The script takes some input from a dummy text file and prints it +# to the screen. It is in no way intended to be robust, just to show +# basic usage of an executable command functioning. + + +# Print a boring string out to the console screen +STRING="This is a hello world executable" +echo $STRING + +# list the contents of this directory +ls -lrt + +# Read in a dummy file +echo "I will read from files "$1 "and "$2 + +# Set a variable to the argument value +input=$1 +otherinput=$2 + +# Read in the data +#while IFS= read -r lineA +#do +# echo "$lineA" +#done < "$input" + +paste $input $otherinput | while IFS="$(printf '\t')" read -r f1 f2 +do + printf 'first file: %s\n' "$f1" + printf 'second file: %s\n' "$f2" +done + +