A middleware abstraction library that provides a simple programming interface to various compute and storage resources.
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
config Added task to check files for license header. Nov 17, 2015
docs prepare for 2.6.2 release Jun 11, 2018
gradle Enable javadoc linting Nov 22, 2018
legal updated legal overview Oct 4, 2017
lib Added support for SSH tunneling. Fixes #575 Oct 25, 2017
src Revert startSingleProcess needs more work Nov 22, 2018
.gitignore Merge branch 'xenon-3.0.0-attempt2' into merge-2-and-3 Jun 8, 2018
.travis.yml Make AdaptorDocGenerator html5 compliant Nov 22, 2018
.zenodo.json added a .zenodo.json file to control metadata on Zenodo Mar 15, 2018
CHANGELOG.md prepare for 2.6.2 release Jun 11, 2018
CITATION.cff added conceptdoi as doi to avoid errors on research-software.nl Mar 19, 2018
CODE_OF_CONDUCT.md added code of conduct. partly fixes issue #442 Jun 21, 2017
CONTRIBUTING.md Fix GitHub Pages build error + updates to release docs Oct 25, 2017
LICENSE license text of appendix should not be changed to include the name of… Oct 4, 2017
NOTICE added a copyright notice for Xenon to the NOTICE file Oct 4, 2017
README.md Merged origin/master Oct 25, 2018
ROADMAP.md Update ROADMAP.md Oct 5, 2017
TESTING.md Ignore */build Feb 5, 2018
appveyor.yml appveyor set java home before build Nov 22, 2018
build.gradle Merge branch 'at-adaptor-master' into at-adaptor Nov 20, 2018
gradlew Update of gradle to 4.1. Removed println Aug 16, 2017
gradlew.bat Update of gradle to 4.1. Removed println Aug 16, 2017
jitpack.yml Explain what jitpack config file is for Jul 24, 2017
runAllChecks.sh Fixed javadoc devel errors Jul 26, 2018
sonar-project.properties Update sonarcloud config Jun 14, 2018

README.md

Xenon

Build Status Build status Windows codecov.io SonarCloud GitHub license Download DOI Research Software Directory

Copyright 2013-2018 The Netherlands eScience Center

What problem does Xenon solve?

Many applications use remote storage and compute resources. To do so, they need to include code to interact with the scheduling systems and file transfer protocols used on those remote machines.

Unfortunately, many different scheduler systems and file transfer protocols exist, often with completely different programming interfaces. This makes it hard for applications to switch to a different system or support multiple remote systems simultaneously.

Xenon solves this problem by providing a single programming interface to many different types of remote resources, allowing applications to switch without changing a single line of code.

Xenon abstraction

How does Xenon work?

Xenon is an abstraction layer that sits between your application and the remote resource it uses. Xenon is written in Java, but is also accessible from other languages (e.g. Python) through its gRPC interface.

Xenon API

Overview of the Xenon ecosystem of tools

component repository
Xenon library https://github.com/NLeSC/Xenon
Xenon cloud adaptors like s3 https://github.com/NLeSC/xenon-adaptors-cloud
Xenon grid adaptors like gridftp https://github.com/NLeSC/xenon-adaptors-grid
Xenon hadoop adaptors like hdfs https://github.com/NLeSC/xenon-adaptors-hadoop
gRPC extension for Xenon https://github.com/NLeSC/xenon-grpc
command line interface to Xenon https://github.com/NLeSC/xenon-cli
Python API for Xenon https://github.com/NLeSC/pyxenon
Docker images https://github.com/NLeSC/xenon-docker-images

Supported middleware

Xenon currently supports the following file access mechanisms:

  • file (local file manipulation)
  • ftp
  • sftp
  • webdav
  • s3
  • hdfs

Xenon currently supports the following job submission mechanisms:

  • local
  • ssh
  • gridengine
  • slurm
  • torque

See the roadmap for the planned extensions.

Adding Xenon as a dependency to your project

Follow the instructions from bintray.com to include Xenon as a dependency for Gradle, Maven, SBT, or Leiningen projects, e.g. Gradle:

	allprojects {
		repositories {
			...
			jcenter()
		}
	}

and

	dependencies {
	        compile group: 'nl.esciencecenter.xenon', name: 'xenon', version: '3.0.0'
	}

This will give the core adaptors to get cloud, grid and hadoop adaptors add the following dependencies:

    compile group: 'nl.esciencecenter.xenon.adaptors', name: 'xenon-adaptors-cloud', version: '3.0.0'
    compile group: 'nl.esciencecenter.xenon.adaptors', name: 'xenon-adaptors-grid', version: '3.0.0'
    compile group: 'nl.esciencecenter.xenon.adaptors', name: 'xenon-adaptors-hadoop', version: '3.0.0'

Simple examples

Here are some examples of basic operations you can perform with Xenon:

Copying a file from a local filesystem to a remote filesystem

import nl.esciencecenter.xenon.XenonException;
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.CopyStatus;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;

public class CopyFileLocalToSftpAbsolutePaths {

    public static void main(String[] args) throws Exception {

        // Use the file system adaptors to create file system representations; the remote file system
        // requires credentials, so we need to create those too.
        //
        // Assume the remote system is actually just a Docker container (e.g.
        // https://hub.docker.com/r/nlesc/xenon-ssh/), accessible via
        // port 10022 on localhost
        String location = "localhost:10022";
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        PasswordCredential credential = new PasswordCredential(username, password);
        FileSystem localFileSystem = FileSystem.create("file");
        FileSystem remoteFileSystem = FileSystem.create("sftp", location, credential);

        // create Paths for the source and destination files, using absolute paths
        Path sourceFile = new Path("/etc/passwd");
        Path destFile = new Path("/tmp/password");

        // create the destination file only if the destination path doesn't exist yet
        CopyMode mode = CopyMode.CREATE;
        boolean recursive = false;

        // perform the copy and wait 1000 ms for the successful or otherwise
        // completion of the operation
        String copyId = localFileSystem.copy(sourceFile, remoteFileSystem, destFile, mode, recursive);
        long timeoutMilliSecs = 1000;
        CopyStatus copyStatus = localFileSystem.waitUntilDone(copyId, timeoutMilliSecs);

        // throw any exceptions
        XenonException copyException = copyStatus.getException();
        if (copyException != null) {
          throw copyException;
        }
    }
}

Submitting a job

The following code performs a wordcount of a file residing on a remote machine:

import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.schedulers.JobDescription;
import nl.esciencecenter.xenon.schedulers.JobStatus;
import nl.esciencecenter.xenon.schedulers.Scheduler;

public class SlurmSubmitWordCountJob {

    public static void main(String[] args) throws Exception {

        // Assume the remote system is actually just a Docker container (e.g.
        // https://hub.docker.com/r/nlesc/xenon-slurm/), accessible to user 'xenon' via
        // port 10022 on localhost, using password 'javagat'
        String location = "localhost:10022";
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        PasswordCredential credential = new PasswordCredential(username, password);

        // create the SLURM scheduler representation
        Scheduler scheduler = Scheduler.create("slurm", location, credential);

        JobDescription description = new JobDescription();
        description.setExecutable("/usr/bin/wc");
        description.setArguments("-l", "/etc/passwd");
        description.setStdout("/tmp/wc.stdout.txt");

        // submit the job
        String jobId = scheduler.submitBatchJob(description);

        long WAIT_INDEFINITELY = 0;
        JobStatus jobStatus = scheduler.waitUntilDone(jobId, WAIT_INDEFINITELY);

        // print any exceptions
        Exception jobException = jobStatus.getException();
        if (jobException != null)  {
          throw jobException;
        }

    }
}

The output of the job will be written to /tmp/wc.stdout.txt file in the nlesc/xenon-slurm Docker container.

For more examples, see the tutorial at Read The Docs.

Documentation

Xenon's JavaDoc is available online at http://nlesc.github.io/Xenon/versions/2.6.2/javadoc/.

Legal

The Xenon library is copyrighted by the Netherlands eScience Center and released under the Apache License, Version 2.0. A copy of the license may be obtained from http://www.apache.org/licenses/LICENSE-2.0.

Xenon uses several third-party libraries that have their own (permissive, open source) licenses. See the file legal/README.md for an overview.