Permalink
Browse files

Directory listing bug seems to be working, written integration tests,…

… changies in tests run configurations, merged docs and directory listing from Steve https://github.com/hortonworks/Hadoop-and-Swift-integration
  • Loading branch information...
1 parent 9befd06 commit f4ae0e6c47d6db078a9a081f7f20c025793b66f5 dmezhenskiy committed Feb 6, 2013
Showing with 505 additions and 206 deletions.
  1. +37 −0 README.md
  2. +2 −16 swift-file-system/src/main/java/org/apache/hadoop/fs/swift/block/SwiftBlockFileSystemStore.java
  3. +2 −0 swift-file-system/src/main/java/org/apache/hadoop/fs/swift/http/SwiftProtocolConstants.java
  4. +95 −35 swift-file-system/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java
  5. +2 −2 swift-file-system/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
  6. +207 −86 swift-file-system/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystemStore.java
  7. +8 −18 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/InMemorySwiftNativeStore.java
  8. +0 −3 ...t-file-system/src/test/java/org/apache/hadoop/fs/swift/NativeSwiftFileSystemContractBaseTest.java
  9. +3 −3 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/SwiftTestUtils.java
  10. +2 −1 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/TestSwiftConfig.java
  11. +3 −10 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
  12. +0 −6 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemContract.java
  13. +3 −7 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemExtendedContract.java
  14. +1 −1 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/TestSwiftObjectPath.java
  15. +85 −15 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/functional/tests/SwiftFileSystemTest.java
  16. +3 −3 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/http/TestRestClientBindings.java
  17. +52 −0 swift-file-system/src/test/resources/core-site.xml
View
37 README.md
@@ -9,3 +9,40 @@ The use and distribution terms for this software are covered by Apache License 2
## References
Reference to Hadoop issue [HADOOP-8545](https://issues.apache.org/jira/browse/HADOOP-8545)
+
+## Abstract
+This patch enables support of OpenStack Swift Object Storage.
+Swift storage hierarchy is __account -> container -> object__. Account contains containers (Amazon S3 __buckets__ analogue), each container holds huge number of objects, which are stored as blob.
+
+Hadoop can work with different Swift regions/installations, public or private.
+
+It can also work with the same object stores using multiple login details.
+
+## Hadoop and Swift integration concepts
+Containers are created by users with accounts on the Swift filestore, and hold objects.
+
+* Objects can be zero bytes long, or they can contain data.
+* Objects in the container can be up to 5GB; there is a special support for larger files than this, which merges multiple objects in to one.
+* Each object is referenced by it's name. An object is named by its full name, such as this-is-an-object-name.
+* You can use any characters in an object name that can be 'URL-encoded'; the maximum length of a name is 1034 characters -after URL encoding.
+* Names can have / charcters in them, which are used to create the illusion of a directory structure. For example dir/dir2/name. Even though this looks like a directory, it is still just a name. There is no requirement to have any entries in the container called dir or dir/dir2
+* That said. if the container has zero-byte objects that look like directory names above other objects, they can pretend to be directories. Continuing the example, a 0-byte object called dir would tell clients that it is a directory while dir/dir2 or dir/dir2/name were present. This creates an illusion of containers holding a filesystem.
+
+Client applications talk to Swift over HTTP or HTTPS, reading, writing and deleting objects using standard HTTP operations (GET, PUT and DELETE, respectively). There is also a COPY operation, that creates a new object in the container, with a new name, containing the old data. There is no rename operation itself, objects need to be copied -then the original entry deleted.
+
+The Swift Filesystem is eventually consistent: an operation on an object may not be immediately visible to that client, or other clients. This is a consequence of the goal of the filesystem: to span a set of machines, across multiple datacentres, in such a way that the data can still be available when many of them fail. (In contrast, the Hadoop HDFS filesystem is immediately consistent, but it does not span datacenters.)
+
+Eventual consistency can cause surprises for client applications that expect immediate consistency: after an object is deleted or overwritten, the object may still be visible -or the old data still retrievable. The Swift Filesystem client for Apache Hadoop attempts to handle this, in conjunction with the MapReduce engine, but there may be still be occasions when eventual consistency causes suprises.
+
+## Warnings
+1. Do not share your login details with anyone, which means do not log the details, or check the XML configuration files into any revision control system to which you do not have exclusive access.
+2. Similarly, no use your real account details in any documentation.
+3. Do not use the public service endpoint from within an OpenStack cluster, as it will run up large bills.
+4. Remember: it's not a real filesystem or hierarchical directory structure. Some operations (directory rename and delete) take time and are not atomic or isolated from other operations taking place.
+5. Append is not supported.
+6. Unix-style permissions are not supported. All accounts with write access to a repository have unlimited access; the same goes for those with read access.
+7. In the public clouds, do not make the containers public unless you are happy with anyone reading your data, and are prepared to pay the costs of their downloads.
+
+## Limits
+* Maximum length of an object path: 1024 characters
+* Maximum size of a binary object: no absolute limit. Files > 5GB are partitioned into separate files in the native filesystem, and merged during retrieval.
View
18 ...file-system/src/main/java/org/apache/hadoop/fs/swift/block/SwiftBlockFileSystemStore.java
@@ -29,23 +29,9 @@
import org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore;
import org.apache.hadoop.fs.swift.util.SwiftObjectPath;
-import java.io.BufferedInputStream;
-import java.io.BufferedOutputStream;
-import java.io.ByteArrayOutputStream;
-import java.io.Closeable;
-import java.io.File;
-import java.io.FileInputStream;
-import java.io.FileNotFoundException;
-import java.io.FileOutputStream;
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
+import java.io.*;
import java.net.URI;
-import java.util.Collections;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Set;
-import java.util.StringTokenizer;
+import java.util.*;
/**
* Block store for Swift. Implements Hadoop S3 FileSystemStore interface.
View
2 swift-file-system/src/main/java/org/apache/hadoop/fs/swift/http/SwiftProtocolConstants.java
@@ -34,6 +34,8 @@
public static final int SWIFT_HTTPS_PORT = 443;
public static final String HEADER_RANGE = HttpHeaders.RANGE;
public static final String HEADER_DESTINATION = HttpHeaders.DESTINATION;
+ public static final String HEADER_LAST_MODIFIED = "Last-Modified";
+ public static final String HEADER_CONTENT_LENGTH = "Content-Length";
public static final String SWIFT_RANGE_HEADER_FORMAT_PATTERN = "bytes=%d-%d";
public static final String SERVICE_CATALOG_SWIFT = "swift";
public static final String SERVICE_CATALOG_CLOUD_FILES = "cloudFiles";
View
130 swift-file-system/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java
@@ -19,15 +19,7 @@
package org.apache.hadoop.fs.swift.http;
import org.apache.commons.httpclient.*;
-
-import static org.apache.commons.httpclient.HttpStatus.*;
-import org.apache.commons.httpclient.methods.DeleteMethod;
-import org.apache.commons.httpclient.methods.GetMethod;
-import org.apache.commons.httpclient.methods.HeadMethod;
-import org.apache.commons.httpclient.methods.InputStreamRequestEntity;
-import org.apache.commons.httpclient.methods.PostMethod;
-import org.apache.commons.httpclient.methods.PutMethod;
-import org.apache.commons.httpclient.methods.StringRequestEntity;
+import org.apache.commons.httpclient.methods.*;
import org.apache.commons.httpclient.params.HttpMethodParams;
import org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory;
import org.apache.commons.httpclient.protocol.Protocol;
@@ -37,26 +29,14 @@
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.swift.auth.AuthenticationRequest;
-import org.apache.hadoop.fs.swift.auth.AuthenticationRequestWrapper;
-import org.apache.hadoop.fs.swift.auth.AuthenticationResponse;
-import org.apache.hadoop.fs.swift.auth.AuthenticationWrapper;
-import org.apache.hadoop.fs.swift.auth.PasswordCredentials;
+import org.apache.hadoop.fs.swift.auth.*;
import org.apache.hadoop.fs.swift.auth.entities.AccessToken;
import org.apache.hadoop.fs.swift.auth.entities.Catalog;
import org.apache.hadoop.fs.swift.auth.entities.Endpoint;
-import org.apache.hadoop.fs.swift.exceptions.SwiftBadRequestException;
-import org.apache.hadoop.fs.swift.exceptions.SwiftConfigurationException;
-import org.apache.hadoop.fs.swift.exceptions.SwiftConnectionException;
-import org.apache.hadoop.fs.swift.exceptions.SwiftException;
-import org.apache.hadoop.fs.swift.exceptions.SwiftInternalStateException;
-import org.apache.hadoop.fs.swift.exceptions.SwiftInvalidResponseException;
+import org.apache.hadoop.fs.swift.exceptions.*;
import org.apache.hadoop.fs.swift.ssl.EasySSLProtocolSocketFactory;
import org.apache.hadoop.fs.swift.util.JSONUtil;
import org.apache.hadoop.fs.swift.util.SwiftObjectPath;
-
-import static org.apache.hadoop.fs.swift.http.SwiftProtocolConstants.*;
-
import org.apache.hadoop.fs.swift.util.SwiftUtils;
import org.apache.http.conn.params.ConnRoutePNames;
import org.jets3t.service.impl.rest.httpclient.HttpMethodReleaseInputStream;
@@ -72,6 +52,9 @@
import java.util.Properties;
import java.util.regex.Pattern;
+import static org.apache.commons.httpclient.HttpStatus.*;
+import static org.apache.hadoop.fs.swift.http.SwiftProtocolConstants.*;
+
/**
* This implements the client-side of the Swift REST API
*/
@@ -556,7 +539,7 @@ private URI pathToObjectLocation(SwiftObjectPath path) {
if (object.startsWith("/")) {
object = object.substring(1);
}
-
+ object = encodeUrl(object);
dataLocationURI = dataLocationURI.concat("/")
.concat(path.getContainer())
.concat("/?prefix=")
@@ -594,6 +577,79 @@ protected void setup(GetMethod method) {
}
/**
+ * Find objects in a directory
+ *
+ * @param path path prefix
+ * @param requestHeaders optional request headers
+ * @return byte[] file data or null if the object was not found
+ * @throws IOException on IO Faults
+ * @throws FileNotFoundException if nothing is at the end of the URI -that is,
+ * the directory is empty
+ */
+ public byte[] listObjectsInDirectory(SwiftObjectPath path,
+ final Header... requestHeaders) throws IOException {
+ preRemoteCommand("listObjectsInPath");
+ URI uri;
+ String endpoint = getEndpointURI().toString();
+ StringBuilder dataLocationURI = new StringBuilder();
+ dataLocationURI.append(endpoint);
+ String object = path.getObject();
+ if (object.startsWith("/")) {
+ object = object.substring(1);
+ }
+ if (!object.endsWith("/")) {
+ object = object.concat("/");
+ }
+
+/* dataLocationURI = dataLocationURI.append("/")
+.append(path.getContainer())
+.append("/?path=")
+.append(object);*/
+ dataLocationURI = dataLocationURI.append("/")
+ .append(path.getContainer())
+ .append("/?prefix=")
+ .append(object)
+ .append("&delimiter=/")
+ ;
+ return findObjects(dataLocationURI.toString(), requestHeaders);
+ }
+
+ private byte[] findObjects(String location, final Header[] requestHeaders) throws
+ IOException {
+ URI uri;
+ preRemoteCommand("findObjects");
+ try {
+ uri = new URI(location);
+ } catch (URISyntaxException e) {
+ throw new SwiftException("Bad URI: " + location, e);
+ }
+
+ return perform(uri, new GetMethodProcessor<byte[]>() {
+ @Override
+ public byte[] extractResult(GetMethod method) throws IOException {
+ if (method.getStatusCode() == SC_NOT_FOUND) {
+ //no result
+ throw new FileNotFoundException("Not found " + method.getURI());
+ }
+ return method.getResponseBody();
+ }
+
+ @Override
+ protected int[] getAllowedStatusCodes() {
+ return new int[] {
+ SC_OK,
+ SC_NOT_FOUND
+ };
+ }
+
+ @Override
+ protected void setup(GetMethod method) {
+ setHeaders(method, requestHeaders);
+ }
+ });
+ }
+
+ /**
* Copy an object. This is done by sending a COPY method to the filesystem
* which is required to handle this WebDAV-level extension to the
* base HTTP operations.
@@ -1111,21 +1167,25 @@ public static URI pathToURI(SwiftObjectPath path,
String dataLocationURI = endpointURI.toString();
try {
- String url = path.toUriPath();
- if (url.matches(".*\\s+.*")) {
+
+ dataLocationURI = SwiftUtils.joinPaths(dataLocationURI, encodeUrl(path.toUriPath()));
+ return new URI(dataLocationURI);
+ } catch (URISyntaxException e) {
+ throw new SwiftException("Failed to create URI from " + dataLocationURI, e);
+ }
+ }
+
+ private static String encodeUrl(String url) {
+ if (url.matches(".*\\s+.*")) {
+ try {
url = URLEncoder.encode(url, "UTF-8");
- // URLEncoder despite it's incorrect name was not designed to
- // encode URLs.Spaces are encoded as '+', but correct encoding is '%20'
url = url.replace("+", "%20");
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException("failed to encode URI", e);
}
- dataLocationURI = SwiftUtils.joinPaths(dataLocationURI, url);
- return new URI(dataLocationURI);
- } catch (URISyntaxException e) {
- throw new SwiftException("Failed to create URI from " + dataLocationURI,
- e);
- } catch (UnsupportedEncodingException e) {
- throw new RuntimeException("failed to encode URI", e);
}
+
+ return url;
}
/**
View
4 ...t-file-system/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java
@@ -185,7 +185,7 @@ public boolean isDirectory(Path f) throws IOException {
// each block has its own location -which may be determinable
// from the Swift client API, depending on the remote server
- final FileStatus[] listOfFileBlocks = store.listSubPaths(file.getPath());
+ final FileStatus[] listOfFileBlocks = store.listSubPaths(file.getPath(), false, true);
List<URI> locations = new ArrayList<URI>();
if (listOfFileBlocks.length > 1) {
for (FileStatus fileStatus : listOfFileBlocks) {
@@ -287,7 +287,7 @@ private boolean mkdir(Path path) throws IOException {
if (LOG.isDebugEnabled()) {
LOG.debug("SwiftFileSystem.listStatus for: " + f);
}
- return store.listSubPaths(f);
+ return store.listSubPaths(f, false, false);
}
/**
View
293 ...e-system/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystemStore.java
@@ -14,7 +14,6 @@
import org.apache.hadoop.fs.swift.http.SwiftRestClient;
import org.apache.hadoop.fs.swift.util.SwiftObjectPath;
import org.apache.hadoop.fs.swift.util.SwiftUtils;
-import org.apache.http.HttpHeaders;
import java.io.ByteArrayInputStream;
import java.io.FileNotFoundException;
@@ -43,8 +42,8 @@
/**
* Initalize the filesystem store -this creates the REST client binding.
*
- * @param fsURI URI of the filesystem, which is used to map to the filesystem-specific
- * options in the configuration file
+ * @param fsURI URI of the filesystem, which is used to map to the filesystem-specific
+ * options in the configuration file
* @param configuration configuration
* @throws IOException on any failure.
*/
@@ -62,9 +61,9 @@ public String toString() {
/**
* Upload a file
*
- * @param path destination path in the swift filesystem
+ * @param path destination path in the swift filesystem
* @param inputStream input data
- * @param length length of the data
+ * @param length length of the data
* @throws IOException on a problem
*/
public void uploadFile(Path path, InputStream inputStream, long length) throws IOException {
@@ -74,10 +73,10 @@ public void uploadFile(Path path, InputStream inputStream, long length) throws I
/**
* Upload part of a larger file.
*
- * @param path destination path
- * @param partNumber item number in the path
+ * @param path destination path
+ * @param partNumber item number in the path
* @param inputStream input data
- * @param length length of the data
+ * @param length length of the data
* @throws IOException on a problem
*/
public void uploadFilePart(Path path, int partNumber, InputStream inputStream, long length) throws IOException {
@@ -118,12 +117,13 @@ public void createManifestForPartUpload(Path path) throws IOException {
*
* @param path path
* @return file metadata. -or null if no headers were received back from the server.
- * @throws IOException on a problem
+ * @throws IOException on a problem
* @throws FileNotFoundException if there is nothing at the end
*/
public FileStatus getObjectMetadata(Path path) throws IOException {
+ SwiftObjectPath objectPath = toObjectPath(path);
final Header[] headers;
- headers = swiftRestClient.headRequest(toObjectPath(path),
+ headers = swiftRestClient.headRequest(objectPath,
SwiftRestClient.NEWEST);
//no headers is treated as a missing file
if (headers.length == 0) {
@@ -140,10 +140,10 @@ public FileStatus getObjectMetadata(Path path) throws IOException {
length = 0;
isDir = true;
}
- if (HttpHeaders.CONTENT_LENGTH.equals(headerName)) {
+ if (SwiftProtocolConstants.HEADER_CONTENT_LENGTH.equals(headerName)) {
length = Long.parseLong(header.getValue());
}
- if (HttpHeaders.LAST_MODIFIED.equals(headerName)) {
+ if (SwiftProtocolConstants.HEADER_LAST_MODIFIED.equals(headerName)) {
final SimpleDateFormat simpleDateFormat = new SimpleDateFormat(PATTERN);
try {
lastModified = simpleDateFormat.parse(header.getValue()).getTime();
@@ -153,12 +153,7 @@ public FileStatus getObjectMetadata(Path path) throws IOException {
}
}
- final Path correctSwiftPath;
- try {
- correctSwiftPath = getCorrectSwiftPath(path);
- } catch (URISyntaxException e) {
- throw new SwiftException("Specified path " + path + " is incorrect", e);
- }
+ Path correctSwiftPath = getCorrectSwiftPath(path);
return new FileStatus(length, isDir, 0, 0L, lastModified, correctSwiftPath);
}
@@ -168,7 +163,7 @@ public FileStatus getObjectMetadata(Path path) throws IOException {
*
* @param path object path
* @return the input stream -this must be closed to terminate the connection
- * @throws IOException IO problems
+ * @throws IOException IO problems
* @throws FileNotFoundException path doesn't resolve to an object
*/
public InputStream getObject(Path path) throws IOException {
@@ -179,9 +174,9 @@ public InputStream getObject(Path path) throws IOException {
/**
* Get the input stream starting from a specific point.
*
- * @param path path to object
+ * @param path path to object
* @param byteRangeStart starting point
- * @param length no. of bytes
+ * @param length no. of bytes
* @return an input stream that must be closed
* @throws IOException IO problems
*/
@@ -191,11 +186,40 @@ public InputStream getObject(Path path, long byteRangeStart, long length)
toObjectPath(path), byteRangeStart, length);
}
- public FileStatus[] listSubPaths(Path path) throws IOException {
+ /**
+ * List all elements in this directory
+ *
+ * @param path path to work with
+ * @param nameOnly should the status be minimal and not make any calls
+ * to the system to determine attributes beyond the name?
+ * @return the file statuses, or an empty array if there are no children
+ * @throws IOException on IO problems
+ * @throws FileNotFoundException if the path is nonexistent
+ */
+ public FileStatus[] listSubPaths(Path path,
+ boolean recursive,
+ boolean nameOnly) throws IOException {
final Collection<FileStatus> fileStatuses;
- fileStatuses = listDirectory(toDirPath(path));
+ fileStatuses = listDirectory(toDirPath(path), recursive, nameOnly);
return fileStatuses.toArray(new FileStatus[fileStatuses.size()]);
}
+ /**
+ * List all elements in this directory
+ *
+ *
+ * @param path path to work with
+ * @param nameOnly should the status be minimal and not make any calls
+ * to the system to determine attributes beyond the name?
+ * @return the file statuses, or an empty list if there are no children
+ * @throws IOException on IO problems
+ * @throws FileNotFoundException if the path is nonexistent
+ */
+ public List<FileStatus> listDirectory(Path path,
+ boolean recursive,
+ boolean nameOnly) throws IOException {
+ return listDirectory(toDirPath(path), recursive, nameOnly);
+ }
+
/**
* Create a directory
@@ -250,12 +274,26 @@ public boolean rmdir(Path path) throws IOException {
*
* @param path object path
* @return true if the metadata of an object could be retrieved
- * @throws IOException IO problems
+ * @throws IOException IO problems other than FileNotFound, which
+ * is downgraded to an object does not exist return code
*/
public boolean objectExists(Path path) throws IOException {
+ return objectExists(toObjectPath(path));
+ }
+ /**
+ * Does the object exist
+ *
+ * @param path swift object path
+ * @return true if the metadata of an object could be retrieved
+ * @throws IOException IO problems other than FileNotFound, which
+ * is downgraded to an object does not exist return code
+ */
+ public boolean objectExists(SwiftObjectPath path) throws IOException {
try {
- getObjectMetadata(path);
- return true;
+ Header[] headers = swiftRestClient.headRequest(path,
+ SwiftRestClient.NEWEST);
+ //no headers is treated as a missing file
+ return headers.length != 0;
} catch (FileNotFoundException e) {
return false;
}
@@ -295,7 +333,6 @@ public boolean renameDirectory(Path src, Path dst) throws IOException {
return false;
}
-
final FileStatus srcMetadata;
try {
srcMetadata = getObjectMetadata(src);
@@ -312,16 +349,20 @@ public boolean renameDirectory(Path src, Path dst) throws IOException {
dstMetadata = null;
}
- //check to see if the parent exists
- Path destParent = dst.getParent();
- try {
- FileStatus destParentStat = destParent != null
- ? getObjectMetadata(destParent)
- : null;
- } catch (FileNotFoundException e) {
- //destination parent doesn't exist; bail out
- LOG.debug("destination parent directory " + destParent + " doesn't exist");
- return false;
+ //check to see if the destination parent directory exists
+ Path srcParent = src.getParent();
+ Path dstParent = dst.getParent();
+ //skip the overhead of a HEAD call if the src and dest share the same
+ //parent dir (in which case the dest dir exists), or the destination
+ //directory is root, in which case it must also exist
+ if (dstParent !=null && !dstParent.equals(srcParent)) {
+ try {
+ getObjectMetadata(dstParent);
+ } catch (FileNotFoundException e) {
+ //destination parent doesn't exist; bail out
+ LOG.debug("destination parent directory "+ dstParent + " doesn't exist");
+ return false;
+ }
}
boolean destExists = dstMetadata != null;
@@ -331,14 +372,13 @@ public boolean renameDirectory(Path src, Path dst) throws IOException {
boolean srcIsFile = !SwiftUtils.isDirectory(srcMetadata);
- if (!srcIsFile) {
+ if (srcIsFile) {
//source is a simple file
// outcomes:
- // #1 dest exists and is file: fail
+ // #1 dest exists and is file: fail
// #2 dest exists and is dir: destination path becomes under dest dir
// #3 dest does not exist: use dest as name
-
if (destExists) {
if (destIsDir) {
@@ -354,15 +394,15 @@ public boolean renameDirectory(Path src, Path dst) throws IOException {
destPath = toObjectPath(dst);
}
-
- boolean copySucceeded = swiftRestClient.copyObject(srcObject, destPath);
- if (copySucceeded) {
- //if the copy worked delete the original
+ boolean success = swiftRestClient.
+ copyObject(srcObject, destPath);
+ if (success) {
swiftRestClient.delete(srcObject);
+ } else {
+ throw new SwiftException("Copy of " + srcObject + " to "
+ + destPath + "failed");
}
- return copySucceeded;
-
-
+ return true;
} else {
//here the source exists and is a directory
@@ -387,50 +427,71 @@ public boolean renameDirectory(Path src, Path dst) throws IOException {
}
SwiftObjectPath targetObjectPath = toObjectPath(targetPath);
- //enum the child entries
- List<FileStatus> fileStatuses = listDirectory(toObjectPath(src.getParent()));
+ //enum the child entries amd everything underneath
+ List<FileStatus> fileStatuses = listDirectory(toObjectPath(src), true, false);
boolean result = true;
//iterative copy of everything under the directory
for (FileStatus fileStatus : fileStatuses) {
- if (!fileStatus.isDir()) {
- boolean copied =
- swiftRestClient.copyObject(toObjectPath(fileStatus.getPath()),
- targetObjectPath);
- result &= copied;
-
- //if client couldn't copy, data will be lost
- if (copied) {
- swiftRestClient.delete(toObjectPath(fileStatus.getPath()));
+ Path copySourcePath = fileStatus.getPath();
+ SwiftObjectPath copyDestination = toObjectPath(
+ new Path(targetPath, copySourcePath.getName()));
+ try {
+ boolean success = swiftRestClient.
+ copyObject(toObjectPath(copySourcePath), copyDestination);
+ if (success) {
+ swiftRestClient.delete(toObjectPath(copySourcePath));
+ } else {
+ throw new SwiftException("Copy of " + toObjectPath(fileStatus.getPath()) + " to "
+ + targetObjectPath + "failed");
}
+ } catch (FileNotFoundException e) {
+ LOG.info("Skipping rename of " + copySourcePath + " as it not found");
}
}
-
+ try {
+ swiftRestClient.delete(toObjectPath(src));
+ }catch (FileNotFoundException e) {
+ LOG.debug("Source directory " + src.toString() + " doesn't exist");
+ }
return result;
}
}
+ public void copy(Path srcKey, Path dstKey) throws IOException {
+ SwiftObjectPath srcObject = toObjectPath(srcKey);
+ SwiftObjectPath destObject = toObjectPath(dstKey);
+ swiftRestClient.copyObject(srcObject, destObject);
+ }
+
+
/**
- * List a directory
+ * List a directory.
+ * This is O(n) for the number of objects in this path.
*
* @param path path to list
+ * @param nameOnly should the status be minimal (name) or should
+ * the (expensive) operation be made to ask for it.
* @return the filestats of all the entities in the directory -or
- * an empty list if no objects were found listed under that prefix
+ * an empty list if no objects were found listed under that prefix
* @throws IOException IO problems
*/
- private List<FileStatus> listDirectory(SwiftObjectPath path) throws IOException {
- String pathURI = path.toUriPath();
- if (!pathURI.endsWith(Path.SEPARATOR)) {
- pathURI += Path.SEPARATOR;
- }
-
+ private List<FileStatus> listDirectory(SwiftObjectPath path,
+ boolean recursive,
+ boolean nameOnly) throws IOException {
final byte[] bytes;
+ final ArrayList<FileStatus> files = new ArrayList<FileStatus>();
try {
- bytes = swiftRestClient.findObjectsByPrefix(path);
+ if (recursive) {
+ bytes = swiftRestClient.findObjectsByPrefix(path);
+ } else {
+ bytes = swiftRestClient.listObjectsInDirectory(path);
+ }
} catch (FileNotFoundException e) {
if (LOG.isDebugEnabled()) {
- LOG.debug("Directory not found " + path);
+ LOG.debug("" +
+ "File/Directory not found " + path);
}
if (SwiftUtils.isRootDir(path)) {
return Collections.emptyList();
@@ -444,46 +505,106 @@ public boolean renameDirectory(Path src, Path dst) throws IOException {
if (LOG.isDebugEnabled()) {
LOG.debug("lsdir " + path +
" status code says NO_CONTENT; "
- + e.toString());
+ + e.toString(),
+ e);
}
if (SwiftUtils.isRootDir(path)) {
return Collections.emptyList();
} else {
- throw new FileNotFoundException("Not found: " + path);
+ //NO_CONTENT returned on something other than the root directory;
+ //see if it is there, and conver to empty list or not found
+ //depending on whether the entry exists.
+ FileStatus stat = getObjectMetadata(getCorrectSwiftPath(path));
+
+ if (SwiftUtils.isDirectory(stat)) {
+ //it's an empty directory. state that
+ return Collections.emptyList();
+ } else {
+ //it's a file -return that as the status
+ files.add(stat);
+ return files;
+ }
}
} else {
+ //a different status code: rethrow immediately
throw e;
}
}
- final StringTokenizer tokenizer = new StringTokenizer(new String(bytes), "\n");
- final ArrayList<FileStatus> files = new ArrayList<FileStatus>();
+ //the byte array contains the files separated by newlines
+ final StringTokenizer tokenizer =
+ new StringTokenizer(new String(bytes), "\n");
+
+ Map<String, Boolean> names = new HashMap<String, Boolean>();
+ //insert own name as one to skip
+ names.put(path.getObject(), true);
while (tokenizer.hasMoreTokens()) {
String pathInSwift = tokenizer.nextToken();
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("entry: " + pathInSwift);
+ }
if (!pathInSwift.startsWith("/")) {
pathInSwift = "/".concat(pathInSwift);
}
- //this contains all
- final FileStatus metadata = getObjectMetadata(new Path(pathInSwift));
- if (metadata != null) {
- files.add(metadata);
+ Path childPath = new Path(pathInSwift);
+ if (!names.containsKey(pathInSwift)) {
+ names.put(pathInSwift, true);
+ names.put(pathInSwift + "/", true);
+ //For each entry, get the metadata.
+ try {
+ FileStatus metadata;
+ if (nameOnly) {
+ metadata = new FileStatus(0, false, 0, 0, 0, childPath);
+ } else {
+ metadata = getObjectMetadata(childPath);
+ }
+ files.add(metadata);
+ } catch (FileNotFoundException e) {
+ //get Object metadata failed
+ LOG.info(
+ "Object " + childPath + " was deleting during directory listing");
+ }
+ } else {
+ //hash map
+ LOG.debug("skipping adding self to path");
}
}
-
return files;
}
- private Path getCorrectSwiftPath(Path path) throws URISyntaxException {
- final URI fullUri = new URI(uri.getScheme(),
- uri.getAuthority(),
- path.toUri().getPath(),
- null,
- null);
+ private Path getCorrectSwiftPath(Path path) throws
+ SwiftException {
+ try {
+ final URI fullUri = new URI(uri.getScheme(),
+ uri.getAuthority(),
+ path.toUri().getPath(),
+ null,
+ null);
- return new Path(fullUri);
+ return new Path(fullUri);
+ } catch (URISyntaxException e) {
+ throw new SwiftException("Specified path " + path + " is incorrect", e);
+ }
}
+ private Path getCorrectSwiftPath(SwiftObjectPath path) throws
+ SwiftException {
+ try {
+ final URI fullUri = new URI(uri.getScheme(),
+ uri.getAuthority(),
+ path.getObject(),
+ null,
+ null);
+
+ return new Path(fullUri);
+ } catch (URISyntaxException e) {
+ throw new SwiftException("Specified path " + path + " is incorrect", e);
+ }
+ }
+
+
+
/**
* extracts URIs from json
*
@@ -499,4 +620,4 @@ private Path getCorrectSwiftPath(Path path) throws URISyntaxException {
}
return result;
}
-}
+}
View
26 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/InMemorySwiftNativeStore.java
@@ -8,21 +8,16 @@
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore;
-import java.io.ByteArrayInputStream;
-import java.io.ByteArrayOutputStream;
-import java.io.FileNotFoundException;
-import java.io.IOException;
-import java.io.InputStream;
+import java.io.*;
import java.net.URI;
import java.util.List;
import java.util.SortedMap;
import java.util.TreeMap;
/**
* In-memory Swift emulator for tests.
- *
+ * <p/>
* This is very unrealistic, and so the functional tests should be preferred.
- *
*/
public class InMemorySwiftNativeStore extends SwiftNativeFileSystemStore {
private static final Log LOG = LogFactory.getLog(InMemorySwiftNativeStore.class);
@@ -52,8 +47,8 @@ public void uploadFile(Path path, InputStream inputStream, long length) {
IOUtils.closeQuietly(inputStream);
}
metadataMap.put(path.toUri().toString(),
- new FileStatus(size, false, 0, 0, System.currentTimeMillis(),
- path));
+ new FileStatus(size, false, 0, 0, System.currentTimeMillis(),
+ path));
dataMap.put(path.toUri().toString(), out.toByteArray());
}
@@ -73,7 +68,7 @@ public InputStream getObject(Path path) throws IOException {
@Override
public InputStream getObject(Path path, long byteRangeStart, long length) throws
- IOException {
+ IOException {
byte[] data = dataMap.get(path.toUri().toString());
if (data == null) {
throw new FileNotFoundException("Not found" + path.toUri());
@@ -82,15 +77,10 @@ public InputStream getObject(Path path, long byteRangeStart, long length) throws
}
@Override
- public FileStatus[] listSubPaths(Path path) throws IOException {
- throw new UnsupportedOperationException("not implemented for testing purposes");
- }
-
- @Override
public void createDirectory(Path path) {
metadataMap.put(path.toUri().toString(),
- new FileStatus(0, false, 0, 0, System.currentTimeMillis(),
- path));
+ new FileStatus(0, false, 0, 0, System.currentTimeMillis(),
+ path));
}
@Override
@@ -100,7 +90,7 @@ public void createDirectory(Path path) {
@Override
public boolean deleteObject(Path path) throws IOException {
- boolean found = null!= metadataMap.remove(path.toUri().toString());
+ boolean found = null != metadataMap.remove(path.toUri().toString());
dataMap.remove(path.toUri().toString());
return found;
}
View
3 ...ystem/src/test/java/org/apache/hadoop/fs/swift/NativeSwiftFileSystemContractBaseTest.java
@@ -3,10 +3,7 @@
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FSDataInputStream;
-import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystemContractBaseTest;
-import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem;
import java.io.IOException;
View
6 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/SwiftTestUtils.java
@@ -21,13 +21,13 @@
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.swift.exceptions.SwiftConfigurationException;
-import static org.junit.Assert.*;
-
-import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.Properties;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
/**
* Utilities used across test cases
*/
View
3 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/TestSwiftConfig.java
@@ -2,13 +2,14 @@
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.swift.http.SwiftRestClient;
-import static org.apache.hadoop.fs.swift.http.SwiftProtocolConstants.*;
import org.junit.Test;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
+import static org.apache.hadoop.fs.swift.http.SwiftProtocolConstants.*;
+
/**
*
*/
View
13 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java
@@ -21,26 +21,19 @@
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FSDataInputStream;
-import org.apache.hadoop.fs.FSDataOutputStream;
-import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.LocatedFileStatus;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.fs.*;
import org.apache.hadoop.fs.swift.exceptions.SwiftBadRequestException;
import org.apache.hadoop.fs.swift.exceptions.SwiftException;
import org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem;
-
-import static org.junit.Assert.*;
-
import org.junit.Before;
import org.junit.Test;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.URI;
+import static org.junit.Assert.*;
+
public class TestSwiftFileSystemBasicOps {
private static final Log LOG =
View
6 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemContract.java
@@ -22,17 +22,11 @@
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.swift.http.RestClientBindings;
import org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem;
-import org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore;
-import org.apache.hadoop.fs.swift.util.SwiftUtils;
-import org.junit.Test;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
-import java.util.Properties;
/**
* This is the full filesystem contract test -which requires the
View
10 ...-system/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemExtendedContract.java
@@ -22,26 +22,22 @@
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FSDataInputStream;
-import org.apache.hadoop.fs.FSDataOutputStream;
-import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.*;
import org.apache.hadoop.fs.swift.http.RestClientBindings;
import org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
-import static org.junit.Assert.*;
-
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.Locale;
import java.util.Properties;
+import static org.junit.Assert.*;
+
public class TestSwiftFileSystemExtendedContract {
private static final Log LOG =
View
2 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/TestSwiftObjectPath.java
@@ -10,7 +10,7 @@
import java.net.URI;
-import static org.junit.Assert.*;
+import static org.junit.Assert.assertEquals;
/**
* Unit tests for SwiftObjectPath class.
View
100 ...system/src/test/java/org/apache/hadoop/fs/swift/functional/tests/SwiftFileSystemTest.java
@@ -3,14 +3,14 @@
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
-import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.swift.snative.SwiftFileSystemForFunctionalTests;
import org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem;
import org.junit.Before;
import org.junit.Test;
-import java.io.*;
+import java.io.FileNotFoundException;
+import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.concurrent.ExecutorService;
@@ -20,8 +20,9 @@
import static org.junit.Assert.*;
/**
- * these tests currently are unit tests, but will be
- * moved to functional/integration tests
+ * Integration tests, to run it read documentation
+ * https://github.com/DmitryMezhensky/Hadoop-and-Swift-integration/wiki/HowTo
+ *
*/
public class SwiftFileSystemTest {
URI uri;
@@ -30,18 +31,10 @@
@Before
public void initialization() throws URISyntaxException {
this.conf = new Configuration();
- this.conf.set("fs.swift.service.rackspace.auth.url", "");
- this.conf.set("fs.swift.service.rackspace.tenant", "");
- this.conf.set("fs.swift.service.rackspace.username", "");
- this.conf.set("fs.swift.service.rackspace.password", "");
- this.conf.set("fs.swift.service.rackspace.public", "true");
- this.conf.setInt("fs.swift.service.rackspace.http.port", 8080);
- this.conf.setInt("fs.swift.service.rackspace.https.port", 443);
this.uri = new URI("swift://data.rackspace");
}
-
/**
* tests functionality for big files ( > 5Gb) upload
*/
@@ -121,13 +114,13 @@ public void run() {
}
@Test
- public void testRenameDirWithSubDis() throws IOException {
+ public void testRenameFile() throws IOException {
final SwiftNativeFileSystem fileSystem = new SwiftNativeFileSystem();
fileSystem.initialize(uri, conf);
final String message = "message";
- final Path filePath = new Path("/home/user/documents/file.txt");
- final Path newFilePath = new Path("/home/user/documents/file2.txt");
+ final Path filePath = new Path("/home/tom/documents/file");
+ final Path newFilePath = new Path("/home/tom/documents/file1");
final FSDataOutputStream fsDataOutputStream = fileSystem.create(filePath);
fsDataOutputStream.write(message.getBytes());
@@ -141,6 +134,83 @@ public void testRenameDirWithSubDis() throws IOException {
assertEquals(message.length(), read);
assertEquals(message, new String(data, 0, read));
+
+ fileSystem.delete(filePath, true);
+ fileSystem.delete(newFilePath, true);
+ }
+
+ @Test
+ public void testRenameDirectoryWithFile() throws Exception {
+ final SwiftNativeFileSystem fileSystem = new SwiftNativeFileSystem();
+ fileSystem.initialize(uri, conf);
+
+ final Path filePath = new Path("/home/user/files/secret file.docx");
+ final Path newFilePath = new Path("/home/user/docs");
+
+ final FSDataOutputStream fsDataOutputStream = fileSystem.create(filePath);
+ fsDataOutputStream.write("this is a file".getBytes());
+ fsDataOutputStream.close();
+
+ fileSystem.create(newFilePath).close();
+ assertTrue(fileSystem.rename(filePath, newFilePath));
+
+ assertNotNull(fileSystem.getFileStatus(new Path("/home/user/docs/secret file.docx")));
+ fileSystem.delete(newFilePath, true);
+ }
+
+ @Test
+ public void testRenameDirectoryWithFiles() throws Exception {
+ final SwiftNativeFileSystem fileSystem = new SwiftNativeFileSystem();
+ fileSystem.initialize(uri, conf);
+
+ final Path logPath1 = new Path("/var/log/hadoop/logs/log1");
+ final Path logPath2 = new Path("/var/log/hadoop/logs/log2");
+ final Path logPath3 = new Path("/var/log/hadoop/logs/log3");
+ final Path logPath4 = new Path("/var/log/hadoop/logs/log4");
+ final Path logPath5 = new Path("/var/log/hadoop/logs/log5");
+ final Path newFilePath = new Path("/var/log/user");
+
+ fileSystem.create(logPath1).close();
+ fileSystem.create(logPath2).close();
+ fileSystem.create(logPath3).close();
+ fileSystem.create(logPath4).close();
+ fileSystem.create(logPath5).close();
+
+ fileSystem.create(newFilePath);
+ assertTrue(fileSystem.rename(new Path("/var/log/hadoop"), newFilePath));
+
+ try {
+ fileSystem.getFileStatus(logPath1);
+ throw new AssertionError("Directory exists after renaming");
+ } catch (FileNotFoundException e) {
+ }
+ try {
+ fileSystem.getFileStatus(logPath2);
+ throw new AssertionError("Directory exists after renaming");
+ } catch (FileNotFoundException e) {
+ }
+ try {
+ fileSystem.getFileStatus(logPath3);
+ throw new AssertionError("Directory exists after renaming");
+ } catch (FileNotFoundException e) {
+ }
+ try {
+ fileSystem.getFileStatus(logPath4);
+ throw new AssertionError("Directory exists after renaming");
+ } catch (FileNotFoundException e) {
+ }
+ try {
+ fileSystem.getFileStatus(logPath5);
+ throw new AssertionError("Directory exists after renaming");
+ } catch (FileNotFoundException e) {
+ }
+
+
+ assertNotNull(fileSystem.getFileStatus(new Path("/var/log/user/log1")));
+ assertNotNull(fileSystem.getFileStatus(new Path("/var/log/user/log2")));
+ assertNotNull(fileSystem.getFileStatus(new Path("/var/log/user/log3")));
+ assertNotNull(fileSystem.getFileStatus(new Path("/var/log/user/log4")));
+ assertNotNull(fileSystem.getFileStatus(new Path("/var/log/user/log5")));
}
private static String createDataSize(int msgSize) {
View
6 swift-file-system/src/test/java/org/apache/hadoop/fs/swift/http/TestRestClientBindings.java
@@ -18,9 +18,6 @@
package org.apache.hadoop.fs.swift.http;
-import static org.apache.hadoop.fs.swift.http.SwiftProtocolConstants.*;
-import static org.apache.hadoop.fs.swift.SwiftTestUtils.*;
-
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.swift.exceptions.SwiftConfigurationException;
import org.junit.Assert;
@@ -31,6 +28,9 @@
import java.net.URISyntaxException;
import java.util.Properties;
+import static org.apache.hadoop.fs.swift.SwiftTestUtils.assertPropertyEquals;
+import static org.apache.hadoop.fs.swift.http.SwiftProtocolConstants.*;
+
public class TestRestClientBindings extends Assert {
private static final String SERVICE = "sname";
View
52 swift-file-system/src/test/resources/core-site.xml
@@ -0,0 +1,52 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<!-- Values used when running unit tests. This is mostly empty, to -->
+<!-- use of the default values, overriding the potentially -->
+<!-- user-editted core-site.xml in the conf/ directory. -->
+
+<configuration>
+
+
+ <property>
+ <name>hadoop.tmp.dir</name>
+ <value>target/build/test</value>
+ <description>A base for other temporary directories.</description>
+ <final>true</final>
+ </property>
+
+ <!-- Turn security off for tests by default -->
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>simple</value>
+ </property>
+
+
+ <!--
+ To run these tests.
+
+ # Create a file auth-keys.xml - DO NOT ADD TO REVISION CONTROL
+ # add the property test.fs.swift.name to point to a swift filesystem URL
+ # Add the credentials for the service you are testing against
+ -->
+ <include xmlns="http://www.w3.org/2001/XInclude"
+ href="auth-keys.xml"/>
+
+</configuration>

0 comments on commit f4ae0e6

Please sign in to comment.