Skip to content
This repository has been archived by the owner on Aug 4, 2022. It is now read-only.

[REEF-1827] new Uri(path) in REEF.NET IO Returns Lowercase String #1329

Closed

Conversation

shouhengyi-microsoft
Copy link
Contributor

@shouhengyi-microsoft shouhengyi-microsoft commented Jul 13, 2017

This corresponds to REEF-1827. In the Org.Apache.REEF.IO.FileSystem.IFileSystem, we have many methods that are based on C#'s Uri. For example:

void CopyToLocal(Uri remoteFileUri, string localName);
void CreateDirectory(Uri directoryUri);

From the logging and my experiments, this converts path to lowercase characters. For example:

new Uri("hdfs://XYZ123/mypath/data.txt") // Returns "hdfs://xyz123/mypath/data.txt"

If we are interacting with hdfs, in the subsequent dfs command in Org.Apache.REEF.IO.FileSystem.Hadoop:

public void CopyToLocal(Uri remoteFileUri, string localName)
{
    _commandRunner.Run("dfs -get " + remoteFileUri + " " + localName);
}

this will fail, since hdfs is case-sensitive.

JIRA:
REEF-1827

Pull request:
#1329

Copy link
Contributor

@motus motus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I can merge it. Do we have unit tests that reveal that problem, by the way?

Copy link
Contributor

@motus motus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, could you please edit the PR title and prepend a JIRA id to it?

@shouhengyi-microsoft shouhengyi-microsoft changed the title Fixed Uri lowercase problem in hdfs by passing .AbsolutePath in the d… REEF-1827 new Uri(path) in REEF.NET IO Returns Lowercase String Jul 14, 2017
@shouhengyi-microsoft shouhengyi-microsoft changed the title REEF-1827 new Uri(path) in REEF.NET IO Returns Lowercase String [REEF-1827] new Uri(path) in REEF.NET IO Returns Lowercase String Jul 14, 2017
@shouhengyi-microsoft shouhengyi-microsoft deleted the REEF-1826 branch July 14, 2017 01:22
}

public IEnumerable<Uri> GetChildren(Uri directoryUri)
{
return _commandRunner.Run("dfs -ls " + directoryUri)
return _commandRunner.Run("dfs -ls " + directoryUri.AbsolutePath)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean even URI object converted the path into lower case, but uri.AbsolutePath will return case sensitive path, right?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
3 participants