Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add vsihdfs Support #714

Closed
wants to merge 21 commits into from
Closed

Add vsihdfs Support #714

Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
b71b9ea
HDFS: Build Configuration
Jun 23, 2018
b499edf
HDFS: Changes to Generated Files
Jun 23, 2018
038d8d0
HDFS: Use stdin Driver as a Template
Jun 22, 2018
7bab777
HDFS: r/o HDFS Driver
Jun 25, 2018
48d6ff1
HDFS: Provide Empty VSIInstallHdfsHandler
Jun 26, 2018
372e9c4
HDFS: Move String Out of Global Namespace
Jun 26, 2018
955f743
HDFS: Remove Uses of `this->`
Jun 26, 2018
f5ba058
HDFS: Move `hdfsConnect` Out of Constructor
Jun 26, 2018
b41ab52
HDFS: Use --with-java to Locate libjvm.so
Jun 26, 2018
169e130
HDFS: Replace Instance of `NULL` with `nullptr`
Jun 26, 2018
9a11c02
HDFS: De-Inline `Length` Function
Jun 26, 2018
c715894
HDFS: `Read`, `Eof`
Jun 26, 2018
d32f50f
HDFS: Use Macros for Directory Listing
Jun 26, 2018
e6ed0a9
HDFS: CI Fixes
Jun 27, 2018
77af4c4
HDFS: `Read` Semantics
Jun 27, 2018
c7036f5
HDFS: Add Mutexes
Jun 27, 2018
e2e37ba
HDFS: Silence Stack Traces
Jun 27, 2018
cd329f9
HDFS: Ensure Filesystem Handle
Jun 27, 2018
7db62d0
HDFS: Cleanup
Jun 27, 2018
089afe0
HDFS: Unit Tests
Jun 29, 2018
8d94089
HDFS: Documentation
Jul 1, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
34 changes: 30 additions & 4 deletions gdal/port/cpl_vsil_hdfs.cpp
View file
Open in desktop
Expand Up @@ -217,6 +217,11 @@ class VSIHdfsFilesystemHandler final : public VSIFilesystemHandler
VSIHdfsFilesystemHandler();
~VSIHdfsFilesystemHandler() override;

void EnsureFilesystem() {
if (poFilesystem == nullptr)
poFilesystem = hdfsConnect("default", 0);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

inconsistent indentation. And implementation would be best outside of the class declaration. And you could also move the CPLMutexHolder in this method, since it is needed just for this lazy instanciation

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, done.

}

VSIVirtualHandle *Open(const char *pszFilename,
const char *pszAccess,
bool bSetError ) override;
Expand Down Expand Up @@ -250,10 +255,9 @@ VSIHdfsFilesystemHandler::Open( const char *pszFilename,
const char *pszAccess,
bool)
{
CPLMutexHolder oHolder( &hMutex );
CPLMutexHolder oHolder(&hMutex);

if (poFilesystem == nullptr)
poFilesystem = hdfsConnect("default", 0);
EnsureFilesystem();

if (strchr(pszAccess, 'w') != nullptr || strchr(pszAccess, 'a') != nullptr) {
CPLError(CE_Failure, CPLE_AppDefined, "HDFS driver is read-only");
Expand Down Expand Up @@ -281,7 +285,9 @@ VSIHdfsFilesystemHandler::Open( const char *pszFilename,
int
VSIHdfsFilesystemHandler::Stat( const char *pszeFilename, VSIStatBufL *pStatBuf, int)
{
CPLMutexHolder oHolder( &hMutex );
CPLMutexHolder oHolder(&hMutex);

EnsureFilesystem();

hdfsFileInfo * poInfo = hdfsGetPathInfo(poFilesystem, pszeFilename);
memset(pStatBuf, 0, sizeof(*pStatBuf));
Expand Down Expand Up @@ -319,6 +325,11 @@ VSIHdfsFilesystemHandler::Stat( const char *pszeFilename, VSIStatBufL *pStatBuf,
int
VSIHdfsFilesystemHandler::Unlink(const char *pszFilename)
{
{
CPLMutexHolder oHolder(&hMutex);
EnsureFilesystem();
}

return hdfsDelete(poFilesystem, pszFilename, 0);
}

Expand All @@ -332,12 +343,22 @@ VSIHdfsFilesystemHandler::Mkdir(const char *pszDirname, long nMode)
int
VSIHdfsFilesystemHandler::Rmdir(const char *pszDirname)
{
{
CPLMutexHolder oHolder(&hMutex);
EnsureFilesystem();
}

return hdfsDelete(poFilesystem, pszDirname, 1);
}

char **
VSIHdfsFilesystemHandler::ReadDir(const char *pszDirname)
{
{
CPLMutexHolder oHolder(&hMutex);
EnsureFilesystem();
}

int mEntries = 0;
hdfsFileInfo * paoInfo = hdfsListDirectory(poFilesystem, pszDirname, &mEntries);
char ** retval = nullptr;
Expand All @@ -356,6 +377,11 @@ VSIHdfsFilesystemHandler::ReadDir(const char *pszDirname)
int
VSIHdfsFilesystemHandler::Rename(const char *oldpath, const char *newpath)
{
{
CPLMutexHolder oHolder(&hMutex);
EnsureFilesystem();
}

return hdfsRename(poFilesystem, oldpath, newpath);
}

Expand Down