Skip to content

Commit

Permalink
Merge pull request #1014 from williamfgc/big_endian_no_lfs
Browse files Browse the repository at this point in the history 
Big endian no git lfs
  • Loading branch information
@williamfgc
williamfgc committed Nov 27, 2018
2 parents 3a2df8b + ff571cc commit 1a534b8
Show file tree
Hide file tree
Showing 13 changed files with 417 additions and 197 deletions.
3 changes: 3 additions & 0 deletions CMakeLists.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ adios_option(HDF5 "Enable support for the HDF5 engine" AUTO)
adios_option(Python "Enable support for Python bindings" AUTO)
adios_option(Fortran "Enable support for Fortran bindings" AUTO)
adios_option(SysVShMem "Enable support for SysV Shared Memory IPC on *NIX" AUTO)
adios_option(Endian_Reverse "Enable support for Little/Big Endian Interoprability" AUTO)
include(${PROJECT_SOURCE_DIR}/cmake/DetectOptions.cmake)

if(ADIOS2_HAVE_MPI)
Expand All @@ -142,8 +143,10 @@ if(ADIOS2_HAVE_MPI)
add_definitions(-DOMPI_SKIP_MPICXX -DMPICH_SKIP_MPICXX)
endif()


set(ADIOS2_CONFIG_OPTS
BZip2 ZFP SZ MGARD MPI DataMan SST ZeroMQ HDF5 Python Fortran SysVShMem
Endian_Reverse
)
GenerateADIOSHeaderConfig(${ADIOS2_CONFIG_OPTS})
configure_file(
Expand Down
4 changes: 4 additions & 0 deletions cmake/DetectOptions.cmake
View file
Original file line number Diff line number Diff line change
Expand Up @@ -177,5 +177,9 @@ else()
set(ADIOS2_HAVE_SysVShMem OFF)
endif()

if(ADIOS2_USE_Endian_Reverse STREQUAL ON)
set(ADIOS2_HAVE_Endian_Reverse TRUE)
endif()

# Multithreading
find_package(Threads REQUIRED)
29 changes: 15 additions & 14 deletions docs/user_guide/source/installation/cmake.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -193,20 +193,21 @@ Build CMake -DVAR=VALUE Options
The following options can be specified with CMake's `-DVAR=VALUE` syntax to control which features get enabled or disabled, default option (ON/OFF) is highlighted:
====================== ========================= ==========================================================================================================================================================================================================================
CMake VAR Option Values Description
====================== ========================= ==========================================================================================================================================================================================================================
`ADIOS2_USE_MPI` **`AUTO`**/``ON``/OFF Enable MPI or non-MPI (serial) build.
`ADIOS2_USE_DataMan` **`AUTO`**/``ON``/OFF Enable the DataMan engine for Wide-Area-Network transports.
`ADIOS2_USE_ZeroMQ` **`AUTO`**/``ON``/OFF Enable `ZeroMQ <http://zeromq.org/>`_ for the DataMan engine.
`ADIOS2_USE_HDF5` **`AUTO`**/``ON``/OFF Enable the `HDF5 <https://www.hdfgroup.org>`_ engine. If HDF5 is not in the path or not the correct version is in the path, set the correct path by the -DHDF5_ROOT=... option
`ADIOS2_USE_Python` **`AUTO`**/``ON``/OFF Enable the Python >= 2.7 bindings. mpi4py and numpy. Python 3 will be used if Python 2 and 3 are found. If you want a python version not in the path then choose the right pyhton executable by -DPYTHON_EXECUTABLE=...
`ADIOS2_USE_Fortran` **`AUTO`**/``ON``/OFF Enable the Fortran 90 or above bindings. Must have a Fortran compiler. Default is OFF, must be explicitly set to ON.
`ADIOS2_USE_SST` **`AUTO`**/``ON``/OFF Enable the Simplified Staging Engine (SST) and its dependencies, requires MPI.
`ADIOS2_USE_BZip2` **`AUTO`**/``ON``/OFF Enable `BZip2 <http://www.bzip.org>`_ compression (experimental, not yet implemented).
`ADIOS2_USE_ZFP` **`AUTO`**/``ON``/OFF Enable `ZFP <https://github.com/LLNL/zfp>`_ compression (experimental).
`ADIOS2_USE_SZ` **`AUTO`**/``ON``/OFF Enable `ZFP <https://github.com/disheng222/SZ>`_ compression (experimental).
====================== ========================= ==========================================================================================================================================================================================================================
============================= ========================= ==========================================================================================================================================================================================================================
CMake VAR Option Values Description
============================= ========================= ==========================================================================================================================================================================================================================
`ADIOS2_USE_MPI` **`AUTO`**/``ON``/OFF MPI or non-MPI (serial) build.
`ADIOS2_USE_ZeroMQ` **`AUTO`**/``ON``/OFF `ZeroMQ <http://zeromq.org/>`_ for the DataMan engine.
`ADIOS2_USE_HDF5` **`AUTO`**/``ON``/OFF `HDF5 <https://www.hdfgroup.org>`_ engine. If HDF5 is not in the path or not the correct version is in the path, set the correct path by the -DHDF5_ROOT=... option
`ADIOS2_USE_Python` **`AUTO`**/``ON``/OFF Python >= 2.7 bindings. mpi4py and numpy. Python 3 will be used if Python 2 and 3 are found. If you want a python version not in the path then choose the right pyhton executable by -DPYTHON_EXECUTABLE=...
`ADIOS2_USE_Fortran` **`AUTO`**/``ON``/OFF Fortran 90 or above bindings. Must have a Fortran compiler. Default is OFF, must be explicitly set to ON.
`ADIOS2_USE_SST` **`AUTO`**/``ON``/OFF Simplified Staging Engine (SST) and its dependencies, requires MPI.
`ADIOS2_USE_BZip2` **`AUTO`**/``ON``/OFF `BZip2 <http://www.bzip.org>`_ compression (experimental, not yet implemented).
`ADIOS2_USE_ZFP` **`AUTO`**/``ON``/OFF `ZFP <https://github.com/LLNL/zfp>`_ compression (experimental).
`ADIOS2_USE_SZ` **`AUTO`**/``ON``/OFF `SZ <https://github.com/disheng222/SZ>`_ compression (experimental).
`ADIOS2_USE_MGARD` **`AUTO`**/``ON``/OFF `MGARD <https://github.com/CODARcode/MGARD>`_ compression (experimental).
`ADIOS2_USE_Endian_Reverse` **`AUTO`**/ON/``OFF`` Big/Little Endian Interoperability for different endianness platforms at write and read.
============================= ========================= ==========================================================================================================================================================================================================================
Examples: Enable Fortran, disable Python bindings and ZeroMQ functionality
Expand Down
12 changes: 11 additions & 1 deletion source/adios2/engine/bp3/BP3Reader.tcc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -146,12 +146,22 @@ void BP3Reader::ReadVariableBlocks(Variable<T> &variable)
helper::StartCountBox(subStreamInfo.BlockBox.first,
subStreamInfo.BlockBox.second);

// TODO: this should be a single BP3 deserializer function
#ifdef ADIOS2_HAVE_ENDIAN_REVERSE
const bool endianReverse =
(helper::IsLittleEndian() !=
m_BP3Deserializer.m_Minifooter.IsLittleEndian)
? true
: false;
#else
constexpr bool endianReverse = false;
#endif
helper::CopyMemory(
blockInfo.Data, blockInfo.Start, blockInfo.Count,
helper::IsRowMajor(m_IO.m_HostLanguage),
reinterpret_cast<T *>(variable.m_RawMemory[0].data()),
sourceStartCount.first, sourceStartCount.second,
m_BP3Deserializer.m_IsRowMajor);
m_BP3Deserializer.m_IsRowMajor, endianReverse);
}
// advance pointer to next step
blockInfo.Data += helper::GetTotalSize(blockInfo.Count);
Expand Down
91 changes: 76 additions & 15 deletions source/adios2/helper/adiosMemory.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@

#include <algorithm>

#include "adios2/helper/adiosType.h"

namespace adios2
{
namespace helper
Expand All @@ -20,6 +22,65 @@ namespace helper
namespace
{

template <class T>
inline void CopyEndianReverse(const char *src, const size_t payloadStride,
T *dest)
{
if (sizeof(T) == 1)
{
std::copy(src, src + payloadStride, reinterpret_cast<char *>(dest));
return;
}

std::reverse_copy(src, src + payloadStride, reinterpret_cast<char *>(dest));
std::reverse(dest, dest + payloadStride / sizeof(T));
}

template <>
void CopyEndianReverse(const char *src, const size_t payloadStride,
std::complex<float> *dest)
{
std::reverse_copy(src, src + payloadStride, reinterpret_cast<char *>(dest));
float *destF = reinterpret_cast<float *>(dest);
std::reverse(destF, destF + payloadStride / sizeof(float));
}

template <>
void CopyEndianReverse(const char *src, const size_t payloadStride,
std::complex<double> *dest)
{
std::reverse_copy(src, src + payloadStride, reinterpret_cast<char *>(dest));
double *destF = reinterpret_cast<double *>(dest);
std::reverse(destF, destF + payloadStride / sizeof(double));
}

void CopyPayloadStride(const char *src, const size_t payloadStride, char *dest,
const bool endianReverse, const std::string destType)
{
#ifdef ADIOS2_HAVE_ENDIAN_REVERSE
if (endianReverse)
{
if (destType == "")
{
}
#define declare_type(T) \
else if (destType == GetType<T>()) \
{ \
CopyEndianReverse<T>(src, payloadStride, reinterpret_cast<T *>(dest)); \
}

ADIOS2_FOREACH_PRIMITIVE_TYPE_1ARG(declare_type)
#undef declare_type
}
else
{
std::copy(src, src + payloadStride, dest);
}
#else
std::copy(src, src + payloadStride, dest);
#endif
}

Dims DestDimsFinal(const Dims &destDims, const bool destRowMajor,
const bool srcRowMajor)
{
Expand All @@ -35,7 +96,8 @@ void ClipRowMajor(char *dest, const Dims &destStart, const Dims &destCount,
const bool destRowMajor, const char *src,
const Dims &srcStart, const Dims &srcCount,
const Dims & /*destMemStart*/, const Dims & /*destMemCount*/,
const Dims &srcMemStart, const Dims &srcMemCount)
const Dims &srcMemStart, const Dims &srcMemCount,
const bool endianReverse, const std::string destType)
{
const Dims destStartFinal = DestDimsFinal(destStart, destRowMajor, true);
const Dims destCountFinal = DestDimsFinal(destCount, destRowMajor, true);
Expand Down Expand Up @@ -95,13 +157,11 @@ void ClipRowMajor(char *dest, const Dims &destStart, const Dims &destCount,
srcMemStart),
true);

const size_t srcEndOffset = srcBeginOffset + stride;

const size_t destBeginOffset = helper::LinearIndex(
destStartFinal, destCountFinal, currentPoint, true);

std::copy(src + srcBeginOffset, src + srcEndOffset,
dest + destBeginOffset);
CopyPayloadStride(src + srcBeginOffset, stride, dest + destBeginOffset,
endianReverse, destType);

size_t p = startCoord;
while (true)
Expand Down Expand Up @@ -133,7 +193,8 @@ void ClipColumnMajor(char *dest, const Dims &destStart, const Dims &destCount,
const Dims &srcStart, const Dims &srcCount,
const Dims & /*destMemStart*/,
const Dims & /*destMemCount*/, const Dims &srcMemStart,
const Dims &srcMemCount)
const Dims &srcMemCount, const bool endianReverse,
const std::string destType)
{
const Dims destStartFinal = DestDimsFinal(destStart, destRowMajor, false);
const Dims destCountFinal = DestDimsFinal(destCount, destRowMajor, false);
Expand Down Expand Up @@ -187,8 +248,8 @@ void ClipColumnMajor(char *dest, const Dims &destStart, const Dims &destCount,
const size_t destBeginOffset = helper::LinearIndex(
destStartFinal, destCountFinal, currentPoint, false);

std::copy(src + srcBeginOffset, src + srcEndOffset,
dest + destBeginOffset);
CopyPayloadStride(src + srcBeginOffset, stride, dest + destBeginOffset,
endianReverse, destType);
size_t p = startCoord;

while (true)
Expand Down Expand Up @@ -221,7 +282,8 @@ void CopyPayload(char *dest, const Dims &destStart, const Dims &destCount,
const bool destRowMajor, const char *src, const Dims &srcStart,
const Dims &srcCount, const bool srcRowMajor,
const Dims &destMemStart, const Dims &destMemCount,
const Dims &srcMemStart, const Dims &srcMemCount) noexcept
const Dims &srcMemStart, const Dims &srcMemCount,
const bool endianReverse, const std::string destType) noexcept
{
if (srcStart.size() == 1) // 1D copy memory
{
Expand All @@ -235,26 +297,25 @@ void CopyPayload(char *dest, const Dims &destStart, const Dims &destCount,
? interStart.front() - srcStart.front()
: interStart.front() - srcStart.front() + srcMemStart.front();

const size_t srcEndOffset = srcBeginOffset + interCount.front();

const size_t stride = interCount.front();
const size_t destBeginOffset = interStart.front() - destStart.front();

std::copy(src + srcBeginOffset, src + srcEndOffset,
dest + destBeginOffset);
CopyPayloadStride(src + srcBeginOffset, stride, dest + destBeginOffset,
endianReverse, destType);
return;
}

if (srcRowMajor) // stored with C, C++, Python
{
ClipRowMajor(dest, destStart, destCount, destRowMajor, src, srcStart,
srcCount, destMemStart, destMemCount, srcMemStart,
srcMemCount);
srcMemCount, endianReverse, destType);
}
else // stored with Fortran, R
{
ClipColumnMajor(dest, destStart, destCount, destRowMajor, src, srcStart,
srcCount, destMemStart, destMemCount, srcMemStart,
srcMemCount);
srcMemCount, endianReverse, destType);
}
}

Expand Down
12 changes: 10 additions & 2 deletions source/adios2/helper/adiosMemory.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,10 @@ void CopyToBufferThreads(std::vector<char> &buffer, size_t &position,
const T *source, const size_t elements = 1,
const unsigned int threads = 1) noexcept;

template <class T>
void ReverseCopyFromBuffer(const std::vector<char> &buffer, size_t &position,
T *destination, const size_t elements = 1) noexcept;

/**
* Copy memory from a buffer at a certain input position
* @param buffer data source
Expand All @@ -82,7 +86,8 @@ template <class T>
void InsertU64(std::vector<char> &buffer, const T element) noexcept;

template <class T>
T ReadValue(const std::vector<char> &buffer, size_t &position) noexcept;
T ReadValue(const std::vector<char> &buffer, size_t &position,
const bool isLittleEndian = true) noexcept;

/**
* General function to copy memory between blocks of different type and start
Expand All @@ -104,6 +109,7 @@ template <class T, class U>
void CopyMemory(T *dest, const Dims &destStart, const Dims &destCount,
const bool destRowMajor, const U *src, const Dims &srcStart,
const Dims &srcCount, const bool srcRowMajor,
const bool endianReverse = false,
const Dims &destMemStart = Dims(),
const Dims &destMemCount = Dims(),
const Dims &srcMemStart = Dims(),
Expand All @@ -115,7 +121,9 @@ void CopyPayload(char *dest, const Dims &destStart, const Dims &destCount,
const Dims &destMemStart = Dims(),
const Dims &destMemCount = Dims(),
const Dims &srcMemStart = Dims(),
const Dims &srcMemCount = Dims()) noexcept;
const Dims &srcMemCount = Dims(),
const bool endianReverse = false,
const std::string destType = "") noexcept;

/**
* Clips the contiguous memory corresponding to an intersection and puts it in
Expand Down

0 comments on commit 1a534b8

Please sign in to comment.