Skip to content

Commit

Permalink
Revert "[HIP] Support compressing device binary (#67162)"
Browse files Browse the repository at this point in the history
This reverts commit a1e81d2.

Revert "Fix test hip-offload-compress-zlib.hip"

This reverts commit ba01ce6.

Revert due to sanity fail at

https://lab.llvm.org/buildbot/#/builders/5/builds/37188

https://lab.llvm.org/buildbot/#/builders/238/builds/5955

/b/sanitizer-aarch64-linux-bootstrap-ubsan/build/llvm-project/clang/lib/Driver/OffloadBundler.cpp:1012:25: runtime error: load of misaligned address 0xaaaae2d90e7c for type 'const uint64_t' (aka 'const unsigned long'), which requires 8 byte alignment
0xaaaae2d90e7c: note: pointer points here
  bc 00 00 00 94 dc 29 9a  89 fb ca 2b 78 9c 8b 8f  77 f6 71 f4 73 8f f7 77  73 f3 f1 77 74 89 77 0a
              ^
    #0 0xaaaaba125f70 in clang::CompressedOffloadBundle::decompress(llvm::MemoryBuffer const&, bool) /b/sanitizer-aarch64-linux-bootstrap-ubsan/build/llvm-project/clang/lib/Driver/OffloadBundler.cpp:1012:25
    #1 0xaaaaba126150 in clang::OffloadBundler::ListBundleIDsInFile(llvm::StringRef, clang::OffloadBundlerConfig const&) /b/sanitizer-aarch64-linux-bootstrap-ubsan/build/llvm-project/clang/lib/Driver/OffloadBundler.cpp:1089:7

Will reland after fixing it.
  • Loading branch information
yxsamliu committed Oct 5, 2023
1 parent f28f09d commit c6ed5a6
Show file tree
Hide file tree
Showing 16 changed files with 60 additions and 652 deletions.
27 changes: 0 additions & 27 deletions clang/docs/ClangOffloadBundler.rst
Original file line number Diff line number Diff line change
Expand Up @@ -309,30 +309,3 @@ target by comparing bundle ID's. Two bundle ID's are considered compatible if:
* Their offload kind are the same
* Their target triple are the same
* Their GPUArch are the same

Compression and Decompression
=============================

``clang-offload-bundler`` provides features to compress and decompress the full
bundle, leveraging inherent redundancies within the bundle entries. Use the
`-compress` command-line option to enable this compression capability.

The compressed offload bundle begins with a header followed by the compressed binary data:

- **Magic Number (4 bytes)**:
This is a unique identifier to distinguish compressed offload bundles. The value is the string 'CCOB' (Compressed Clang Offload Bundle).

- **Version Number (16-bit unsigned int)**:
This denotes the version of the compressed offload bundle format. The current version is `1`.

- **Compression Method (16-bit unsigned int)**:
This field indicates the compression method used. The value corresponds to either `zlib` or `zstd`, represented as a 16-bit unsigned integer cast from the LLVM compression enumeration.

- **Uncompressed Binary Size (32-bit unsigned int)**:
This is the size (in bytes) of the binary data before it was compressed.

- **Hash (64-bit unsigned int)**:
This is a 64-bit truncated MD5 hash of the uncompressed binary data. It serves for verification and caching purposes.

- **Compressed Data**:
The actual compressed binary data follows the header. Its size can be inferred from the total size of the file minus the header size.
37 changes: 0 additions & 37 deletions clang/include/clang/Driver/OffloadBundler.h
Original file line number Diff line number Diff line change
Expand Up @@ -19,23 +19,18 @@

#include "llvm/Support/Error.h"
#include "llvm/TargetParser/Triple.h"
#include <llvm/Support/MemoryBuffer.h>
#include <string>
#include <vector>

namespace clang {

class OffloadBundlerConfig {
public:
OffloadBundlerConfig();

bool AllowNoHost = false;
bool AllowMissingBundles = false;
bool CheckInputArchive = false;
bool PrintExternalCommands = false;
bool HipOpenmpCompatible = false;
bool Compress = false;
bool Verbose = false;

unsigned BundleAlignment = 1;
unsigned HostInputIndex = ~0u;
Expand Down Expand Up @@ -89,38 +84,6 @@ struct OffloadTargetInfo {
std::string str() const;
};

// CompressedOffloadBundle represents the format for the compressed offload
// bundles.
//
// The format is as follows:
// - Magic Number (4 bytes) - A constant "CCOB".
// - Version (2 bytes)
// - Compression Method (2 bytes) - Uses the values from
// llvm::compression::Format.
// - Uncompressed Size (4 bytes).
// - Truncated MD5 Hash (8 bytes).
// - Compressed Data (variable length).

class CompressedOffloadBundle {
private:
static inline const size_t MagicSize = 4;
static inline const size_t VersionFieldSize = sizeof(uint16_t);
static inline const size_t MethodFieldSize = sizeof(uint16_t);
static inline const size_t SizeFieldSize = sizeof(uint32_t);
static inline const size_t HashFieldSize = 8;
static inline const size_t HeaderSize = MagicSize + VersionFieldSize +
MethodFieldSize + SizeFieldSize +
HashFieldSize;
static inline const llvm::StringRef MagicNumber = "CCOB";
static inline const uint16_t Version = 1;

public:
static llvm::Expected<std::unique_ptr<llvm::MemoryBuffer>>
compress(const llvm::MemoryBuffer &Input, bool Verbose = false);
static llvm::Expected<std::unique_ptr<llvm::MemoryBuffer>>
decompress(const llvm::MemoryBuffer &Input, bool Verbose = false);
};

} // namespace clang

#endif // LLVM_CLANG_DRIVER_OFFLOADBUNDLER_H
4 changes: 0 additions & 4 deletions clang/include/clang/Driver/Options.td
Original file line number Diff line number Diff line change
Expand Up @@ -1183,10 +1183,6 @@ def fgpu_inline_threshold_EQ : Joined<["-"], "fgpu-inline-threshold=">,
def fgpu_sanitize : Flag<["-"], "fgpu-sanitize">, Group<f_Group>,
HelpText<"Enable sanitizer for supported offloading devices">;
def fno_gpu_sanitize : Flag<["-"], "fno-gpu-sanitize">, Group<f_Group>;

def offload_compress : Flag<["--"], "offload-compress">,
HelpText<"Compress offload device binaries (HIP only)">;
def no_offload_compress : Flag<["--"], "no-offload-compress">;
}

// CUDA options
Expand Down

0 comments on commit c6ed5a6

Please sign in to comment.