Skip to content

Commit

Permalink
Merge pull request #345 from sisong/dev
Browse files Browse the repository at this point in the history 
Add support for block devices on linux
  • Loading branch information
@sisong
sisong committed May 19, 2023
2 parents ec9bc70 + f21ff10 commit b2555ed
Show file tree
Hide file tree
Showing 10 changed files with 147 additions and 31 deletions.
9 changes: 5 additions & 4 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ full changelog at: https://github.com/sisong/HDiffPatch/commits
## [v4.5.0](https://github.com/sisong/HDiffPatch/tree/v4.5.0) - 2022-11-23
### Added
* cmdline added option "-VCD[-compressLevel[-dictSize]]", create diffFile compatible with VCDIFF format;
* cmdline hpatchz support apply VCDIFF format diffFile, created by hdiffz -VCD,xdelta3,open-vcdiff;
* cmdline hpatchz support apply VCDIFF format diffFile, created by `$hdiffz -VCD`,`$xdelta3`,`$open-vcdiff`;
* add function create_vcdiff(),create_vcdiff_stream(), vcpatch_with_cache();
* cmdline option "-BSD" support run with -s (before only supported -m), to fast create diffFile compatible with bsdiff4;
* add function create_bsdiff_stream(), same as create_bsdiff(), but can control memory requires and run speed by different kMatchBlockSize value;
Expand All @@ -37,9 +37,10 @@ full changelog at: https://github.com/sisong/HDiffPatch/commits
## [v4.1.0](https://github.com/sisong/HDiffPatch/tree/v4.1.0) - 2021-11-27
### Added
* cmdline add option "-BSD", to create diffFile compatible with bsdiff4;
* cmdline hpatchz support apply bsdiff4 format diffFile, created by `$hdiffz -BSD`,`$bsdiff`;
* cmdline add option "-cache", optimize `$hdiffz -m` speed; note:the big cache max used O(oldFileSize) memory, and build slow.
* cmdline add option "-block", optimize `$hdiffz -m` speed;
* add function create_bsdiff() & bspatch_with_cache().
* cmdline add option "-cache", optimize hdiffz -m speed; note:the big cache max used O(oldFileSize) memory, and build slow.
* cmdline add option "-block", optimize hdiffz -m speed;
* add function create_single_compressed_diff_block() & create_compressed_diff_block() & create_bsdiff_block().

## [v4.0.0](https://github.com/sisong/HDiffPatch/tree/v4.0.0) - 2021-06-14
Expand Down Expand Up @@ -73,7 +74,7 @@ full changelog at: https://github.com/sisong/HDiffPatch/commits

## [v3.0.7](https://github.com/sisong/HDiffPatch/tree/v3.0.7) - 2019-12-19
### Fixed
* fix a bug when create dir's manifest file without checksum (hdiffz -C-no -M...);
* fix a bug when create dir's manifest file without checksum (`$hdiffz -C-no -M...`);
* fix a bug when create dir's manifest file on windows operating system;

## [v3.0.6](https://github.com/sisong/HDiffPatch/tree/v3.0.6) - 2019-09-18
Expand Down
4 changes: 2 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# [HDiffPatch](https://github.com/sisong/HDiffPatch)
[![release](https://img.shields.io/badge/release-v4.6.2-blue.svg)](https://github.com/sisong/HDiffPatch/releases)
[![release](https://img.shields.io/badge/release-v4.6.3-blue.svg)](https://github.com/sisong/HDiffPatch/releases)
[![license](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/sisong/HDiffPatch/blob/master/LICENSE)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-blue.svg)](https://github.com/sisong/HDiffPatch/pulls)
[![+issue Welcome](https://img.shields.io/github/issues-raw/sisong/HDiffPatch?color=green&label=%2Bissue%20welcome)](https://github.com/sisong/HDiffPatch/issues)
Expand All @@ -17,7 +17,7 @@ update your own Android Apk? Jar or Zip file diff & patch? try [ApkDiffPatch](ht

[sfpatcher](https://github.com/sisong/sfpatcher) not require re-signing apks (like [archive-patcher](https://github.com/google/archive-patcher)), is designed for Android app store, patch speed up by a factor of xx than archive-patcher & run with O(1) memory.

if you not have the old versions(too many or not obtain or have been modified), thus cannot create the patch in advance. you can see demo [hsynz](https://github.com/sisong/hsynz) (like [zsync](http://zsync.moria.org.uk)), the new version is only need released once and the owners of the old version get the information about the new version and do the diff&patch themselves. hsynz support zstd compressor & run faster than zsync.
if you not have the old versions(too many or not obtain or have been modified), thus cannot create the delta in advance. you can see demo [hsynz](https://github.com/sisong/hsynz) (like [zsync](http://zsync.moria.org.uk)), the new version is only need released once and the owners of the old version get the information about the new version and do the diff&patch themselves. hsynz support zstd compressor & run faster than zsync.

NOTE: *This library does not deal with file metadata, such as file last wirte time, permissions, link file, etc... To this library, a file is just as a stream of bytes; You can extend this library or use other tools.*

Expand Down
2 changes: 1 addition & 1 deletion builds/vc/testHashClash.vcxproj
View file
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@
<SubSystem>Console</SubSystem>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
<GenerateDebugInformation>false</GenerateDebugInformation>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
</ItemDefinitionGroup>
<ItemGroup>
Expand Down
42 changes: 39 additions & 3 deletions file_for_patch.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,34 @@
#endif
*/

#if (_IS_NEED_BLOCK_DEV)
#include <sys/types.h>
#include <fcntl.h>
#include <sys/ioctl.h> // ioctl
#include <linux/fs.h> //BLKGETSIZE64

static hpatch_BOOL _get_block_dev_size(const char* blkdev,hpatch_uint64_t* bsize){
int fd;
assert(blkdev&&bsize);

fd=open(blkdev,O_RDONLY);
if (fd == -1){
perror("_get_block_dev_size() open");
return hpatch_FALSE;
}

if (ioctl(fd,BLKGETSIZE64,bsize) == -1) {
perror("_get_block_dev_size() ioctl");
close(fd);
return hpatch_FALSE;
}

close(fd);
return hpatch_TRUE;
}
#endif


hpatch_BOOL _hpatch_getPathStat_noEndDirSeparator(const char* path_utf8,hpatch_TPathType* out_type,
hpatch_StreamPos_t* out_fileSize,size_t* out_st_mode){
#if (_IS_USED_WIN32_UTF8_WAPI)
Expand Down Expand Up @@ -63,6 +91,7 @@ hpatch_BOOL _hpatch_getPathStat_noEndDirSeparator(const char* path_utf8,hpatch_T
rt = stat(path_utf8,&s);
# endif
#endif
if (out_st_mode) *out_st_mode=s.st_mode;

if(rt!=0){
if (errno==ENOENT){
Expand All @@ -73,13 +102,20 @@ hpatch_BOOL _hpatch_getPathStat_noEndDirSeparator(const char* path_utf8,hpatch_T
}else if ((s.st_mode&S_IFMT)==S_IFREG){
*out_type=kPathType_file;
if (out_fileSize) *out_fileSize=s.st_size;
if (out_st_mode) *out_st_mode=s.st_mode;
return hpatch_TRUE;
}else if ((s.st_mode&S_IFMT)==S_IFDIR){
*out_type=kPathType_dir;
if (out_fileSize) *out_fileSize=0;
if (out_st_mode) *out_st_mode=s.st_mode;
return hpatch_TRUE;
#if (_IS_NEED_BLOCK_DEV)
}else if ((s.st_mode&S_IFMT)==S_IFBLK){
hpatch_uint64_t bsize=0;
if (!_get_block_dev_size(path_utf8, &bsize))
return hpatch_FALSE;
*out_type=kPathType_file;
if (out_fileSize) *out_fileSize=bsize;
return hpatch_TRUE;
#endif
}else{
return hpatch_FALSE; //as error; unknow how to dispose
}
Expand Down Expand Up @@ -543,4 +579,4 @@ hpatch_BOOL hpatch_TFileStreamOutput_close(hpatch_TFileStreamOutput* self){
if (!_import_fileClose(&self->m_file))
_ferr_return();
return hpatch_TRUE;
}
}
6 changes: 6 additions & 0 deletions file_for_patch.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,12 @@
# define mix_ferr(_saved_errno,_throw_errno) set_ferr(_saved_errno,_throw_errno)
#endif

#if ( __linux__ )
# ifndef _IS_NEED_BLOCK_DEV
# define _IS_NEED_BLOCK_DEV 1
# endif
#endif

#ifndef _IS_USED_WIN32_UTF8_WAPI
# if (defined(_WIN32) && defined(_MSC_VER))
# define _IS_USED_WIN32_UTF8_WAPI 1 // used utf8 string + wchar_t API
Expand Down
2 changes: 1 addition & 1 deletion libHDiffPatch/HPatch/patch_types.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ extern "C" {

#define HDIFFPATCH_VERSION_MAJOR 4
#define HDIFFPATCH_VERSION_MINOR 6
#define HDIFFPATCH_VERSION_RELEASE 2
#define HDIFFPATCH_VERSION_RELEASE 3

#define _HDIFFPATCH_VERSION HDIFFPATCH_VERSION_MAJOR.HDIFFPATCH_VERSION_MINOR.HDIFFPATCH_VERSION_RELEASE
#define _HDIFFPATCH_QUOTE(str) #str
Expand Down
4 changes: 2 additions & 2 deletions libhsync/sync_client/match_in_old.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ struct TOldDataCache:public TOldDataCache_base {
:TOldDataCache_base(oldStream,oldRollBegin,oldRollEnd,kSyncBlockSize,
strongChecksumPlugin,_mt){
if (isEnd()) return;
m_rollHash=roll_hash_start((tm_roll_uint*)0,m_cur,m_kSyncBlockSize);
m_rollHash=roll_hash_start(m_cur,m_kSyncBlockSize);
}
bool _cacheAndRoll(){
TOldDataCache_base::_cache();
Expand All @@ -260,7 +260,7 @@ struct TOldDataCache:public TOldDataCache_base {
TOldDataCache_base::_cache();
if (isEnd()) return false;
}
m_rollHash=roll_hash_start((tm_roll_uint*)0,m_cur,m_kSyncBlockSize);
m_rollHash=roll_hash_start(m_cur,m_kSyncBlockSize);
return true;
}
protected:
Expand Down
2 changes: 1 addition & 1 deletion libhsync/sync_client/sync_client_type_private.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ uint32_t TNewDataSyncInfo_syncBlockSize(const TNewDataSyncInfo* self,uint32_t bl
return TNewDataSyncInfo_newDataBlockSize(self,blockIndex);
}

inline static uint64_t roll_hash_start(uint64_t*,const adler_data_t* pdata,size_t n){
inline static uint64_t roll_hash_start(const adler_data_t* pdata,size_t n){
return fast_adler64_start(pdata,n); }
inline static uint64_t roll_hash_roll(uint64_t adler,size_t blockSize,
adler_data_t out_data,adler_data_t in_data){
Expand Down
2 changes: 1 addition & 1 deletion libhsync/sync_make/sync_make.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -297,7 +297,7 @@ static void _create_sync_data_part(_TCreateDatas& cd,TWorkBuf* workData,
if (out_hsyni->savedSizes) //save compressedSize
out_hsyni->savedSizes[i]=(uint32_t)compressedSize;

const uint64_t rollHash=roll_hash_start((uint64_t*)0,dataBuf,kSyncBlockSize);
const uint64_t rollHash=roll_hash_start(dataBuf,kSyncBlockSize);
//strong hash
checksumBlockData.appendBegin();
checksumBlockData.append(dataBuf,dataBuf+kSyncBlockSize);
Expand Down
105 changes: 89 additions & 16 deletions test/testHashClash.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,27 @@ typedef unsigned char TByte;
typedef ptrdiff_t TInt;
typedef size_t TUInt;

#define _IS_USES_MY_RAND
#ifdef _IS_USES_MY_RAND
class CMyRand {
public:
unsigned int _my_holdrand;
public:
inline CMyRand() :_my_holdrand(1) {}
inline int _my_rand() {
unsigned int result = _my_holdrand * 214013 + 2531011;
_my_holdrand = result;
return (result >> 16) & RAND_MAX;
}
};
static CMyRand _MyRand;
inline int _rand() { return _MyRand._my_rand(); }
inline void _srand(unsigned int seed) { _MyRand._my_holdrand = seed; }
#else
#define _rand rand
#define _srand srand
#endif

/* //interface
struct THash{
typename TValue;
Expand Down Expand Up @@ -108,11 +129,23 @@ struct THash_adler32h{
inline void hash_begin() { _hv=adler32_start(0,0); }
inline void hash(const TByte* pdata,const TByte* pdata_end)
{ _hv=adler32_append(_hv,pdata,(pdata_end-pdata));
//assert(_hv==(TValue)adler32(1,pdata,(pdata_end-pdata)));
//assert(_hv==(TValue)adler32(_hv,pdata,(pdata_end-pdata)));
}
inline void hash_end(TValue* hv) { *hv=_hv; }
};

template<size_t kTestBit>
struct THash_adler32h_bit{
typedef uint32_t TValue;
inline static const char* name() { return "adler32h_bit"; }
TValue _hv;
inline void hash_begin() { _hv=adler32_start(0,0); }
inline void hash(const TByte* pdata,const TByte* pdata_end)
{ _hv=adler32_append(_hv,pdata,(pdata_end-pdata)); }
inline void hash_end(TValue* hv) { *hv=_hv&((1<<kTestBit)-1); }
};


struct THash_fadler32{
typedef uint32_t TValue;
inline static const char* name() { return "fadler32"; }
Expand Down Expand Up @@ -167,8 +200,9 @@ struct _adler128_t_cmp{

const uint64_t kMaxMapNodeSize=80000000ull; //run test memory ctrl
const size_t kRandTestMaxSize=1024*1024*1024;//test rand data size
const size_t kMaxHashDataSize=256;
size_t kMinClash=0; //run test max time ctrl
size_t kMinHashDataSize=0;
size_t kMaxHashDataSize=256;
size_t kMinClash=0; //run test max time ctrl

template <class THash,class TUInt,int kTestBit=sizeof(TUInt)*8,TUInt kTestMask=(TUInt)~(TUInt)0>
void test(const TByte* data,const TByte* data_end){
Expand All @@ -182,9 +216,9 @@ void test(const TByte* data,const TByte* data_end){
for (size_t m=0;m<clip_count;++m)
maps[m].reserve(kMaxMapNodeSize*3/clip_count);
unsigned int rand_seed=7;
srand(rand_seed);
_srand(rand_seed);
if (kTestMask!=(TUInt)~(TUInt)0) printf("mask[%08X %08X] ",(int)((uint64_t)kTestMask>>32),(int)kTestMask);
printf("%s%s ",THash::name(),std::string(10-strlen(THash::name()),' ').c_str());
printf("%s%s ",THash::name(),std::string(12-strlen(THash::name()),' ').c_str());

uint64_t curClashMin=0;
uint64_t clashs[clip_count]={0};
Expand All @@ -197,8 +231,8 @@ void test(const TByte* data,const TByte* data_end){
}
curClashMin=clashMin;

size_t dlen=rand() % kMaxHashDataSize;
size_t dstrat=((uint64_t)rand()+((uint64_t)rand()*(RAND_MAX+1))) % ((data_end-data)-dlen);
size_t dlen =kMinHashDataSize+(_rand() % (kMaxHashDataSize+1-kMinHashDataSize));
size_t dstrat=((uint64_t)_rand()+((uint64_t)_rand()*(RAND_MAX+1))) % ((data_end-data)-dlen);
assert(dstrat+dlen<=(size_t)(data_end-data));
const TByte* pv =data+dstrat;
const TByte* pv_end=pv+dlen;
Expand Down Expand Up @@ -283,7 +317,7 @@ void test_fadler128(const TByte* data,const TByte* data_end){
for (size_t m=0;m<clip_count;++m)
maps[m].reserve(kMaxMapNodeSize*3/clip_count);
unsigned int rand_seed=7;
srand(rand_seed);
_srand(rand_seed);
printf("mask[%08X %08X ",(int)(kTestMask1>>32),(int)kTestMask1);
printf("%08X %08X] ",(int)(kTestMask0>>32),(int)kTestMask0);
printf("%s%s ",THash::name(),std::string(10-strlen(THash::name()),' ').c_str());
Expand All @@ -299,8 +333,8 @@ void test_fadler128(const TByte* data,const TByte* data_end){
}
curClashMin=clashMin;

size_t dlen=rand() % kMaxHashDataSize;
size_t dstrat=rand() % ((data_end-data) - dlen);
size_t dlen =kMinHashDataSize+(_rand() % (kMaxHashDataSize+1-kMinHashDataSize));
size_t dstrat=_rand() % ((data_end-data) - dlen);
assert(dstrat+dlen<=(data_end-data));
const TByte* pv =data+dstrat;
const TByte* pv_end=pv+dlen;
Expand Down Expand Up @@ -388,19 +422,58 @@ int main() {

std::vector<TByte> data(kRandTestMaxSize);
unsigned int rand_seed=0;
srand(rand_seed);
_srand(rand_seed);
for (size_t i=0; i<data.size(); ++i) {
data[i]=(TByte)rand();
data[i]=(TByte)_rand();
}
const uint64_t kTestMask8=((((uint64_t)1)<<8)-1);
const uint64_t kTestMask16=((((uint64_t)1)<<16)-1);
const uint64_t kTestMask32=((((uint64_t)1)<<32)-1);
const uint64_t kTestMask64=~(uint64_t)0;


/* //only test adler32h by bit
kMinHashDataSize=1024;
kMaxHashDataSize=1024;
kMinClash=10000;
test<THash_adler32h_bit<1>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<2>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<3>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<4>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<5>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<6>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<7>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<8>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<9>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<10>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<11>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<12>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<13>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<14>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<15>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<16>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<17>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<18>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<19>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<20>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<21>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<22>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<23>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<24>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<25>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<26>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<27>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<28>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<29>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<30>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<31>,uint32_t>(data.data(),data.data()+data.size());
test<THash_adler32h_bit<32>,uint32_t>(data.data(),data.data()+data.size());
return 0;
//*/

//*
kMinClash=100000;
test<THash_adler32h,uint8_t>(data.data(), data.data() + data.size());
test<THash_adler32h,uint16_t>(data.data(), data.data() + data.size());
test<THash_adler32h,uint8_t>(data.data(),data.data()+data.size());
test<THash_adler32h,uint16_t>(data.data(),data.data()+data.size());
test<THash_adler32h,uint32_t>(data.data(),data.data()+data.size());
#if (_IS_NEED_ZLIB)
test<THash_adler32,uint32_t>(data.data(),data.data()+data.size());
Expand Down Expand Up @@ -431,7 +504,7 @@ int main() {
printf("\n");
//*/
//*
printf("NOTE: test fadler64 32bit ...\n");
printf("NOTE: test fadler64 32bit ...\n");
kMinClash=100000;
test<THash_fadler64,uint64_t,32,kTestMask32>(data.data(),data.data()+data.size());
test<THash_fadler64,uint64_t,32,kTestMask32<<32>(data.data(),data.data()+data.size());
Expand Down

0 comments on commit b2555ed

Please sign in to comment.