Optimize repair #1127
Merged
Optimize repair #1127
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* readPool unlock * feat (mobile): expose ReadPoolUnLock * show chunk size in listReq response * Revert "show chunk size in listReq response" This reverts commit c62a183.
* multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com>
* added memfilechan * added memfilechan * clean up * refactor download methods to accept file handler * fix unit tests * refactor download methods, implement new downloader, fix unit tests * clean up * fix file callback --------- Co-authored-by: dabasov <yuderbasov@gmail.com>
* Expose multioperation in winsdk * Temporary remove callback args from multiupload * Export method * pass callback * test multiop * fix multiUpload * Revert "test multiop" This reverts commit 08ad565. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> --------- Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com>
* add path in thumbnail hash * lint fix * fix typo * fix input s3_migration_cli_branch system_tests.yml * updated workflow input to s3_migration_cli_branch in system_tests.yml * fix hash calculation * check prevWM * refactor * rmv prevWM check --------- Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com>
* add repair in multi op * rmv check status from chunk upload and delete * rmv isRepair flag from worker * rmv fileID from fileMetaHash * add mut for repair callback * repair changes * delete file after upload in repair * add delete mask * add with mask option * fix uint tests * fix multi createDir * add path in filemetahash --------- Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com>
Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com>
* fix interface conversion * use sdk.multiUpload * Revert "use sdk.multiUpload" This reverts commit 428960e. * use multi upload
* add chunk feature * check fileshards * check chunkNum * check fileShards * add check in processUpload * check hash len --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com>
* resume download * clean up
* cleanup preferred blobbers from chain config * fix mobilesdk/CreateAllocation * fix lint
* Fix exclude path * Resolved comment
* display correct size values * fix consensus threshold * fix unit tests * fix allocation unit tests * request from all blobbers but read until consensus * fix file meta inconsistency * revert consensus threshold * fix get file meta --------- Co-authored-by: Yury <yuderbasov@gmail.com>
* do not send readmarkers for free reads * remove unused download worker count
* add multi op fix * fix lock connection mask * fix op mask * create connObj * add rollback in multi op * fix ctx in streamReader * use multiop ctx in reader * cleanup * fix repair in multi op * fix repair cb --------- Co-authored-by: Yury <yuderbasov@gmail.com>
Co-authored-by: Yury <yuderbasov@gmail.com>
* events information, do a consensus among 3 sharders * use a const * init from config.yaml * fix comment * fix UTs
|
Manual system tests [cancelled] with the following config
|
|
Manual system tests [failure] with the following config
|
Hitenjain14
requested review from
peterlimg,
dabasov,
guruhubb and
cnlangzi
as code owners
YarikRevich
pushed a commit
that referenced
this pull request
Jul 22, 2023
* feat wasm: readPool unlock (#1076) * readPool unlock * feat (mobile): expose ReadPoolUnLock * show chunk size in listReq response * Revert "show chunk size in listReq response" This reverts commit c62a183. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Refactor downloads to use sys.File (#1073) * added memfilechan * added memfilechan * clean up * refactor download methods to accept file handler * fix unit tests * refactor download methods, implement new downloader, fix unit tests * clean up * fix file callback --------- Co-authored-by: dabasov <yuderbasov@gmail.com> * Feature/expose multiop winsdk (#1034) * Expose multioperation in winsdk * Temporary remove callback args from multiupload * Export method * pass callback * test multiop * fix multiUpload * Revert "test multiop" This reverts commit 08ad565. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> --------- Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Add path in thumbnail hash (#1013) * add path in thumbnail hash * lint fix * fix typo * fix input s3_migration_cli_branch system_tests.yml * updated workflow input to s3_migration_cli_branch in system_tests.yml * fix hash calculation * check prevWM * refactor * rmv prevWM check --------- Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Refactor repair (#1004) * add repair in multi op * rmv check status from chunk upload and delete * rmv isRepair flag from worker * rmv fileID from fileMetaHash * add mut for repair callback * repair changes * delete file after upload in repair * add delete mask * add with mask option * fix uint tests * fix multi createDir * add path in filemetahash --------- Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Change the contract of Sharder and Miner update settings (#1085) Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix interface conversion error (#1088) * fix interface conversion * use sdk.multiUpload * Revert "use sdk.multiUpload" This reverts commit 428960e. * use multi upload * Add support for sending chunks (#1084) * add chunk feature * check fileshards * check chunkNum * check fileShards * add check in processUpload * check hash len --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix panic (#1090) * Resume download (#1096) * resume download * clean up * cleanup preferred blobbers from chain config (#1099) * cleanup preferred blobbers from chain config * fix mobilesdk/CreateAllocation * fix lint * Fix exclude path (#1093) * Fix exclude path * Resolved comment * Display correct size values (#1033) * display correct size values * fix consensus threshold * fix unit tests * fix allocation unit tests * request from all blobbers but read until consensus * fix file meta inconsistency * revert consensus threshold * fix get file meta --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Do not send readmarkers for free reads (#1101) * do not send readmarkers for free reads * remove unused download worker count * rmv streamreader * optimize repair * fix lint and test * rmv readCtx * fix upload check * add multi op fix (#1077) * add multi op fix * fix lock connection mask * fix op mask * create connObj * add rollback in multi op * fix ctx in streamReader * use multiop ctx in reader * cleanup * fix repair in multi op * fix repair cb --------- Co-authored-by: Yury <yuderbasov@gmail.com> * fix encrypt in multi op (#1097) Co-authored-by: Yury <yuderbasov@gmail.com> * take sharder consensous from wallet config (#1104) * events information, do a consensus among 3 sharders * use a const * init from config.yaml * fix comment * fix UTs * hotfix: fix `upload` and `download` in wasm (#1121) * Fix wasm download (#1117) * fix wasm download * fix syntax * fix reader * reverting reader changes --------- Co-authored-by: Dinmukhammed <52813950+din-mukhammed@users.noreply.github.com> Co-authored-by: din-mukhammed <dimash.r53@gmail.com> * add wasm download changes * fix merge * fix typo * fix repair cb * fix unit test * rmv log * check if env is wasm * cleanup * defer lock --------- Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Dinmukhammed <52813950+din-mukhammed@users.noreply.github.com> Co-authored-by: Roshan Kumar <56060325+Roshan-Mehta@users.noreply.github.com> Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Sunil Kumar <sunilkr.kumar26@gmail.com> Co-authored-by: Jayash Satolia <73050737+Jayashsatolia403@users.noreply.github.com> Co-authored-by: stewartie4 <ryanstewart456@gmail.com> Co-authored-by: din-mukhammed <dimash.r53@gmail.com>
cnlangzi
added a commit
that referenced
this pull request
Jul 28, 2023
* feat wasm: readPool unlock (#1076) * readPool unlock * feat (mobile): expose ReadPoolUnLock * show chunk size in listReq response * Revert "show chunk size in listReq response" This reverts commit c62a183. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Refactor downloads to use sys.File (#1073) * added memfilechan * added memfilechan * clean up * refactor download methods to accept file handler * fix unit tests * refactor download methods, implement new downloader, fix unit tests * clean up * fix file callback --------- Co-authored-by: dabasov <yuderbasov@gmail.com> * Feature/expose multiop winsdk (#1034) * Expose multioperation in winsdk * Temporary remove callback args from multiupload * Export method * pass callback * test multiop * fix multiUpload * Revert "test multiop" This reverts commit 08ad565. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> --------- Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Add path in thumbnail hash (#1013) * add path in thumbnail hash * lint fix * fix typo * fix input s3_migration_cli_branch system_tests.yml * updated workflow input to s3_migration_cli_branch in system_tests.yml * fix hash calculation * check prevWM * refactor * rmv prevWM check --------- Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Refactor repair (#1004) * add repair in multi op * rmv check status from chunk upload and delete * rmv isRepair flag from worker * rmv fileID from fileMetaHash * add mut for repair callback * repair changes * delete file after upload in repair * add delete mask * add with mask option * fix uint tests * fix multi createDir * add path in filemetahash --------- Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Change the contract of Sharder and Miner update settings (#1085) Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix interface conversion error (#1088) * fix interface conversion * use sdk.multiUpload * Revert "use sdk.multiUpload" This reverts commit 428960e. * use multi upload * Add support for sending chunks (#1084) * add chunk feature * check fileshards * check chunkNum * check fileShards * add check in processUpload * check hash len --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix panic (#1090) * Resume download (#1096) * resume download * clean up * cleanup preferred blobbers from chain config (#1099) * cleanup preferred blobbers from chain config * fix mobilesdk/CreateAllocation * fix lint * Fix exclude path (#1093) * Fix exclude path * Resolved comment * Display correct size values (#1033) * display correct size values * fix consensus threshold * fix unit tests * fix allocation unit tests * request from all blobbers but read until consensus * fix file meta inconsistency * revert consensus threshold * fix get file meta --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Do not send readmarkers for free reads (#1101) * do not send readmarkers for free reads * remove unused download worker count * rmv streamreader * optimize repair * fix lint and test * rmv readCtx * fix upload check * add multi op fix (#1077) * add multi op fix * fix lock connection mask * fix op mask * create connObj * add rollback in multi op * fix ctx in streamReader * use multiop ctx in reader * cleanup * fix repair in multi op * fix repair cb --------- Co-authored-by: Yury <yuderbasov@gmail.com> * fix encrypt in multi op (#1097) Co-authored-by: Yury <yuderbasov@gmail.com> * take sharder consensous from wallet config (#1104) * events information, do a consensus among 3 sharders * use a const * init from config.yaml * fix comment * fix UTs * hotfix: fix `upload` and `download` in wasm (#1121) * Fix wasm download (#1117) * fix wasm download * fix syntax * fix reader * reverting reader changes --------- Co-authored-by: Dinmukhammed <52813950+din-mukhammed@users.noreply.github.com> Co-authored-by: din-mukhammed <dimash.r53@gmail.com> * add wasm download changes * fix merge * fix typo * fix repair cb * fix unit test * rmv log * check if env is wasm * cleanup * defer lock --------- Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Dinmukhammed <52813950+din-mukhammed@users.noreply.github.com> Co-authored-by: Roshan Kumar <56060325+Roshan-Mehta@users.noreply.github.com> Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Sunil Kumar <sunilkr.kumar26@gmail.com> Co-authored-by: Jayash Satolia <73050737+Jayashsatolia403@users.noreply.github.com> Co-authored-by: stewartie4 <ryanstewart456@gmail.com> Co-authored-by: din-mukhammed <dimash.r53@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Fixes
Tests
Tasks to complete before merging PR:
Associated PRs (Link as appropriate):