New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/expose multiop winsdk #1034
Conversation
FYI: your methods are not exported yet. please check keyword |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
Manual system tests [success] with the following config
|
Manual system tests [failure] with the following config
|
Manual system tests [success] with the following config
|
Windows team is not able to use these changes. So will using multi Download and multiUpload through cli
* multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com>
…se-multiop-winsdk
Manual system tests [failure] with the following config
|
* sprint-july-2: Do not send readmarkers for free reads (#1101) Display correct size values (#1033) Fix exclude path (#1093) cleanup preferred blobbers from chain config (#1099) Resume download (#1096) fix panic (#1090) Add support for sending chunks (#1084) fix interface conversion error (#1088) Change the contract of Sharder and Miner update settings (#1085) Refactor repair (#1004) Add path in thumbnail hash (#1013) Feature/expose multiop winsdk (#1034) Refactor downloads to use sys.File (#1073) Multi download operation (#1045) feat wasm: readPool unlock (#1076)
* feat wasm: readPool unlock (#1076) * readPool unlock * feat (mobile): expose ReadPoolUnLock * show chunk size in listReq response * Revert "show chunk size in listReq response" This reverts commit c62a183. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Refactor downloads to use sys.File (#1073) * added memfilechan * added memfilechan * clean up * refactor download methods to accept file handler * fix unit tests * refactor download methods, implement new downloader, fix unit tests * clean up * fix file callback --------- Co-authored-by: dabasov <yuderbasov@gmail.com> * Feature/expose multiop winsdk (#1034) * Expose multioperation in winsdk * Temporary remove callback args from multiupload * Export method * pass callback * test multiop * fix multiUpload * Revert "test multiop" This reverts commit 08ad565. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> --------- Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Add path in thumbnail hash (#1013) * add path in thumbnail hash * lint fix * fix typo * fix input s3_migration_cli_branch system_tests.yml * updated workflow input to s3_migration_cli_branch in system_tests.yml * fix hash calculation * check prevWM * refactor * rmv prevWM check --------- Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Refactor repair (#1004) * add repair in multi op * rmv check status from chunk upload and delete * rmv isRepair flag from worker * rmv fileID from fileMetaHash * add mut for repair callback * repair changes * delete file after upload in repair * add delete mask * add with mask option * fix uint tests * fix multi createDir * add path in filemetahash --------- Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Change the contract of Sharder and Miner update settings (#1085) Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix interface conversion error (#1088) * fix interface conversion * use sdk.multiUpload * Revert "use sdk.multiUpload" This reverts commit 428960e. * use multi upload * Add support for sending chunks (#1084) * add chunk feature * check fileshards * check chunkNum * check fileShards * add check in processUpload * check hash len --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix panic (#1090) * Resume download (#1096) * resume download * clean up * cleanup preferred blobbers from chain config (#1099) * cleanup preferred blobbers from chain config * fix mobilesdk/CreateAllocation * fix lint * Fix exclude path (#1093) * Fix exclude path * Resolved comment * Display correct size values (#1033) * display correct size values * fix consensus threshold * fix unit tests * fix allocation unit tests * request from all blobbers but read until consensus * fix file meta inconsistency * revert consensus threshold * fix get file meta --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Do not send readmarkers for free reads (#1101) * do not send readmarkers for free reads * remove unused download worker count * add multi op fix (#1077) * add multi op fix * fix lock connection mask * fix op mask * create connObj * add rollback in multi op * fix ctx in streamReader * use multiop ctx in reader * cleanup * fix repair in multi op * fix repair cb --------- Co-authored-by: Yury <yuderbasov@gmail.com> * fix encrypt in multi op (#1097) Co-authored-by: Yury <yuderbasov@gmail.com> * take sharder consensous from wallet config (#1104) * events information, do a consensus among 3 sharders * use a const * init from config.yaml * fix comment * fix UTs * Optimize repair (#1105) * rmv streamreader * optimize repair * fix lint and test * rmv readCtx * fix upload check * Fix (#1110) * Export balance get function (#1103) Co-authored-by: Yury <yuderbasov@gmail.com> * Add getUpdateAllocationMinLock to wasm (#1107) * Add getUpdateAllocationMinLock * Fix demon index.html * fixed lint --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Updating blobber and validator update setting models. (#1095) * Updating blobber and validatory update setting models. * Updating the Terms struct as well. * Updating the StakePoolSettings struct as well. * pass only updted fields in validator --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Accept absolute path when downloading (#1111) * accept abs path also * fix unit tests * check if localpath has ext * fix seek for resume * Updating the validator node which is being passed to 0chain in update settings call (#1114) * fixing json name for the validator id. * fixing the blockchain validator udpate settings to pass to 0chain * fixing the json name for validator * Revert "Optimize repair (#1105)" (#1116) This reverts commit d293d7d. --------- Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Dinmukhammed <52813950+din-mukhammed@users.noreply.github.com> Co-authored-by: Roshan Kumar <56060325+Roshan-Mehta@users.noreply.github.com> Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Sunil Kumar <sunilkr.kumar26@gmail.com> Co-authored-by: Jayash Satolia <73050737+Jayashsatolia403@users.noreply.github.com> Co-authored-by: stewartie4 <ryanstewart456@gmail.com> Co-authored-by: peterlimg <54137706+peterlimg@users.noreply.github.com>
* feat wasm: readPool unlock (#1076) * readPool unlock * feat (mobile): expose ReadPoolUnLock * show chunk size in listReq response * Revert "show chunk size in listReq response" This reverts commit c62a183. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Refactor downloads to use sys.File (#1073) * added memfilechan * added memfilechan * clean up * refactor download methods to accept file handler * fix unit tests * refactor download methods, implement new downloader, fix unit tests * clean up * fix file callback --------- Co-authored-by: dabasov <yuderbasov@gmail.com> * Feature/expose multiop winsdk (#1034) * Expose multioperation in winsdk * Temporary remove callback args from multiupload * Export method * pass callback * test multiop * fix multiUpload * Revert "test multiop" This reverts commit 08ad565. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> --------- Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Add path in thumbnail hash (#1013) * add path in thumbnail hash * lint fix * fix typo * fix input s3_migration_cli_branch system_tests.yml * updated workflow input to s3_migration_cli_branch in system_tests.yml * fix hash calculation * check prevWM * refactor * rmv prevWM check --------- Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Refactor repair (#1004) * add repair in multi op * rmv check status from chunk upload and delete * rmv isRepair flag from worker * rmv fileID from fileMetaHash * add mut for repair callback * repair changes * delete file after upload in repair * add delete mask * add with mask option * fix uint tests * fix multi createDir * add path in filemetahash --------- Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Change the contract of Sharder and Miner update settings (#1085) Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix interface conversion error (#1088) * fix interface conversion * use sdk.multiUpload * Revert "use sdk.multiUpload" This reverts commit 428960e. * use multi upload * Add support for sending chunks (#1084) * add chunk feature * check fileshards * check chunkNum * check fileShards * add check in processUpload * check hash len --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix panic (#1090) * Resume download (#1096) * resume download * clean up * cleanup preferred blobbers from chain config (#1099) * cleanup preferred blobbers from chain config * fix mobilesdk/CreateAllocation * fix lint * Fix exclude path (#1093) * Fix exclude path * Resolved comment * Display correct size values (#1033) * display correct size values * fix consensus threshold * fix unit tests * fix allocation unit tests * request from all blobbers but read until consensus * fix file meta inconsistency * revert consensus threshold * fix get file meta --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Do not send readmarkers for free reads (#1101) * do not send readmarkers for free reads * remove unused download worker count * add multi op fix (#1077) * add multi op fix * fix lock connection mask * fix op mask * create connObj * add rollback in multi op * fix ctx in streamReader * use multiop ctx in reader * cleanup * fix repair in multi op * fix repair cb --------- Co-authored-by: Yury <yuderbasov@gmail.com> * fix encrypt in multi op (#1097) Co-authored-by: Yury <yuderbasov@gmail.com> * take sharder consensous from wallet config (#1104) * events information, do a consensus among 3 sharders * use a const * init from config.yaml * fix comment * fix UTs * Optimize repair (#1105) * rmv streamreader * optimize repair * fix lint and test * rmv readCtx * fix upload check * Fix * Fix * Fix update allocation * Fix create allocation * Fix (#1110) * Export balance get function (#1103) Co-authored-by: Yury <yuderbasov@gmail.com> * Add getUpdateAllocationMinLock to wasm (#1107) * Add getUpdateAllocationMinLock * Fix demon index.html * fixed lint --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Updating blobber and validator update setting models. (#1095) * Updating blobber and validatory update setting models. * Updating the Terms struct as well. * Updating the StakePoolSettings struct as well. * pass only updted fields in validator --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Accept absolute path when downloading (#1111) * accept abs path also * fix unit tests * check if localpath has ext * fix seek for resume * Updating the validator node which is being passed to 0chain in update settings call (#1114) * fixing json name for the validator id. * fixing the blockchain validator udpate settings to pass to 0chain * fixing the json name for validator * Fix * Revert "Optimize repair (#1105)" (#1116) This reverts commit d293d7d. --------- Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Dinmukhammed <52813950+din-mukhammed@users.noreply.github.com> Co-authored-by: Roshan Kumar <56060325+Roshan-Mehta@users.noreply.github.com> Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Sunil Kumar <sunilkr.kumar26@gmail.com> Co-authored-by: stewartie4 <ryanstewart456@gmail.com> Co-authored-by: peterlimg <54137706+peterlimg@users.noreply.github.com>
* feat wasm: readPool unlock (#1076) * readPool unlock * feat (mobile): expose ReadPoolUnLock * show chunk size in listReq response * Revert "show chunk size in listReq response" This reverts commit c62a183. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Refactor downloads to use sys.File (#1073) * added memfilechan * added memfilechan * clean up * refactor download methods to accept file handler * fix unit tests * refactor download methods, implement new downloader, fix unit tests * clean up * fix file callback --------- Co-authored-by: dabasov <yuderbasov@gmail.com> * Feature/expose multiop winsdk (#1034) * Expose multioperation in winsdk * Temporary remove callback args from multiupload * Export method * pass callback * test multiop * fix multiUpload * Revert "test multiop" This reverts commit 08ad565. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> --------- Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Add path in thumbnail hash (#1013) * add path in thumbnail hash * lint fix * fix typo * fix input s3_migration_cli_branch system_tests.yml * updated workflow input to s3_migration_cli_branch in system_tests.yml * fix hash calculation * check prevWM * refactor * rmv prevWM check --------- Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Refactor repair (#1004) * add repair in multi op * rmv check status from chunk upload and delete * rmv isRepair flag from worker * rmv fileID from fileMetaHash * add mut for repair callback * repair changes * delete file after upload in repair * add delete mask * add with mask option * fix uint tests * fix multi createDir * add path in filemetahash --------- Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Change the contract of Sharder and Miner update settings (#1085) Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix interface conversion error (#1088) * fix interface conversion * use sdk.multiUpload * Revert "use sdk.multiUpload" This reverts commit 428960e. * use multi upload * Add support for sending chunks (#1084) * add chunk feature * check fileshards * check chunkNum * check fileShards * add check in processUpload * check hash len --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix panic (#1090) * Resume download (#1096) * resume download * clean up * cleanup preferred blobbers from chain config (#1099) * cleanup preferred blobbers from chain config * fix mobilesdk/CreateAllocation * fix lint * Fix exclude path (#1093) * Fix exclude path * Resolved comment * Display correct size values (#1033) * display correct size values * fix consensus threshold * fix unit tests * fix allocation unit tests * request from all blobbers but read until consensus * fix file meta inconsistency * revert consensus threshold * fix get file meta --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Do not send readmarkers for free reads (#1101) * do not send readmarkers for free reads * remove unused download worker count * rmv streamreader * optimize repair * fix lint and test * rmv readCtx * fix upload check * add multi op fix (#1077) * add multi op fix * fix lock connection mask * fix op mask * create connObj * add rollback in multi op * fix ctx in streamReader * use multiop ctx in reader * cleanup * fix repair in multi op * fix repair cb --------- Co-authored-by: Yury <yuderbasov@gmail.com> * fix encrypt in multi op (#1097) Co-authored-by: Yury <yuderbasov@gmail.com> * take sharder consensous from wallet config (#1104) * events information, do a consensus among 3 sharders * use a const * init from config.yaml * fix comment * fix UTs * hotfix: fix `upload` and `download` in wasm (#1121) * Fix wasm download (#1117) * fix wasm download * fix syntax * fix reader * reverting reader changes --------- Co-authored-by: Dinmukhammed <52813950+din-mukhammed@users.noreply.github.com> Co-authored-by: din-mukhammed <dimash.r53@gmail.com> * add wasm download changes * fix merge * fix typo * fix repair cb * fix unit test * rmv log * check if env is wasm * cleanup * defer lock --------- Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Dinmukhammed <52813950+din-mukhammed@users.noreply.github.com> Co-authored-by: Roshan Kumar <56060325+Roshan-Mehta@users.noreply.github.com> Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Sunil Kumar <sunilkr.kumar26@gmail.com> Co-authored-by: Jayash Satolia <73050737+Jayashsatolia403@users.noreply.github.com> Co-authored-by: stewartie4 <ryanstewart456@gmail.com> Co-authored-by: din-mukhammed <dimash.r53@gmail.com>
* feat wasm: readPool unlock (#1076) * readPool unlock * feat (mobile): expose ReadPoolUnLock * show chunk size in listReq response * Revert "show chunk size in listReq response" This reverts commit c62a183. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Refactor downloads to use sys.File (#1073) * added memfilechan * added memfilechan * clean up * refactor download methods to accept file handler * fix unit tests * refactor download methods, implement new downloader, fix unit tests * clean up * fix file callback --------- Co-authored-by: dabasov <yuderbasov@gmail.com> * Feature/expose multiop winsdk (#1034) * Expose multioperation in winsdk * Temporary remove callback args from multiupload * Export method * pass callback * test multiop * fix multiUpload * Revert "test multiop" This reverts commit 08ad565. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> --------- Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Add path in thumbnail hash (#1013) * add path in thumbnail hash * lint fix * fix typo * fix input s3_migration_cli_branch system_tests.yml * updated workflow input to s3_migration_cli_branch in system_tests.yml * fix hash calculation * check prevWM * refactor * rmv prevWM check --------- Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Refactor repair (#1004) * add repair in multi op * rmv check status from chunk upload and delete * rmv isRepair flag from worker * rmv fileID from fileMetaHash * add mut for repair callback * repair changes * delete file after upload in repair * add delete mask * add with mask option * fix uint tests * fix multi createDir * add path in filemetahash --------- Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Change the contract of Sharder and Miner update settings (#1085) Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix interface conversion error (#1088) * fix interface conversion * use sdk.multiUpload * Revert "use sdk.multiUpload" This reverts commit 428960e. * use multi upload * Add support for sending chunks (#1084) * add chunk feature * check fileshards * check chunkNum * check fileShards * add check in processUpload * check hash len --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix panic (#1090) * Resume download (#1096) * resume download * clean up * cleanup preferred blobbers from chain config (#1099) * cleanup preferred blobbers from chain config * fix mobilesdk/CreateAllocation * fix lint * Fix exclude path (#1093) * Fix exclude path * Resolved comment * Display correct size values (#1033) * display correct size values * fix consensus threshold * fix unit tests * fix allocation unit tests * request from all blobbers but read until consensus * fix file meta inconsistency * revert consensus threshold * fix get file meta --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Do not send readmarkers for free reads (#1101) * do not send readmarkers for free reads * remove unused download worker count * add multi op fix (#1077) * add multi op fix * fix lock connection mask * fix op mask * create connObj * add rollback in multi op * fix ctx in streamReader * use multiop ctx in reader * cleanup * fix repair in multi op * fix repair cb --------- Co-authored-by: Yury <yuderbasov@gmail.com> * fix encrypt in multi op (#1097) Co-authored-by: Yury <yuderbasov@gmail.com> * take sharder consensous from wallet config (#1104) * events information, do a consensus among 3 sharders * use a const * init from config.yaml * fix comment * fix UTs * Optimize repair (#1105) * rmv streamreader * optimize repair * fix lint and test * rmv readCtx * fix upload check * Fix * Fix * Fix update allocation * Fix create allocation * Fix (#1110) * Export balance get function (#1103) Co-authored-by: Yury <yuderbasov@gmail.com> * Add getUpdateAllocationMinLock to wasm (#1107) * Add getUpdateAllocationMinLock * Fix demon index.html * fixed lint --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Updating blobber and validator update setting models. (#1095) * Updating blobber and validatory update setting models. * Updating the Terms struct as well. * Updating the StakePoolSettings struct as well. * pass only updted fields in validator --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Accept absolute path when downloading (#1111) * accept abs path also * fix unit tests * check if localpath has ext * fix seek for resume * Updating the validator node which is being passed to 0chain in update settings call (#1114) * fixing json name for the validator id. * fixing the blockchain validator udpate settings to pass to 0chain * fixing the json name for validator * Fix * Revert "Optimize repair (#1105)" (#1116) This reverts commit d293d7d. --------- Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Dinmukhammed <52813950+din-mukhammed@users.noreply.github.com> Co-authored-by: Roshan Kumar <56060325+Roshan-Mehta@users.noreply.github.com> Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Sunil Kumar <sunilkr.kumar26@gmail.com> Co-authored-by: stewartie4 <ryanstewart456@gmail.com> Co-authored-by: peterlimg <54137706+peterlimg@users.noreply.github.com>
* feat wasm: readPool unlock (#1076) * readPool unlock * feat (mobile): expose ReadPoolUnLock * show chunk size in listReq response * Revert "show chunk size in listReq response" This reverts commit c62a183. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Refactor downloads to use sys.File (#1073) * added memfilechan * added memfilechan * clean up * refactor download methods to accept file handler * fix unit tests * refactor download methods, implement new downloader, fix unit tests * clean up * fix file callback --------- Co-authored-by: dabasov <yuderbasov@gmail.com> * Feature/expose multiop winsdk (#1034) * Expose multioperation in winsdk * Temporary remove callback args from multiupload * Export method * pass callback * test multiop * fix multiUpload * Revert "test multiop" This reverts commit 08ad565. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> --------- Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Add path in thumbnail hash (#1013) * add path in thumbnail hash * lint fix * fix typo * fix input s3_migration_cli_branch system_tests.yml * updated workflow input to s3_migration_cli_branch in system_tests.yml * fix hash calculation * check prevWM * refactor * rmv prevWM check --------- Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Refactor repair (#1004) * add repair in multi op * rmv check status from chunk upload and delete * rmv isRepair flag from worker * rmv fileID from fileMetaHash * add mut for repair callback * repair changes * delete file after upload in repair * add delete mask * add with mask option * fix uint tests * fix multi createDir * add path in filemetahash --------- Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Change the contract of Sharder and Miner update settings (#1085) Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix interface conversion error (#1088) * fix interface conversion * use sdk.multiUpload * Revert "use sdk.multiUpload" This reverts commit 428960e. * use multi upload * Add support for sending chunks (#1084) * add chunk feature * check fileshards * check chunkNum * check fileShards * add check in processUpload * check hash len --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix panic (#1090) * Resume download (#1096) * resume download * clean up * cleanup preferred blobbers from chain config (#1099) * cleanup preferred blobbers from chain config * fix mobilesdk/CreateAllocation * fix lint * Fix exclude path (#1093) * Fix exclude path * Resolved comment * Display correct size values (#1033) * display correct size values * fix consensus threshold * fix unit tests * fix allocation unit tests * request from all blobbers but read until consensus * fix file meta inconsistency * revert consensus threshold * fix get file meta --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Do not send readmarkers for free reads (#1101) * do not send readmarkers for free reads * remove unused download worker count * rmv streamreader * optimize repair * fix lint and test * rmv readCtx * fix upload check * add multi op fix (#1077) * add multi op fix * fix lock connection mask * fix op mask * create connObj * add rollback in multi op * fix ctx in streamReader * use multiop ctx in reader * cleanup * fix repair in multi op * fix repair cb --------- Co-authored-by: Yury <yuderbasov@gmail.com> * fix encrypt in multi op (#1097) Co-authored-by: Yury <yuderbasov@gmail.com> * take sharder consensous from wallet config (#1104) * events information, do a consensus among 3 sharders * use a const * init from config.yaml * fix comment * fix UTs * hotfix: fix `upload` and `download` in wasm (#1121) * Fix wasm download (#1117) * fix wasm download * fix syntax * fix reader * reverting reader changes --------- Co-authored-by: Dinmukhammed <52813950+din-mukhammed@users.noreply.github.com> Co-authored-by: din-mukhammed <dimash.r53@gmail.com> * add wasm download changes * fix merge * fix typo * fix repair cb * fix unit test * rmv log * check if env is wasm * cleanup * defer lock --------- Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Dinmukhammed <52813950+din-mukhammed@users.noreply.github.com> Co-authored-by: Roshan Kumar <56060325+Roshan-Mehta@users.noreply.github.com> Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Sunil Kumar <sunilkr.kumar26@gmail.com> Co-authored-by: Jayash Satolia <73050737+Jayashsatolia403@users.noreply.github.com> Co-authored-by: stewartie4 <ryanstewart456@gmail.com> Co-authored-by: din-mukhammed <dimash.r53@gmail.com>
* feat wasm: readPool unlock (#1076) * readPool unlock * feat (mobile): expose ReadPoolUnLock * show chunk size in listReq response * Revert "show chunk size in listReq response" This reverts commit c62a183. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Refactor downloads to use sys.File (#1073) * added memfilechan * added memfilechan * clean up * refactor download methods to accept file handler * fix unit tests * refactor download methods, implement new downloader, fix unit tests * clean up * fix file callback --------- Co-authored-by: dabasov <yuderbasov@gmail.com> * Feature/expose multiop winsdk (#1034) * Expose multioperation in winsdk * Temporary remove callback args from multiupload * Export method * pass callback * test multiop * fix multiUpload * Revert "test multiop" This reverts commit 08ad565. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> --------- Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Add path in thumbnail hash (#1013) * add path in thumbnail hash * lint fix * fix typo * fix input s3_migration_cli_branch system_tests.yml * updated workflow input to s3_migration_cli_branch in system_tests.yml * fix hash calculation * check prevWM * refactor * rmv prevWM check --------- Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Refactor repair (#1004) * add repair in multi op * rmv check status from chunk upload and delete * rmv isRepair flag from worker * rmv fileID from fileMetaHash * add mut for repair callback * repair changes * delete file after upload in repair * add delete mask * add with mask option * fix uint tests * fix multi createDir * add path in filemetahash --------- Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Change the contract of Sharder and Miner update settings (#1085) Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix interface conversion error (#1088) * fix interface conversion * use sdk.multiUpload * Revert "use sdk.multiUpload" This reverts commit 428960e. * use multi upload * Add support for sending chunks (#1084) * add chunk feature * check fileshards * check chunkNum * check fileShards * add check in processUpload * check hash len --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix panic (#1090) * Resume download (#1096) * resume download * clean up * cleanup preferred blobbers from chain config (#1099) * cleanup preferred blobbers from chain config * fix mobilesdk/CreateAllocation * fix lint * Fix exclude path (#1093) * Fix exclude path * Resolved comment * Display correct size values (#1033) * display correct size values * fix consensus threshold * fix unit tests * fix allocation unit tests * request from all blobbers but read until consensus * fix file meta inconsistency * revert consensus threshold * fix get file meta --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Do not send readmarkers for free reads (#1101) * do not send readmarkers for free reads * remove unused download worker count * add multi op fix (#1077) * add multi op fix * fix lock connection mask * fix op mask * create connObj * add rollback in multi op * fix ctx in streamReader * use multiop ctx in reader * cleanup * fix repair in multi op * fix repair cb --------- Co-authored-by: Yury <yuderbasov@gmail.com> * fix encrypt in multi op (#1097) Co-authored-by: Yury <yuderbasov@gmail.com> * take sharder consensous from wallet config (#1104) * events information, do a consensus among 3 sharders * use a const * init from config.yaml * fix comment * fix UTs * Optimize repair (#1105) * rmv streamreader * optimize repair * fix lint and test * rmv readCtx * fix upload check * Fix * Fix * Fix update allocation * Fix create allocation * Fix (#1110) * Export balance get function (#1103) Co-authored-by: Yury <yuderbasov@gmail.com> * Add getUpdateAllocationMinLock to wasm (#1107) * Add getUpdateAllocationMinLock * Fix demon index.html * fixed lint --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Updating blobber and validator update setting models. (#1095) * Updating blobber and validatory update setting models. * Updating the Terms struct as well. * Updating the StakePoolSettings struct as well. * pass only updted fields in validator --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Accept absolute path when downloading (#1111) * accept abs path also * fix unit tests * check if localpath has ext * fix seek for resume * Updating the validator node which is being passed to 0chain in update settings call (#1114) * fixing json name for the validator id. * fixing the blockchain validator udpate settings to pass to 0chain * fixing the json name for validator * Fix * Revert "Optimize repair (#1105)" (#1116) This reverts commit d293d7d. --------- Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Dinmukhammed <52813950+din-mukhammed@users.noreply.github.com> Co-authored-by: Roshan Kumar <56060325+Roshan-Mehta@users.noreply.github.com> Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Sunil Kumar <sunilkr.kumar26@gmail.com> Co-authored-by: stewartie4 <ryanstewart456@gmail.com> Co-authored-by: peterlimg <54137706+peterlimg@users.noreply.github.com>
* feat wasm: readPool unlock (#1076) * readPool unlock * feat (mobile): expose ReadPoolUnLock * show chunk size in listReq response * Revert "show chunk size in listReq response" This reverts commit c62a183. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Refactor downloads to use sys.File (#1073) * added memfilechan * added memfilechan * clean up * refactor download methods to accept file handler * fix unit tests * refactor download methods, implement new downloader, fix unit tests * clean up * fix file callback --------- Co-authored-by: dabasov <yuderbasov@gmail.com> * Feature/expose multiop winsdk (#1034) * Expose multioperation in winsdk * Temporary remove callback args from multiupload * Export method * pass callback * test multiop * fix multiUpload * Revert "test multiop" This reverts commit 08ad565. * Multi download operation (#1045) * multi download operation in mobilesdk * add description * set thresh to data shards * multiupload func added to wasm * fix wasm build * create localPath * fix typo * Update proxy.go * Add to response * fix append * increase download workers * decrease number of download workers * set workers back to 20 * use multiUpload demo for multi operation clean up logs * feat wasm: UpdateForbidAllocation * add demo for multiDownload * cleanup * rmv worker semaphore * add objURL in callback * fix build * add description * add packages * add filename to callback * ad localPath to statusBar * add blobber lock * add filename to start and progress * export winsdk functions * Revert "export winsdk functions" This reverts commit eed6cd0. * add multiDownload to winsdk * fix typo * Move min lock demand to allocation (#1046) * min_lock_demand * mobile * Feat/caching get fees estimate (#1069) * added caching instead of the network call * clean up * clean up * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * fixed comments --------- Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * Rename blobber field to not_available (#1043) * remove unstake total * not_available * added GRAPHNODE_SC * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * run build on all PRs * run on every PR --------- Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * wasm recover from panic (#1071) * fixes for update image (#1058) * update wasm demo for download * preserver order in blobber of newallocationrequestg * fixes for update image * use multiUpload * demo for multi operation * clean up logs * init Consensus * feat wasm: UpdateForbidAllocation * expose lockReadPool * fix for unique blobbers * init Consensus in Lock function (#1063) * init Consensus * fix panic in dirworker --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * recover from panic wasm --------- Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> * cleanup * set consensus to threshold for authTicket --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> --------- Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * Add path in thumbnail hash (#1013) * add path in thumbnail hash * lint fix * fix typo * fix input s3_migration_cli_branch system_tests.yml * updated workflow input to s3_migration_cli_branch in system_tests.yml * fix hash calculation * check prevWM * refactor * rmv prevWM check --------- Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Refactor repair (#1004) * add repair in multi op * rmv check status from chunk upload and delete * rmv isRepair flag from worker * rmv fileID from fileMetaHash * add mut for repair callback * repair changes * delete file after upload in repair * add delete mask * add with mask option * fix uint tests * fix multi createDir * add path in filemetahash --------- Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Yury <yuderbasov@gmail.com> * Change the contract of Sharder and Miner update settings (#1085) Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix interface conversion error (#1088) * fix interface conversion * use sdk.multiUpload * Revert "use sdk.multiUpload" This reverts commit 428960e. * use multi upload * Add support for sending chunks (#1084) * add chunk feature * check fileshards * check chunkNum * check fileShards * add check in processUpload * check hash len --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> * fix panic (#1090) * Resume download (#1096) * resume download * clean up * cleanup preferred blobbers from chain config (#1099) * cleanup preferred blobbers from chain config * fix mobilesdk/CreateAllocation * fix lint * Fix exclude path (#1093) * Fix exclude path * Resolved comment * Display correct size values (#1033) * display correct size values * fix consensus threshold * fix unit tests * fix allocation unit tests * request from all blobbers but read until consensus * fix file meta inconsistency * revert consensus threshold * fix get file meta --------- Co-authored-by: Yury <yuderbasov@gmail.com> * Do not send readmarkers for free reads (#1101) * do not send readmarkers for free reads * remove unused download worker count * rmv streamreader * optimize repair * fix lint and test * rmv readCtx * fix upload check * add multi op fix (#1077) * add multi op fix * fix lock connection mask * fix op mask * create connObj * add rollback in multi op * fix ctx in streamReader * use multiop ctx in reader * cleanup * fix repair in multi op * fix repair cb --------- Co-authored-by: Yury <yuderbasov@gmail.com> * fix encrypt in multi op (#1097) Co-authored-by: Yury <yuderbasov@gmail.com> * take sharder consensous from wallet config (#1104) * events information, do a consensus among 3 sharders * use a const * init from config.yaml * fix comment * fix UTs * hotfix: fix `upload` and `download` in wasm (#1121) * Fix wasm download (#1117) * fix wasm download * fix syntax * fix reader * reverting reader changes --------- Co-authored-by: Dinmukhammed <52813950+din-mukhammed@users.noreply.github.com> Co-authored-by: din-mukhammed <dimash.r53@gmail.com> * add wasm download changes * fix merge * fix typo * fix repair cb * fix unit test * rmv log * check if env is wasm * cleanup * defer lock --------- Co-authored-by: boddumanohar <b.manu199@gmail.com> Co-authored-by: Piers Shepperson <piers.shepperson@btinternet.com> Co-authored-by: Yury <yuderbasov@gmail.com> Co-authored-by: sarvesh raut <sarveshraut@sarveshs-MacBook-Air.local> Co-authored-by: Manali-Jain-Squareops <manali@squareops.com> Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com> Co-authored-by: Dinmukhammed <52813950+din-mukhammed@users.noreply.github.com> Co-authored-by: Roshan Kumar <56060325+Roshan-Mehta@users.noreply.github.com> Co-authored-by: Lz <imlangzi@qq.com> Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com> Co-authored-by: Sunil Kumar <sunilkr.kumar26@gmail.com> Co-authored-by: Jayash Satolia <73050737+Jayashsatolia403@users.noreply.github.com> Co-authored-by: stewartie4 <ryanstewart456@gmail.com> Co-authored-by: din-mukhammed <dimash.r53@gmail.com>
Changes
Fixes
Tests
Tasks to complete before merging PR:
Associated PRs (Link as appropriate):