WIP: Parallel Decomp - Global pool in separate class and... #3556

HereThereBeDragons · 2024-03-27T08:48:23Z

CVMFS params to adjust memory footprint

TODO:

Adjust Clone() for external downloadmanager. right now a Clone() does nothing and will disable this feature.
Add integration test. Right now nothing tests this feature --> unittest

HereThereBeDragons · 2024-03-27T08:49:01Z

@cernvm-bot cloudtest

cernvm-bot · 2024-03-27T09:09:34Z

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/4343/

cernvm-bot · 2024-03-27T10:21:55Z

cloudtests finished: FAILURE
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/2090/

cernvm-bot · 2024-03-27T17:12:58Z

linter finished with errors:

Done processing mount/mount.cvmftest/unittests/t_download.cc:571:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]
test/unittests/t_download.cc:609:  Missing space after ,  [whitespace/comma] [3]
test/unittests/t_download.cc:619:  Missing space after ,  [whitespace/comma] [3]

HereThereBeDragons · 2024-03-28T15:00:23Z

@cernvm-bot cloudtest

cernvm-bot · 2024-03-28T15:21:32Z

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/4344/

cernvm-bot · 2024-03-28T16:32:54Z

cloudtests finished: FAILURE
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/2091/

HereThereBeDragons · 2024-03-28T16:35:28Z

cloudtest looks good. the 2 commits since then did not touch any code logic --> no new cloudtest need

HereThereBeDragons · 2024-04-04T09:37:59Z

@cernvm-bot cloudtest

cernvm-bot · 2024-04-04T09:38:09Z

linter finished with errors:

Done processingcvmfs/network/download.cc:1445:  Lines should be <= 80 characters long  [whitespace/line_length] [2]
cvmfs/network/download.h:266:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]

cernvm-bot · 2024-04-04T09:59:33Z

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/4348/

cernvm-bot · 2024-04-04T11:07:34Z

cloudtests finished: FAILURE
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/2093/

HereThereBeDragons · 2024-04-04T12:18:07Z

@cernvm-bot cloudtest

cernvm-bot · 2024-04-04T12:40:29Z

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/4349/

cernvm-bot · 2024-04-04T13:54:15Z

cloudtests finished: FAILURE
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/2094/

HereThereBeDragons · 2024-04-04T15:31:50Z

@cernvm-bot cloudtest

cernvm-bot · 2024-04-04T15:54:15Z

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/4350/

cernvm-bot · 2024-04-04T17:07:30Z

cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/2095/

rezan · 2024-04-04T18:11:35Z

cvmfs/network/download.cc

+            } else {
+              const int64_t written = info->sink()->Write(ptr, num_bytes);
+              if (written < 0 || static_cast<uint64_t>(written) != num_bytes) {
+                PANIC(kLogStderr | kLogDebug, "(id %" PRId64 ") "


In the CallbackCurlData() this is a plain LogCvmfs() statement, but here its a PANIC(). Is there a reason why? If possible, it might make sense to refactor these section into a single helper if they are identical. Also, if we cant write to the sync, we should stop the download, right?

thanks for catching this!
yes also the download should be stopped

cernvm-bot · 2024-04-05T13:32:45Z

linter finished with errors:

Done processingcvmfs/network/download.cc:2143:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]
cvmfs/network/download.h:268:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]

HereThereBeDragons · 2024-04-05T13:32:48Z

@cernvm-bot cloudtest

cernvm-bot · 2024-04-05T13:53:29Z

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/4351/

cernvm-bot · 2024-04-05T15:05:27Z

cloudtests finished: FAILURE
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/2096/

HereThereBeDragons · 2024-04-12T15:30:28Z

last thing what is missing is the rebase onto devel

…s for the callbackcurldata for tube element data size: use CURL_MAX_WRITE_SIZE Use GetUnusedDataTubeElement for all cases instead of malloc

…ontrollable by cvmfs params add asserts and class documentation

… for parallel decomp

HereThereBeDragons · 2024-04-15T11:53:51Z

@cernvm-bot cloudtest

cernvm-bot · 2024-04-15T12:12:50Z

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/4374/

cernvm-bot · 2024-04-15T13:25:24Z

cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/2112/

HereThereBeDragons · 2024-04-15T14:12:47Z

this pr is now ready to review
(needs some more performance testing)

it includes the fix for #3563

HereThereBeDragons · 2024-04-18T09:11:38Z

Fixes #3493

HereThereBeDragons marked this pull request as draft March 27, 2024 13:51

HereThereBeDragons marked this pull request as ready for review March 28, 2024 16:03

HereThereBeDragons requested a review from jblomer March 28, 2024 16:13

rezan reviewed Apr 4, 2024

View reviewed changes

HereThereBeDragons added 7 commits April 15, 2024 11:07

add parallel decompression support

28fca19

move all data processing from CallbackCurlData

59d4ae2

have a global queue in downloadmanager to reuse/prealloc tube element…

e0adb2d

…s for the callbackcurldata for tube element data size: use CURL_MAX_WRITE_SIZE Use GetUnusedDataTubeElement for all cases instead of malloc

WIP global pool but encapsulated in its own class; memory footprint c…

7b304d4

…ontrollable by cvmfs params add asserts and class documentation

try to optimize sequential download

c353d82

Split VerifyAndFinalize into multiple funcs and use the split version…

24b2a4f

… for parallel decomp

small fixes + add correct logoutput to parallel download code sections

0e2c64c

HereThereBeDragons force-pushed the dwnld_mgr_parallel_decomp_3493_v2_static_global_pool branch from 28ff63b to 0e2c64c Compare April 15, 2024 11:37

fix typos

092bcf1

HereThereBeDragons requested review from rezan and vvolkl and removed request for rezan April 15, 2024 14:08

HereThereBeDragons mentioned this pull request Apr 15, 2024

Download error can cause stalling and error looping #3563

Open

HereThereBeDragons mentioned this pull request Apr 18, 2024

Remove bottleneck: Allow decompression of download requests in parallel #3493

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Parallel Decomp - Global pool in separate class and... #3556

WIP: Parallel Decomp - Global pool in separate class and... #3556

HereThereBeDragons commented Mar 27, 2024 •

edited

HereThereBeDragons commented Mar 27, 2024

cernvm-bot commented Mar 27, 2024

cernvm-bot commented Mar 27, 2024

cernvm-bot commented Mar 27, 2024

HereThereBeDragons commented Mar 28, 2024

cernvm-bot commented Mar 28, 2024

cernvm-bot commented Mar 28, 2024

HereThereBeDragons commented Mar 28, 2024

HereThereBeDragons commented Apr 4, 2024

cernvm-bot commented Apr 4, 2024

cernvm-bot commented Apr 4, 2024

cernvm-bot commented Apr 4, 2024

HereThereBeDragons commented Apr 4, 2024

cernvm-bot commented Apr 4, 2024

cernvm-bot commented Apr 4, 2024

HereThereBeDragons commented Apr 4, 2024

cernvm-bot commented Apr 4, 2024

cernvm-bot commented Apr 4, 2024

rezan Apr 4, 2024

HereThereBeDragons Apr 5, 2024

cernvm-bot commented Apr 5, 2024

HereThereBeDragons commented Apr 5, 2024

cernvm-bot commented Apr 5, 2024

cernvm-bot commented Apr 5, 2024

HereThereBeDragons commented Apr 12, 2024

HereThereBeDragons commented Apr 15, 2024

cernvm-bot commented Apr 15, 2024

cernvm-bot commented Apr 15, 2024

HereThereBeDragons commented Apr 15, 2024

HereThereBeDragons commented Apr 18, 2024

WIP: Parallel Decomp - Global pool in separate class and... #3556

Are you sure you want to change the base?

WIP: Parallel Decomp - Global pool in separate class and... #3556

Conversation

HereThereBeDragons commented Mar 27, 2024 • edited

HereThereBeDragons commented Mar 27, 2024

cernvm-bot commented Mar 27, 2024

cernvm-bot commented Mar 27, 2024

cernvm-bot commented Mar 27, 2024

HereThereBeDragons commented Mar 28, 2024

cernvm-bot commented Mar 28, 2024

cernvm-bot commented Mar 28, 2024

HereThereBeDragons commented Mar 28, 2024

HereThereBeDragons commented Apr 4, 2024

cernvm-bot commented Apr 4, 2024

cernvm-bot commented Apr 4, 2024

cernvm-bot commented Apr 4, 2024

HereThereBeDragons commented Apr 4, 2024

cernvm-bot commented Apr 4, 2024

cernvm-bot commented Apr 4, 2024

HereThereBeDragons commented Apr 4, 2024

cernvm-bot commented Apr 4, 2024

cernvm-bot commented Apr 4, 2024

rezan Apr 4, 2024

Choose a reason for hiding this comment

HereThereBeDragons Apr 5, 2024

Choose a reason for hiding this comment

cernvm-bot commented Apr 5, 2024

HereThereBeDragons commented Apr 5, 2024

cernvm-bot commented Apr 5, 2024

cernvm-bot commented Apr 5, 2024

HereThereBeDragons commented Apr 12, 2024

HereThereBeDragons commented Apr 15, 2024

cernvm-bot commented Apr 15, 2024

cernvm-bot commented Apr 15, 2024

HereThereBeDragons commented Apr 15, 2024

HereThereBeDragons commented Apr 18, 2024

HereThereBeDragons commented Mar 27, 2024 •

edited