Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

upload! has a memory and cpu hunger ? #18

Open
tillvollmer opened this Issue · 1 comment

1 participant

@tillvollmer

Hi,

I am using this code:
Net::SFTP.start(backup.hostname, backup.username, :password => password, :port => backup.port) do |sftp|
sftp.upload!(filename, backup.path + "/" + File.basename(filename))
end

to upload a file to an external server.

The file is 33 MB in size

The uploading uses 32 GB !!!! and often 100% CPU time (it finished after a while, but ruins my server)

Any idea why ?

net-sftp (2.0.5)
net-ssh (2.6.1)

Linux Debian 64 Bit

I added some debugging in the code

sftp.upload!(filename, backup.path + "/" + File.basename(filename)) do |event, uploader, *args|

      case event
        when :open then
          # args[0] : file metadata
          logger.error "starting upload: #{args[0].local} -> #{args[0].remote} (#{args[0].size} bytes}"
          logger.error("memory= #{MemoryBenchmark.memory}")
        when :put then
          # args[0] : file metadata
          # args[1] : byte offset in remote file
          # args[2] : data being written (as string)
          logger.error "writing #{args[2].length} bytes to #{args[0].remote} starting at #{args[1]}"
          logger.error("memory= #{MemoryBenchmark.memory}")
        when :close then
          # args[0] : file metadata
          logger.error "finished with #{args[0].remote}"
          logger.error("memory= #{MemoryBenchmark.memory}")
        when :mkdir then
          # args[0] : remote path name
          logger.error "creating directory #{args[0]}"
          logger.error("memory= #{MemoryBenchmark.memory}")
        when :finish then
          logger.error "all done!"
          logger.error("memory= #{MemoryBenchmark.memory}")
      end

    end

and MemoryBenchmark does this:

def self.memory
return ps -o rss= -p #{Process.pid}.to_i
end

which results in this:
also the upload is getting slower and slower

The memory is only the RSS the virtual is nearly twice as high and I stopped on 17MB upload
So nearly 3GB
And its exponential growing!

memory= 240928
writing 32000 bytes to /test.zip starting at 0
memory= 240932
writing 32000 bytes to /test.zip starting at 32000
memory= 240932
writing 32000 bytes to /test.zip starting at 64000
memory= 240932
writing 32000 bytes to /test.zip starting at 96000
memory= 240932
writing 32000 bytes to /test.zip starting at 128000
memory= 240932
writing 32000 bytes to /test.zip starting at 160000
memory= 240932
writing 32000 bytes to /test.zip starting at 192000
memory= 240932
writing 32000 bytes to /test.zip starting at 224000
memory= 240932
writing 32000 bytes to /test.zip starting at 256000
memory= 240932
writing 32000 bytes to /test.zip starting at 288000
memory= 240932
writing 32000 bytes to /test.zip starting at 320000
memory= 240932
writing 32000 bytes to /test.zip starting at 352000
memory= 240932
writing 32000 bytes to /test.zip starting at 384000
memory= 240932
writing 32000 bytes to /test.zip starting at 416000
memory= 240932
writing 32000 bytes to /test.zip starting at 448000
memory= 240932
writing 32000 bytes to /test.zip starting at 480000
memory= 240932
writing 32000 bytes to /test.zip starting at 512000
memory= 240932
writing 32000 bytes to /test.zip starting at 544000
memory= 240932
writing 32000 bytes to /test.zip starting at 576000
memory= 240932
writing 32000 bytes to /test.zip starting at 608000
memory= 240932
writing 32000 bytes to /test.zip starting at 640000
memory= 240932
writing 32000 bytes to /test.zip starting at 672000
memory= 240932
writing 32000 bytes to /test.zip starting at 704000
memory= 240932
writing 32000 bytes to /test.zip starting at 736000
memory= 240932
writing 32000 bytes to /test.zip starting at 768000
memory= 240932
writing 32000 bytes to /test.zip starting at 800000
memory= 240932
writing 32000 bytes to /test.zip starting at 832000
memory= 240932
writing 32000 bytes to /test.zip starting at 864000
memory= 240932
writing 32000 bytes to /test.zip starting at 896000
memory= 240932
writing 32000 bytes to /test.zip starting at 928000
memory= 240932
writing 32000 bytes to /test.zip starting at 960000
memory= 240932
writing 32000 bytes to /test.zip starting at 992000
memory= 240932
writing 32000 bytes to /test.zip starting at 1024000
memory= 240932
writing 32000 bytes to /test.zip starting at 1056000
memory= 240932
writing 32000 bytes to /test.zip starting at 1088000
memory= 240932
writing 32000 bytes to /test.zip starting at 1120000
memory= 240932
writing 32000 bytes to /test.zip starting at 1152000
memory= 240932
writing 32000 bytes to /test.zip starting at 1184000
memory= 240932
writing 32000 bytes to /test.zip starting at 1216000
memory= 240932
writing 32000 bytes to /test.zip starting at 1248000
memory= 240932
writing 32000 bytes to /test.zip starting at 1280000
memory= 240932
writing 32000 bytes to /test.zip starting at 1312000
memory= 240932
writing 32000 bytes to /test.zip starting at 1344000
memory= 240932
writing 32000 bytes to /test.zip starting at 1376000
memory= 240932
writing 32000 bytes to /test.zip starting at 1408000
memory= 240956
writing 32000 bytes to /test.zip starting at 1440000
memory= 240956
writing 32000 bytes to /test.zip starting at 1472000
memory= 240956
writing 32000 bytes to /test.zip starting at 1504000
memory= 240956
writing 32000 bytes to /test.zip starting at 1536000
memory= 240956
writing 32000 bytes to /test.zip starting at 1568000
memory= 240956
writing 32000 bytes to /test.zip starting at 1600000
memory= 240956
writing 32000 bytes to /test.zip starting at 1632000
memory= 240956
writing 32000 bytes to /test.zip starting at 1664000
memory= 240956
writing 32000 bytes to /test.zip starting at 1696000
memory= 240956
writing 32000 bytes to /test.zip starting at 1728000
memory= 240956
writing 32000 bytes to /test.zip starting at 1760000
memory= 240956
writing 32000 bytes to /test.zip starting at 1792000
memory= 240956
writing 32000 bytes to /test.zip starting at 1824000
memory= 240956
writing 32000 bytes to /test.zip starting at 1856000
memory= 240956
writing 32000 bytes to /test.zip starting at 1888000
memory= 240956
writing 32000 bytes to /test.zip starting at 1920000
memory= 240956
writing 32000 bytes to /test.zip starting at 1952000
memory= 240956
writing 32000 bytes to /test.zip starting at 1984000
memory= 240956
writing 32000 bytes to /test.zip starting at 2016000
memory= 240956
writing 32000 bytes to /test.zip starting at 2048000
memory= 240956
writing 32000 bytes to /test.zip starting at 2080000
memory= 242964
writing 32000 bytes to /test.zip starting at 2112000
memory= 242964
writing 32000 bytes to /test.zip starting at 2144000
memory= 242964
writing 32000 bytes to /test.zip starting at 2176000
memory= 242964
writing 32000 bytes to /test.zip starting at 2208000
memory= 243152
writing 32000 bytes to /test.zip starting at 2240000
memory= 245316
writing 32000 bytes to /test.zip starting at 2272000
memory= 245316
writing 32000 bytes to /test.zip starting at 2304000
memory= 245316
writing 32000 bytes to /test.zip starting at 2336000
memory= 245452
writing 32000 bytes to /test.zip starting at 2368000
memory= 245452
writing 32000 bytes to /test.zip starting at 2400000
memory= 245452
writing 32000 bytes to /test.zip starting at 2432000
memory= 245628
writing 32000 bytes to /test.zip starting at 2464000
memory= 245748
writing 32000 bytes to /test.zip starting at 2496000
memory= 245748
writing 32000 bytes to /test.zip starting at 2528000
memory= 248192
writing 32000 bytes to /test.zip starting at 2560000
memory= 248192
writing 32000 bytes to /test.zip starting at 2592000
memory= 248192
writing 32000 bytes to /test.zip starting at 2624000
memory= 248536
writing 32000 bytes to /test.zip starting at 2656000
memory= 248536
writing 32000 bytes to /test.zip starting at 2688000
memory= 248536
writing 32000 bytes to /test.zip starting at 2720000
memory= 248536
writing 32000 bytes to /test.zip starting at 2752000
memory= 248536
writing 32000 bytes to /test.zip starting at 2784000
memory= 248536
writing 32000 bytes to /test.zip starting at 2816000
memory= 248536
writing 32000 bytes to /test.zip starting at 2848000
memory= 248536
writing 32000 bytes to /test.zip starting at 2880000
memory= 248536
writing 32000 bytes to /test.zip starting at 2912000
memory= 248536
writing 32000 bytes to /test.zip starting at 2944000
memory= 248536
writing 32000 bytes to /test.zip starting at 2976000
memory= 248536
writing 32000 bytes to /test.zip starting at 3008000
memory= 248536
writing 32000 bytes to /test.zip starting at 3040000
memory= 248536
writing 32000 bytes to /test.zip starting at 3072000
memory= 248536
writing 32000 bytes to /test.zip starting at 3104000
memory= 248536
writing 32000 bytes to /test.zip starting at 3136000
memory= 248536
writing 32000 bytes to /test.zip starting at 3168000
memory= 248536
writing 32000 bytes to /test.zip starting at 3200000
memory= 248536
writing 32000 bytes to /test.zip starting at 3232000
memory= 249256
writing 32000 bytes to /test.zip starting at 3264000
memory= 249288
writing 32000 bytes to /test.zip starting at 3296000
memory= 249288
writing 32000 bytes to /test.zip starting at 3328000
memory= 249348
writing 32000 bytes to /test.zip starting at 3360000
memory= 249368
writing 32000 bytes to /test.zip starting at 3392000
memory= 249368
writing 32000 bytes to /test.zip starting at 3424000
memory= 249444
writing 32000 bytes to /test.zip starting at 3456000
memory= 249476
writing 32000 bytes to /test.zip starting at 3488000
memory= 249476
writing 32000 bytes to /test.zip starting at 3520000
memory= 249528
writing 32000 bytes to /test.zip starting at 3552000
memory= 249568
writing 32000 bytes to /test.zip starting at 3584000
memory= 249600
writing 32000 bytes to /test.zip starting at 3616000
memory= 249600
writing 32000 bytes to /test.zip starting at 3648000
memory= 249664
writing 32000 bytes to /test.zip starting at 3680000
memory= 249696
writing 32000 bytes to /test.zip starting at 3712000
memory= 249696
writing 32000 bytes to /test.zip starting at 3744000
memory= 249756
writing 32000 bytes to /test.zip starting at 3776000
memory= 249760
writing 32000 bytes to /test.zip starting at 3808000
memory= 249760
writing 32000 bytes to /test.zip starting at 3840000
memory= 249852
writing 32000 bytes to /test.zip starting at 3872000
memory= 249884
writing 32000 bytes to /test.zip starting at 3904000
memory= 249884
writing 32000 bytes to /test.zip starting at 3936000
memory= 249920
writing 32000 bytes to /test.zip starting at 3968000
memory= 249976
writing 32000 bytes to /test.zip starting at 4000000
memory= 250008
writing 32000 bytes to /test.zip starting at 4032000
memory= 250008
writing 32000 bytes to /test.zip starting at 4064000
memory= 250068
writing 32000 bytes to /test.zip starting at 4096000
memory= 250080
writing 32000 bytes to /test.zip starting at 4128000
memory= 250080
writing 32000 bytes to /test.zip starting at 4160000
memory= 250164
writing 32000 bytes to /test.zip starting at 4192000
memory= 250196
writing 32000 bytes to /test.zip starting at 4224000
memory= 250196
writing 32000 bytes to /test.zip starting at 4256000
memory= 250240
writing 32000 bytes to /test.zip starting at 4288000
memory= 250288
writing 32000 bytes to /test.zip starting at 4320000
memory= 250320
writing 32000 bytes to /test.zip starting at 4352000
memory= 250320
writing 32000 bytes to /test.zip starting at 4384000
memory= 250384
writing 32000 bytes to /test.zip starting at 4416000
memory= 250400
writing 32000 bytes to /test.zip starting at 4448000
memory= 250400
writing 32000 bytes to /test.zip starting at 4480000
memory= 250464
writing 32000 bytes to /test.zip starting at 4512000
memory= 250464
writing 32000 bytes to /test.zip starting at 4544000
memory= 250464
writing 32000 bytes to /test.zip starting at 4576000
memory= 250464
writing 32000 bytes to /test.zip starting at 4608000
memory= 250464
writing 32000 bytes to /test.zip starting at 4640000
memory= 250464
writing 32000 bytes to /test.zip starting at 4672000
memory= 250464
writing 32000 bytes to /test.zip starting at 4704000
memory= 250464
writing 32000 bytes to /test.zip starting at 4736000
memory= 250464
writing 32000 bytes to /test.zip starting at 4768000
memory= 250464
writing 32000 bytes to /test.zip starting at 4800000
memory= 250784
writing 32000 bytes to /test.zip starting at 4832000
memory= 250788
writing 32000 bytes to /test.zip starting at 4864000
memory= 250788
writing 32000 bytes to /test.zip starting at 4896000
memory= 250852
writing 32000 bytes to /test.zip starting at 4928000
memory= 250912
writing 32000 bytes to /test.zip starting at 4960000
memory= 250916
writing 32000 bytes to /test.zip starting at 4992000
memory= 250948
writing 32000 bytes to /test.zip starting at 5024000
memory= 250948
writing 32000 bytes to /test.zip starting at 5056000
memory= 251008
writing 32000 bytes to /test.zip starting at 5088000
memory= 251040
writing 32000 bytes to /test.zip starting at 5120000
memory= 251040
writing 32000 bytes to /test.zip starting at 5152000
memory= 251104
writing 32000 bytes to /test.zip starting at 5184000
memory= 251136
writing 32000 bytes to /test.zip starting at 5216000
memory= 251164
writing 32000 bytes to /test.zip starting at 5248000
memory= 251164
writing 32000 bytes to /test.zip starting at 5280000
memory= 251292
writing 32000 bytes to /test.zip starting at 5312000
memory= 251328
writing 32000 bytes to /test.zip starting at 5344000
memory= 251328
writing 32000 bytes to /test.zip starting at 5376000
memory= 251484
writing 32000 bytes to /test.zip starting at 5408000
memory= 251540
writing 32000 bytes to /test.zip starting at 5440000
memory= 251604
writing 32000 bytes to /test.zip starting at 5472000
memory= 251604
writing 32000 bytes to /test.zip starting at 5504000
memory= 251732
writing 32000 bytes to /test.zip starting at 5536000
memory= 251792
writing 32000 bytes to /test.zip starting at 5568000
memory= 251824
writing 32000 bytes to /test.zip starting at 5600000
memory= 251824
writing 32000 bytes to /test.zip starting at 5632000
memory= 251984
writing 32000 bytes to /test.zip starting at 5664000
memory= 252044
writing 32000 bytes to /test.zip starting at 5696000
memory= 252044
writing 32000 bytes to /test.zip starting at 5728000
memory= 252144
writing 32000 bytes to /test.zip starting at 5760000
memory= 252236
writing 32000 bytes to /test.zip starting at 5792000
memory= 252292
writing 32000 bytes to /test.zip starting at 5824000
memory= 252292
writing 32000 bytes to /test.zip starting at 5856000
memory= 252420
writing 32000 bytes to /test.zip starting at 5888000
memory= 252460
writing 32000 bytes to /test.zip starting at 5920000
memory= 252460
writing 32000 bytes to /test.zip starting at 5952000
memory= 252616
writing 32000 bytes to /test.zip starting at 5984000
memory= 252668
writing 32000 bytes to /test.zip starting at 6016000
memory= 252668
writing 32000 bytes to /test.zip starting at 6048000
memory= 258668
writing 32000 bytes to /test.zip starting at 6080000
memory= 270584
writing 32000 bytes to /test.zip starting at 6112000
memory= 276732
writing 32000 bytes to /test.zip starting at 6144000
memory= 276764
writing 32000 bytes to /test.zip starting at 6176000
memory= 276764
writing 32000 bytes to /test.zip starting at 6208000
memory= 277064
writing 32000 bytes to /test.zip starting at 6240000
memory= 277336
writing 32000 bytes to /test.zip starting at 6272000
memory= 277552
writing 32000 bytes to /test.zip starting at 6304000
memory= 277552
writing 32000 bytes to /test.zip starting at 6336000
memory= 277796
writing 32000 bytes to /test.zip starting at 6368000
memory= 277880
writing 32000 bytes to /test.zip starting at 6400000
memory= 277880
writing 32000 bytes to /test.zip starting at 6432000
memory= 278124
writing 32000 bytes to /test.zip starting at 6464000
memory= 278172
writing 32000 bytes to /test.zip starting at 6496000
memory= 278172
writing 32000 bytes to /test.zip starting at 6528000
memory= 278352
writing 32000 bytes to /test.zip starting at 6560000
memory= 278504
writing 32000 bytes to /test.zip starting at 6592000
memory= 278548
writing 32000 bytes to /test.zip starting at 6624000
memory= 278548
writing 32000 bytes to /test.zip starting at 6656000
memory= 278736
writing 32000 bytes to /test.zip starting at 6688000
memory= 278824
writing 32000 bytes to /test.zip starting at 6720000
memory= 278824
writing 32000 bytes to /test.zip starting at 6752000
memory= 279076
writing 32000 bytes to /test.zip starting at 6784000
memory= 279112
writing 32000 bytes to /test.zip starting at 6816000
memory= 279112
writing 32000 bytes to /test.zip starting at 6848000
memory= 279300
writing 32000 bytes to /test.zip starting at 6880000
memory= 279392
writing 32000 bytes to /test.zip starting at 6912000
memory= 279392
writing 32000 bytes to /test.zip starting at 6944000
memory= 279544
writing 32000 bytes to /test.zip starting at 6976000
memory= 279624
writing 32000 bytes to /test.zip starting at 7008000
memory= 279600
writing 32000 bytes to /test.zip starting at 7040000
memory= 279600
writing 32000 bytes to /test.zip starting at 7072000
memory= 280496
writing 32000 bytes to /test.zip starting at 7104000
memory= 280492
writing 32000 bytes to /test.zip starting at 7136000
memory= 280492
writing 32000 bytes to /test.zip starting at 7168000
memory= 280480
writing 32000 bytes to /test.zip starting at 7200000
memory= 280504
writing 32000 bytes to /test.zip starting at 7232000
memory= 280504
writing 32000 bytes to /test.zip starting at 7264000
memory= 280516
writing 32000 bytes to /test.zip starting at 7296000
memory= 280504
writing 32000 bytes to /test.zip starting at 7328000
memory= 280464
writing 32000 bytes to /test.zip starting at 7360000
memory= 280464
writing 32000 bytes to /test.zip starting at 7392000
memory= 280464
writing 32000 bytes to /test.zip starting at 7424000
memory= 280492
writing 32000 bytes to /test.zip starting at 7456000
memory= 280492
writing 32000 bytes to /test.zip starting at 7488000
memory= 280504
writing 32000 bytes to /test.zip starting at 7520000
memory= 280448
writing 32000 bytes to /test.zip starting at 7552000
memory= 280448
writing 32000 bytes to /test.zip starting at 7584000
memory= 280436
writing 32000 bytes to /test.zip starting at 7616000
memory= 280532
writing 32000 bytes to /test.zip starting at 7648000
memory= 280464
writing 32000 bytes to /test.zip starting at 7680000
memory= 280464
writing 32000 bytes to /test.zip starting at 7712000
memory= 280528
writing 32000 bytes to /test.zip starting at 7744000
memory= 280476
writing 32000 bytes to /test.zip starting at 7776000
memory= 280476
writing 32000 bytes to /test.zip starting at 7808000
memory= 280472
writing 32000 bytes to /test.zip starting at 7840000
memory= 280480
writing 32000 bytes to /test.zip starting at 7872000
memory= 280480
writing 32000 bytes to /test.zip starting at 7904000
memory= 280448
writing 32000 bytes to /test.zip starting at 7936000
memory= 280504
writing 32000 bytes to /test.zip starting at 7968000
memory= 280404
writing 32000 bytes to /test.zip starting at 8000000
memory= 280404
writing 32000 bytes to /test.zip starting at 8032000
memory= 280468
writing 32000 bytes to /test.zip starting at 8064000
memory= 280500
writing 32000 bytes to /test.zip starting at 8096000
memory= 280500
writing 32000 bytes to /test.zip starting at 8128000
memory= 280484
writing 32000 bytes to /test.zip starting at 8160000
memory= 280372
writing 32000 bytes to /test.zip starting at 8192000
memory= 280372
writing 32000 bytes to /test.zip starting at 8224000
memory= 280436
writing 32000 bytes to /test.zip starting at 8256000
memory= 280468
writing 32000 bytes to /test.zip starting at 8288000
memory= 280468
writing 32000 bytes to /test.zip starting at 8320000
memory= 280416
writing 32000 bytes to /test.zip starting at 8352000
memory= 280448
writing 32000 bytes to /test.zip starting at 8384000
memory= 280356
writing 32000 bytes to /test.zip starting at 8416000
memory= 280356
writing 32000 bytes to /test.zip starting at 8448000
memory= 280420
writing 32000 bytes to /test.zip starting at 8480000
memory= 280452
writing 32000 bytes to /test.zip starting at 8512000
memory= 280452
writing 32000 bytes to /test.zip starting at 8544000
memory= 280484
writing 32000 bytes to /test.zip starting at 8576000
memory= 280508
writing 32000 bytes to /test.zip starting at 8608000
memory= 280508
writing 32000 bytes to /test.zip starting at 8640000
memory= 280444
writing 32000 bytes to /test.zip starting at 8672000
memory= 280484
writing 32000 bytes to /test.zip starting at 8704000
memory= 280372
writing 32000 bytes to /test.zip starting at 8736000
memory= 280372
writing 32000 bytes to /test.zip starting at 8768000
memory= 280436
writing 32000 bytes to /test.zip starting at 8800000
memory= 280468
writing 32000 bytes to /test.zip starting at 8832000
memory= 280468
writing 32000 bytes to /test.zip starting at 8864000
memory= 280404
writing 32000 bytes to /test.zip starting at 8896000
memory= 280420
writing 32000 bytes to /test.zip starting at 8928000
memory= 280420
writing 32000 bytes to /test.zip starting at 8960000
memory= 280340
writing 32000 bytes to /test.zip starting at 8992000
memory= 280388
writing 32000 bytes to /test.zip starting at 9024000
memory= 280248
writing 32000 bytes to /test.zip starting at 9056000
memory= 280276
writing 32000 bytes to /test.zip starting at 9088000
memory= 280276
writing 32000 bytes to /test.zip starting at 9120000
memory= 280340
writing 32000 bytes to /test.zip starting at 9152000
memory= 280200
writing 32000 bytes to /test.zip starting at 9184000
memory= 280212
writing 32000 bytes to /test.zip starting at 9216000
memory= 280212
writing 32000 bytes to /test.zip starting at 9248000
memory= 280276
writing 32000 bytes to /test.zip starting at 9280000
memory= 280304
writing 32000 bytes to /test.zip starting at 9312000
memory= 280304
writing 32000 bytes to /test.zip starting at 9344000
memory= 279928
writing 32000 bytes to /test.zip starting at 9376000
memory= 279936
writing 32000 bytes to /test.zip starting at 9408000
memory= 279936
writing 32000 bytes to /test.zip starting at 9440000
memory= 280000
writing 32000 bytes to /test.zip starting at 9472000
memory= 280056
writing 32000 bytes to /test.zip starting at 9504000
memory= 280060
writing 32000 bytes to /test.zip starting at 9536000
memory= 280060
writing 32000 bytes to /test.zip starting at 9568000
memory= 280124
writing 32000 bytes to /test.zip starting at 9600000
memory= 280156
writing 32000 bytes to /test.zip starting at 9632000
memory= 280156
writing 32000 bytes to /test.zip starting at 9664000
memory= 278476
writing 32000 bytes to /test.zip starting at 9696000
memory= 278508
writing 32000 bytes to /test.zip starting at 9728000
memory= 278508
writing 32000 bytes to /test.zip starting at 9760000
memory= 278576
writing 32000 bytes to /test.zip starting at 9792000
memory= 278600
writing 32000 bytes to /test.zip starting at 9824000
memory= 278600
writing 32000 bytes to /test.zip starting at 9856000
memory= 278664
writing 32000 bytes to /test.zip starting at 9888000
memory= 278704
writing 32000 bytes to /test.zip starting at 9920000
memory= 278728
writing 32000 bytes to /test.zip starting at 9952000
memory= 278728
writing 32000 bytes to /test.zip starting at 9984000
memory= 278792
writing 32000 bytes to /test.zip starting at 10016000
memory= 278820
writing 32000 bytes to /test.zip starting at 10048000
memory= 278820
writing 32000 bytes to /test.zip starting at 10080000
memory= 278896
writing 32000 bytes to /test.zip starting at 10112000
memory= 278916
writing 32000 bytes to /test.zip starting at 10144000
memory= 278916
writing 32000 bytes to /test.zip starting at 10176000
memory= 278980
writing 32000 bytes to /test.zip starting at 10208000
memory= 279024
writing 32000 bytes to /test.zip starting at 10240000
memory= 279040
writing 32000 bytes to /test.zip starting at 10272000
memory= 279040
writing 32000 bytes to /test.zip starting at 10304000
memory= 279104
writing 32000 bytes to /test.zip starting at 10336000
memory= 279156
writing 32000 bytes to /test.zip starting at 10368000
memory= 279156
writing 32000 bytes to /test.zip starting at 10400000
memory= 279236
writing 32000 bytes to /test.zip starting at 10432000
memory= 279248
writing 32000 bytes to /test.zip starting at 10464000
memory= 279248
writing 32000 bytes to /test.zip starting at 10496000
memory= 283460
writing 32000 bytes to /test.zip starting at 10528000
memory= 283596
writing 32000 bytes to /test.zip starting at 10560000
memory= 283636
writing 32000 bytes to /test.zip starting at 10592000
memory= 283636
writing 32000 bytes to /test.zip starting at 10624000
memory= 283764
writing 32000 bytes to /test.zip starting at 10656000
memory= 283820
writing 32000 bytes to /test.zip starting at 10688000
memory= 283820
writing 32000 bytes to /test.zip starting at 10720000
memory= 283976
writing 32000 bytes to /test.zip starting at 10752000
memory= 284012
writing 32000 bytes to /test.zip starting at 10784000
memory= 284012
writing 32000 bytes to /test.zip starting at 10816000
memory= 284132
writing 32000 bytes to /test.zip starting at 10848000
memory= 284172
writing 32000 bytes to /test.zip starting at 10880000
memory= 284180
writing 32000 bytes to /test.zip starting at 10912000
memory= 284180
writing 32000 bytes to /test.zip starting at 10944000
memory= 284244
writing 32000 bytes to /test.zip starting at 10976000
memory= 284272
writing 32000 bytes to /test.zip starting at 11008000
memory= 284272
writing 32000 bytes to /test.zip starting at 11040000
memory= 284336
writing 32000 bytes to /test.zip starting at 11072000
memory= 284340
writing 32000 bytes to /test.zip starting at 11104000
memory= 284340
writing 32000 bytes to /test.zip starting at 11136000
memory= 284380
writing 32000 bytes to /test.zip starting at 11168000
memory= 284376
writing 32000 bytes to /test.zip starting at 11200000
memory= 284376
writing 32000 bytes to /test.zip starting at 11232000
memory= 284408
writing 32000 bytes to /test.zip starting at 11264000
memory= 284428
writing 32000 bytes to /test.zip starting at 11296000
memory= 284484
writing 32000 bytes to /test.zip starting at 11328000
memory= 284484
writing 32000 bytes to /test.zip starting at 11360000
memory= 284536
writing 32000 bytes to /test.zip starting at 11392000
memory= 284536
writing 32000 bytes to /test.zip starting at 11424000
memory= 284536
writing 32000 bytes to /test.zip starting at 11456000
memory= 284692
writing 32000 bytes to /test.zip starting at 11488000
memory= 284652
writing 32000 bytes to /test.zip starting at 11520000
memory= 284652
writing 32000 bytes to /test.zip starting at 11552000
memory= 284716
writing 32000 bytes to /test.zip starting at 11584000
memory= 284708
writing 32000 bytes to /test.zip starting at 11616000
memory= 284776
writing 32000 bytes to /test.zip starting at 11648000
memory= 284776
writing 32000 bytes to /test.zip starting at 11680000
memory= 284792
writing 32000 bytes to /test.zip starting at 11712000
memory= 284844
writing 32000 bytes to /test.zip starting at 11744000
memory= 284844
writing 32000 bytes to /test.zip starting at 11776000
memory= 284892
writing 32000 bytes to /test.zip starting at 11808000
memory= 284816
writing 32000 bytes to /test.zip starting at 11840000
memory= 284816
writing 32000 bytes to /test.zip starting at 11872000
memory= 284932
writing 32000 bytes to /test.zip starting at 11904000
memory= 284956
writing 32000 bytes to /test.zip starting at 11936000
memory= 284860
writing 32000 bytes to /test.zip starting at 11968000
memory= 284860
writing 32000 bytes to /test.zip starting at 12000000
memory= 285012
writing 32000 bytes to /test.zip starting at 12032000
memory= 284928
writing 32000 bytes to /test.zip starting at 12064000
memory= 284928
writing 32000 bytes to /test.zip starting at 12096000
memory= 296836
writing 32000 bytes to /test.zip starting at 12128000
memory= 296872
writing 32000 bytes to /test.zip starting at 12160000
memory= 296872
writing 32000 bytes to /test.zip starting at 12192000
memory= 296996
writing 32000 bytes to /test.zip starting at 12224000
memory= 292516
writing 32000 bytes to /test.zip starting at 12256000
memory= 292548
writing 32000 bytes to /test.zip starting at 12288000
memory= 292548
writing 32000 bytes to /test.zip starting at 12320000
memory= 316784
writing 32000 bytes to /test.zip starting at 12352000
memory= 328964
writing 32000 bytes to /test.zip starting at 12384000
memory= 328964
writing 32000 bytes to /test.zip starting at 12416000
memory= 353384
writing 32000 bytes to /test.zip starting at 12448000
memory= 365600
writing 32000 bytes to /test.zip starting at 12480000
memory= 365600
writing 32000 bytes to /test.zip starting at 12512000
memory= 402584
writing 32000 bytes to /test.zip starting at 12544000
memory= 414916
writing 32000 bytes to /test.zip starting at 12576000
memory= 414916
writing 32000 bytes to /test.zip starting at 12608000
memory= 439716
writing 32000 bytes to /test.zip starting at 12640000
memory= 464564
writing 32000 bytes to /test.zip starting at 12672000
memory= 477100
writing 32000 bytes to /test.zip starting at 12704000
memory= 477100
writing 32000 bytes to /test.zip starting at 12736000
memory= 502152
writing 32000 bytes to /test.zip starting at 12768000
memory= 514684
writing 32000 bytes to /test.zip starting at 12800000
memory= 514684
writing 32000 bytes to /test.zip starting at 12832000
memory= 552600
writing 32000 bytes to /test.zip starting at 12864000
memory= 565272
writing 32000 bytes to /test.zip starting at 12896000
memory= 565272
writing 32000 bytes to /test.zip starting at 12928000
memory= 590684
writing 32000 bytes to /test.zip starting at 12960000
memory= 616168
writing 32000 bytes to /test.zip starting at 12992000
memory= 628932
writing 32000 bytes to /test.zip starting at 13024000
memory= 628932
writing 32000 bytes to /test.zip starting at 13056000
memory= 654612
writing 32000 bytes to /test.zip starting at 13088000
memory= 667500
writing 32000 bytes to /test.zip starting at 13120000
memory= 667548
writing 32000 bytes to /test.zip starting at 13152000
memory= 706336
writing 32000 bytes to /test.zip starting at 13184000
memory= 719280
writing 32000 bytes to /test.zip starting at 13216000
memory= 719280
writing 32000 bytes to /test.zip starting at 13248000
memory= 745324
writing 32000 bytes to /test.zip starting at 13280000
memory= 771480
writing 32000 bytes to /test.zip starting at 13312000
memory= 784544
writing 32000 bytes to /test.zip starting at 13344000
memory= 784544
writing 32000 bytes to /test.zip starting at 13376000
memory= 810848
writing 32000 bytes to /test.zip starting at 13408000
memory= 824056
writing 32000 bytes to /test.zip starting at 13440000
memory= 824056
writing 32000 bytes to /test.zip starting at 13472000
memory= 863808
writing 32000 bytes to /test.zip starting at 13504000
memory= 877056
writing 32000 bytes to /test.zip starting at 13536000
memory= 877056
writing 32000 bytes to /test.zip starting at 13568000
memory= 903764
writing 32000 bytes to /test.zip starting at 13600000
memory= 930532
writing 32000 bytes to /test.zip starting at 13632000
memory= 943908
writing 32000 bytes to /test.zip starting at 13664000
memory= 943908
writing 32000 bytes to /test.zip starting at 13696000
memory= 970864
writing 32000 bytes to /test.zip starting at 13728000
memory= 984320
writing 32000 bytes to /test.zip starting at 13760000
memory= 984320
writing 32000 bytes to /test.zip starting at 13792000
memory= 1011432
writing 32000 bytes to /test.zip starting at 13824000
memory= 1024936
writing 32000 bytes to /test.zip starting at 13856000
memory= 1024936
writing 32000 bytes to /test.zip starting at 13888000
memory= 1065732
writing 32000 bytes to /test.zip starting at 13920000
memory= 1079376
writing 32000 bytes to /test.zip starting at 13952000
memory= 1079376
writing 32000 bytes to /test.zip starting at 13984000
memory= 1106700
writing 32000 bytes to /test.zip starting at 14016000
memory= 1134144
writing 32000 bytes to /test.zip starting at 14048000
memory= 1147848
writing 32000 bytes to /test.zip starting at 14080000
memory= 1147848
writing 32000 bytes to /test.zip starting at 14112000
memory= 1175520
writing 32000 bytes to /test.zip starting at 14144000
memory= 1189296
writing 32000 bytes to /test.zip starting at 14176000
memory= 1189348
writing 32000 bytes to /test.zip starting at 14208000
memory= 1231024
writing 32000 bytes to /test.zip starting at 14240000
memory= 1245036
writing 32000 bytes to /test.zip starting at 14272000
memory= 1245036
writing 32000 bytes to /test.zip starting at 14304000
memory= 1272960
writing 32000 bytes to /test.zip starting at 14336000
memory= 1301080
writing 32000 bytes to /test.zip starting at 14368000
memory= 1315216
writing 32000 bytes to /test.zip starting at 14400000
memory= 1315216
writing 32000 bytes to /test.zip starting at 14432000
memory= 1343496
writing 32000 bytes to /test.zip starting at 14464000
memory= 1351596
writing 32000 bytes to /test.zip starting at 14496000
memory= 1351596
writing 32000 bytes to /test.zip starting at 14528000
memory= 1394508
writing 32000 bytes to /test.zip starting at 14560000
memory= 1408828
writing 32000 bytes to /test.zip starting at 14592000
memory= 1408828
writing 32000 bytes to /test.zip starting at 14624000
memory= 1437560
writing 32000 bytes to /test.zip starting at 14656000
memory= 1466464
writing 32000 bytes to /test.zip starting at 14688000
memory= 1480908
writing 32000 bytes to /test.zip starting at 14720000
memory= 1480908
writing 32000 bytes to /test.zip starting at 14752000
memory= 1509904
writing 32000 bytes to /test.zip starting at 14784000
memory= 1524456
writing 32000 bytes to /test.zip starting at 14816000
memory= 1524456
writing 32000 bytes to /test.zip starting at 14848000
memory= 1568324
writing 32000 bytes to /test.zip starting at 14880000
memory= 1582952
writing 32000 bytes to /test.zip starting at 14912000
memory= 1582952
writing 32000 bytes to /test.zip starting at 14944000
memory= 1612352
writing 32000 bytes to /test.zip starting at 14976000
memory= 1641844
writing 32000 bytes to /test.zip starting at 15008000
memory= 1656592
writing 32000 bytes to /test.zip starting at 15040000
memory= 1656592
writing 32000 bytes to /test.zip starting at 15072000
memory= 1686216
writing 32000 bytes to /test.zip starting at 15104000
memory= 1701092
writing 32000 bytes to /test.zip starting at 15136000
memory= 1701092
writing 32000 bytes to /test.zip starting at 15168000
memory= 1730928
writing 32000 bytes to /test.zip starting at 15200000
memory= 1745868
writing 32000 bytes to /test.zip starting at 15232000
memory= 1745928
writing 32000 bytes to /test.zip starting at 15264000
memory= 1791016
writing 32000 bytes to /test.zip starting at 15296000
memory= 1806072
writing 32000 bytes to /test.zip starting at 15328000
memory= 1806072
writing 32000 bytes to /test.zip starting at 15360000
memory= 1836260
writing 32000 bytes to /test.zip starting at 15392000
memory= 1866492
writing 32000 bytes to /test.zip starting at 15424000
memory= 1881672
writing 32000 bytes to /test.zip starting at 15456000
memory= 1881672
writing 32000 bytes to /test.zip starting at 15488000
memory= 1881672
writing 32000 bytes to /test.zip starting at 15520000
memory= 1942744
writing 32000 bytes to /test.zip starting at 15552000
memory= 1942744
writing 32000 bytes to /test.zip starting at 15584000
memory= 1942744
writing 32000 bytes to /test.zip starting at 15616000
memory= 2004068
writing 32000 bytes to /test.zip starting at 15648000
memory= 2004068
writing 32000 bytes to /test.zip starting at 15680000
memory= 2004068
writing 32000 bytes to /test.zip starting at 15712000
memory= 2065704
writing 32000 bytes to /test.zip starting at 15744000
memory= 2065704
writing 32000 bytes to /test.zip starting at 15776000
memory= 2065704
writing 32000 bytes to /test.zip starting at 15808000
memory= 2112060
writing 32000 bytes to /test.zip starting at 15840000
memory= 2112060
writing 32000 bytes to /test.zip starting at 15872000
memory= 2158664
writing 32000 bytes to /test.zip starting at 15904000
memory= 2174244
writing 32000 bytes to /test.zip starting at 15936000
memory= 2174244
writing 32000 bytes to /test.zip starting at 15968000
memory= 2205540
writing 32000 bytes to /test.zip starting at 16000000
memory= 2236788
writing 32000 bytes to /test.zip starting at 16032000
memory= 2252520
writing 32000 bytes to /test.zip starting at 16064000
memory= 2252520
writing 32000 bytes to /test.zip starting at 16096000
memory= 2284192
writing 32000 bytes to /test.zip starting at 16128000
memory= 2299956
writing 32000 bytes to /test.zip starting at 16160000
memory= 2300008
writing 32000 bytes to /test.zip starting at 16192000
memory= 2347660
writing 32000 bytes to /test.zip starting at 16224000
memory= 2363532
writing 32000 bytes to /test.zip starting at 16256000
memory= 2363532
writing 32000 bytes to /test.zip starting at 16288000
memory= 2383840
writing 32000 bytes to /test.zip starting at 16320000
memory= 2415992
writing 32000 bytes to /test.zip starting at 16352000
memory= 2432072
writing 32000 bytes to /test.zip starting at 16384000
memory= 2432072
writing 32000 bytes to /test.zip starting at 16416000
memory= 2464324
writing 32000 bytes to /test.zip starting at 16448000
memory= 2480496
writing 32000 bytes to /test.zip starting at 16480000
memory= 2480496
writing 32000 bytes to /test.zip starting at 16512000
memory= 2529260
writing 32000 bytes to /test.zip starting at 16544000
memory= 2545524
writing 32000 bytes to /test.zip starting at 16576000
memory= 2545524
writing 32000 bytes to /test.zip starting at 16608000
memory= 2578176
writing 32000 bytes to /test.zip starting at 16640000
memory= 2610952
writing 32000 bytes to /test.zip starting at 16672000
memory= 2627336
writing 32000 bytes to /test.zip starting at 16704000
memory= 2627336
writing 32000 bytes to /test.zip starting at 16736000
memory= 2660252
writing 32000 bytes to /test.zip starting at 16768000
memory= 2676748
writing 32000 bytes to /test.zip starting at 16800000
memory= 2676748
writing 32000 bytes to /test.zip starting at 16832000
memory= 2726464
writing 32000 bytes to /test.zip starting at 16864000
memory= 2743032
writing 32000 bytes to /test.zip starting at 16896000
memory= 2743032
writing 32000 bytes to /test.zip starting at 16928000
memory= 2776316
writing 32000 bytes to /test.zip starting at 16960000
memory= 2809688
writing 32000 bytes to /test.zip starting at 16992000
memory= 2826376
writing 32000 bytes to /test.zip starting at 17024000
memory= 2826376
writing 32000 bytes to /test.zip starting at 17056000
memory= 2859912
writing 32000 bytes to /test.zip starting at 17088000
memory= 2876728
writing 32000 bytes to /test.zip starting at 17120000
memory= 2876728
writing 32000 bytes to /test.zip starting at 17152000
memory= 2910488
writing 32000 bytes to /test.zip starting at 17184000
memory= 2927364
writing 32000 bytes to /test.zip starting at 17216000
memory= 2927424
writing 32000 bytes to /test.zip starting at 17248000
memory= 2978348
writing 32000 bytes to /test.zip starting at 17280000
memory= 2995348
writing 32000 bytes to /test.zip starting at 17312000
memory= 2995348
writing 32000 bytes to /test.zip starting at 17344000
memory= 3029424
writing 32000 bytes to /test.zip starting at 17376000
memory= 3063564

Any help appreciated....

Regards
Till

@tillvollmer

Hi I added ObjectSpace statistics, as we use Ruby Enterprise Edition. As the stats are more or less the same.
Does that mean that it is external memory, like from openssl lib included in Ruby and most probably used by net-ssh/ftp?

Also: It does not happen every time, seems to be dependent on the quality of the connection?!

Any hints are welcome.

Regards
Till

Start of upload

memory= 240924 # 240KB
objectspace = Number of objects : 1429803 (1167816 AST nodes, 81.68%)
Heap slot size : 40
GC cycles so far : 138
Number of heaps : 9
Total size of objects: 55851.68 KB
Total size of heaps : 119579.95 KB (63728.27 KB = 53.29% unused)
Leading free slots : 1111438 (43415.55 KB = 36.31%)
Trailing free slots : 7 (0.27 KB = 0.00%)
Number of contiguous groups of 16 slots: 88962 (46.50%)
Number of terminal objects: 6538 (0.21%)

in the middle of upload
memory= 2931252 # already 2,9 GB
objectspace = Number of objects : 1425260 (1167608 AST nodes, 81.92%)
Heap slot size : 40
GC cycles so far : 1944
Number of heaps : 9
Total size of objects: 55674.22 KB
Total size of heaps : 119579.95 KB (63905.73 KB = 53.44% unused)
Leading free slots : 1287604 (50297.03 KB = 42.06%)
Trailing free slots : 7 (0.27 KB = 0.00%)
Number of contiguous groups of 16 slots: 89203 (46.62%)
Number of terminal objects: 6534 (0.21%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.