New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speeding up Shoes web requests #307

Open
BackOrder opened this Issue Jan 30, 2017 · 29 comments

Comments

2 participants
@BackOrder
Collaborator

BackOrder commented Jan 30, 2017

Shoes web requests are very slow. Installing Typhoeus and libcurl considerably speeds up web requests. Expert-funnies runs in a blink of an eye. It would be a fantastic to include in Shoes.

How to install Typhoeus and libcurl on Windows?

  • Go in Shoes Maintenance > Manage Gems, search remote for Typhoeus and install
  • Download curl-7.40.0-rtmp-ssh2-ssl-sspi-zlib-idn-static-bin-w32.7z
  • Unzip curl in a directory such as C:\Program Files\Curl
  • Go to Control Panel > System and Maintenance > System > Advance System Settings
  • Go to Environment Variables and edit PATH (either the user or system, your choice)
  • Add ;C:\Program Files\Curl to your PATH
  • You may have to reboot to take into effects

Updated expert-funnies.rb:

require("nokogiri")
require("typhoeus")

class Comic
   attr_reader :rss, :title

   def initialize(body)
      @rss = Nokogiri::XML(body)
      @title = @rss.at("//channel/title").inner_text
   end

   def items
      @rss.search("//channel/item")
   end

   def latest_image
      @rss.search("//channel/item").first.inner_html.scan(/src="([^"]+\.\w+)"/).first
   end
end

Shoes.app width: 800, height: 600 do
   background "#555"

   @title = "Web Funnies"
   @feeds = [
      "http://xkcd.com/rss.xml",
      #"http://www.hoodcomputing.com/garfield.php" # gif image causing segfault
   ]

   stack margin: 10 do
      title strong(@title), align: "center", stroke: "#DFA", margin: 0
      para "(loaded from RSS feeds)", align: "center", stroke: "#DFA", margin: 0

      @feeds.each do |feed|
         response = Typhoeus.get(feed, followlocation: true, ssl_verifypeer: false)
         stack width: "100%", margin: 10, border: 1 do
            c = Comic.new response.body
            stack margin_right: gutter do
               background "#333", curve: 4
               caption c.title, stroke: "#CD9", margin: 4
            end
            image c.latest_image.first, margin: 8
         end
      end
   end
end
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jan 30, 2017

Contributor

One of the first things I did many years ago was drop curl from Shoes 3. - It was incredibly hard to cross compile the library, they changed the api into the library dealing with notifications/callbacks which didn't look compatible with Shoes. Back then, open_uri was known to be slow, still is but it's one less dependency. We also know since then it has some other 'bugs'.

Contributor

ccoupe commented Jan 30, 2017

One of the first things I did many years ago was drop curl from Shoes 3. - It was incredibly hard to cross compile the library, they changed the api into the library dealing with notifications/callbacks which didn't look compatible with Shoes. Back then, open_uri was known to be slow, still is but it's one less dependency. We also know since then it has some other 'bugs'.

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Jan 30, 2017

Collaborator

The good news is that Shoes does not have to include libcurl in the same way as before because Typhoeus deals with it. Now your feedback is that including libcurl at C level is cumbersome. Using Typhoeus would just mean including Typhoeus gem and libcurl precompiled libraries. Wouldn't it be something closer to our reach?

Collaborator

BackOrder commented Jan 30, 2017

The good news is that Shoes does not have to include libcurl in the same way as before because Typhoeus deals with it. Now your feedback is that including libcurl at C level is cumbersome. Using Typhoeus would just mean including Typhoeus gem and libcurl precompiled libraries. Wouldn't it be something closer to our reach?

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Jan 30, 2017

Collaborator

There is currently no callback for a progress bar in Typhoeus though libcurl does support it. Reference typhoeus/typhoeus#554.

Collaborator

BackOrder commented Jan 30, 2017

There is currently no callback for a progress bar in Typhoeus though libcurl does support it. Reference typhoeus/typhoeus#554.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jan 30, 2017

Contributor

I was about to ask about that callback. That was part of the problem with C level curl too. I've already noted elsewhere the days of of cross compiling Shoes for Window may be over soon. This is just another reason. It would be easier for us if Typhoeus included a precompiledt/gemified libcurl for mingw but not a requirement.

Contributor

ccoupe commented Jan 30, 2017

I was about to ask about that callback. That was part of the problem with C level curl too. I've already noted elsewhere the days of of cross compiling Shoes for Window may be over soon. This is just another reason. It would be easier for us if Typhoeus included a precompiledt/gemified libcurl for mingw but not a requirement.

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Jan 31, 2017

Collaborator

The callback actually always been available in curl. It was called CURLOPT_PROGRESSFUNCTION but documentation encourage to use CURLOPT_XFERINFOFUNCTION now.

the days of of cross compiling Shoes for Window may be over soon

What does it mean concretely? Why is that?

It would be easier for us if Typhoeus included a precompiledt/gemified libcurl for mingw but not a requirement.

Agreed. We should rely more on Ruby. There are several things that could be written in Ruby rather than C. Typhoeus/libcurl is a prime example that speed is not affected by relying on Ruby. The idea would be to wrap Typhoeus in the infamous Download class and provide download method that uses this gem. It would make very little to no difference in term of speed to be implemented in C code with Shoes.

Why not a requirement? The whole thing would be roughly an additional 800KB including DLL files because some of the DLLs are already in Shoes.

Collaborator

BackOrder commented Jan 31, 2017

The callback actually always been available in curl. It was called CURLOPT_PROGRESSFUNCTION but documentation encourage to use CURLOPT_XFERINFOFUNCTION now.

the days of of cross compiling Shoes for Window may be over soon

What does it mean concretely? Why is that?

It would be easier for us if Typhoeus included a precompiledt/gemified libcurl for mingw but not a requirement.

Agreed. We should rely more on Ruby. There are several things that could be written in Ruby rather than C. Typhoeus/libcurl is a prime example that speed is not affected by relying on Ruby. The idea would be to wrap Typhoeus in the infamous Download class and provide download method that uses this gem. It would make very little to no difference in term of speed to be implemented in C code with Shoes.

Why not a requirement? The whole thing would be roughly an additional 800KB including DLL files because some of the DLLs are already in Shoes.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jan 31, 2017

Contributor

What does it mean concretely? Why is that?

Ruby 2.2.6 won't cross compile. There's some other things don't cross compile anymore unless big hurdles are crossed. Every body wants to run tests or finalize something in their build by running what was built by running some exe on Windows before the build is complete. It happens because nobody thinks about cross compiling or folks like nokogiri create their own build framework which is at odds with how Shoes does cross platform builds. We know we need to build libgif and other dependencies. Might as well do that on Windows and skip the cross compile problems.

Actually, the curl source code is still in the shoes/http/ dir. Out of date of course. download follows a tortuous path in C as it builds the ruby structs needed to call up into Ruby lib/shoes/dowload.rb That mess attempts to emulate the callbacks/event that curl exposed to the download api. We need to fix it for #304 independently of this issue.

We could create a new Shoes method that doesn't have the progress callbacks and uses what ever Typhoeus allows. Or perhaps use ffi to hook into the callbacks. I haven't studied their code.

Contributor

ccoupe commented Jan 31, 2017

What does it mean concretely? Why is that?

Ruby 2.2.6 won't cross compile. There's some other things don't cross compile anymore unless big hurdles are crossed. Every body wants to run tests or finalize something in their build by running what was built by running some exe on Windows before the build is complete. It happens because nobody thinks about cross compiling or folks like nokogiri create their own build framework which is at odds with how Shoes does cross platform builds. We know we need to build libgif and other dependencies. Might as well do that on Windows and skip the cross compile problems.

Actually, the curl source code is still in the shoes/http/ dir. Out of date of course. download follows a tortuous path in C as it builds the ruby structs needed to call up into Ruby lib/shoes/dowload.rb That mess attempts to emulate the callbacks/event that curl exposed to the download api. We need to fix it for #304 independently of this issue.

We could create a new Shoes method that doesn't have the progress callbacks and uses what ever Typhoeus allows. Or perhaps use ffi to hook into the callbacks. I haven't studied their code.

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Jan 31, 2017

Collaborator

Ruby 2.2.6 won't cross compile.

How about setting up a build box on Windows (or a Windows VM)? You could control the build from your favourite OS, sending commands, automatically download the resulting logs and packages.

RE: CURL

This is really up to you. Either way, using open_uri and/or Net::HTTP is just too slow.

We could create a new Shoes method that doesn't have the progress callbacks and uses what ever Typhoeus allows. Or perhaps use ffi to hook into the callbacks. I haven't studied their code.

Attempted to do exactly that! There is something missing to actually callback but no idea what. First day on the job as we would say. :)

https://github.com/BackOrder/ethon
https://github.com/BackOrder/typhoeus

Collaborator

BackOrder commented Jan 31, 2017

Ruby 2.2.6 won't cross compile.

How about setting up a build box on Windows (or a Windows VM)? You could control the build from your favourite OS, sending commands, automatically download the resulting logs and packages.

RE: CURL

This is really up to you. Either way, using open_uri and/or Net::HTTP is just too slow.

We could create a new Shoes method that doesn't have the progress callbacks and uses what ever Typhoeus allows. Or perhaps use ffi to hook into the callbacks. I haven't studied their code.

Attempted to do exactly that! There is something missing to actually callback but no idea what. First day on the job as we would say. :)

https://github.com/BackOrder/ethon
https://github.com/BackOrder/typhoeus

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Jan 31, 2017

Contributor

Aha, they use ffi ! This is good news.

How about setting up a build box on Windows (or a Windows VM)? You could control the build from your favourite OS, sending commands, automatically download the resulting logs and packages.

A lot for me to learn. I know I can start a VM with some Virtual box script but once its started how do I login. How do I run a rake script....

Contributor

ccoupe commented Jan 31, 2017

Aha, they use ffi ! This is good news.

How about setting up a build box on Windows (or a Windows VM)? You could control the build from your favourite OS, sending commands, automatically download the resulting logs and packages.

A lot for me to learn. I know I can start a VM with some Virtual box script but once its started how do I login. How do I run a rake script....

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Jan 31, 2017

Collaborator

A lot for me to learn. I know I can start a VM with some Virtual box script but once its started how do I login. How do I run a rake script....

You should perhaps consider something simple to start with. Running a scheduled rakefile would be something to make you comfortable. The said rakefile could download source from github, libraries, etc., check if it needs to be packaged again, compile when there are changes and then send resulting logs and packaged apps to your ftp. You'd only have to turn on the VM and the scheduler would take care of the rest.

It would otherwise be possible to run a small Ruby client/server if you want live output. The server would also be on a scheduler to make sure it's always running (or restarting when needed). It would look like this:

$ walk download
Connecting to Windows server...
Downloading source code from github. OK.
Downloading libraries. OK
$ walk build
Connecting to Windows server...
Building Shoes for Windows...
[... some compiler output...]
$ walk package
Connecting to Windows server...
Packaging Shoes for Windows...
$ walk upload ftp://walkabout/shoes/windows/
Connecting to Windows server...
Uploading Shoes for Windows on ftp...
Collaborator

BackOrder commented Jan 31, 2017

A lot for me to learn. I know I can start a VM with some Virtual box script but once its started how do I login. How do I run a rake script....

You should perhaps consider something simple to start with. Running a scheduled rakefile would be something to make you comfortable. The said rakefile could download source from github, libraries, etc., check if it needs to be packaged again, compile when there are changes and then send resulting logs and packaged apps to your ftp. You'd only have to turn on the VM and the scheduler would take care of the rest.

It would otherwise be possible to run a small Ruby client/server if you want live output. The server would also be on a scheduler to make sure it's always running (or restarting when needed). It would look like this:

$ walk download
Connecting to Windows server...
Downloading source code from github. OK.
Downloading libraries. OK
$ walk build
Connecting to Windows server...
Building Shoes for Windows...
[... some compiler output...]
$ walk package
Connecting to Windows server...
Packaging Shoes for Windows...
$ walk upload ftp://walkabout/shoes/windows/
Connecting to Windows server...
Uploading Shoes for Windows on ftp...
@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Mar 14, 2017

Collaborator

My attempt to add progress feature to Typhoeus worked with some help from the maintainer. It means we will be able to completely replace the current slow download feature and solve the https problems. It can be integrated in either 3.3.3 or 3.3.4.

Collaborator

BackOrder commented Mar 14, 2017

My attempt to add progress feature to Typhoeus worked with some help from the maintainer. It means we will be able to completely replace the current slow download feature and solve the https problems. It can be integrated in either 3.3.3 or 3.3.4.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Aug 9, 2017

Contributor

I have cross compiled curl 7.50-dev and curl.exe works on win7 - except for https (no certificates). This is good. The build process isn't as tedious as it used to be. The certficate problem is probably a curl config problem and I suspect there may not be good certs in openssl . RubyInstaller is (now) pretty explicit about not including them in 2.x.x - a minor shoes build addition to the rakefiles once I figure out where they are, and where they should be. Odd's are very high they aren't where my curl.exe expects them. There are many options to explore. But, we have a test program - curl.exe.

[Update]
This configuration uses the windows ssl and the curl.exe does work for https: (cross compiled). If ruby could that we wouldn't need openssl. But it doesn't or I don't know to tell it too.

Contributor

ccoupe commented Aug 9, 2017

I have cross compiled curl 7.50-dev and curl.exe works on win7 - except for https (no certificates). This is good. The build process isn't as tedious as it used to be. The certficate problem is probably a curl config problem and I suspect there may not be good certs in openssl . RubyInstaller is (now) pretty explicit about not including them in 2.x.x - a minor shoes build addition to the rakefiles once I figure out where they are, and where they should be. Odd's are very high they aren't where my curl.exe expects them. There are many options to explore. But, we have a test program - curl.exe.

[Update]
This configuration uses the windows ssl and the curl.exe does work for https: (cross compiled). If ruby could that we wouldn't need openssl. But it doesn't or I don't know to tell it too.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Aug 9, 2017

Contributor

https (using openssl) seems to work for me in Shoes 3.3.4 (change the url in samples/simple/downloader.rb and image.rb). It may be slow but it works. So, openssl and certs are probably OK. Thats for the net/http ruby stuff.

I'm not keen on rejuvenating the old Shoes/C code to call curl at the C level for download and image. but it could be done. Since Shoes is currently doing a c->Ruby (lib/shoes/download.rb) that's what needs to be replaced with Typhoeus which means I need to build that gem on the 5 platforms so it and libcurl.so/dll/dylib can be.called from Shoes. A lot of cross platform dragons since we can't depend on linux or osx to have libcurl.so/dyllib

Contributor

ccoupe commented Aug 9, 2017

https (using openssl) seems to work for me in Shoes 3.3.4 (change the url in samples/simple/downloader.rb and image.rb). It may be slow but it works. So, openssl and certs are probably OK. Thats for the net/http ruby stuff.

I'm not keen on rejuvenating the old Shoes/C code to call curl at the C level for download and image. but it could be done. Since Shoes is currently doing a c->Ruby (lib/shoes/download.rb) that's what needs to be replaced with Typhoeus which means I need to build that gem on the 5 platforms so it and libcurl.so/dll/dylib can be.called from Shoes. A lot of cross platform dragons since we can't depend on linux or osx to have libcurl.so/dyllib

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Aug 9, 2017

Collaborator

It is noteworthy to mention that while my tests with Typheous were excellent both on features and speed, the fact remains that GIL (and possibly GTK threads) is preventing it to be an effective solution. It basically turned out to be Shoes UI is not updated while downloading. It's not a big deal for small files but it's a huge problem for larger files.

This may also apply to older curl code at C level but you would know better.

Collaborator

BackOrder commented Aug 9, 2017

It is noteworthy to mention that while my tests with Typheous were excellent both on features and speed, the fact remains that GIL (and possibly GTK threads) is preventing it to be an effective solution. It basically turned out to be Shoes UI is not updated while downloading. It's not a big deal for small files but it's a huge problem for larger files.

This may also apply to older curl code at C level but you would know better.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Aug 10, 2017

Contributor

Oops! I forgot I told download.rb to not verify ssl. So, openssl certificates is likely to still be wrong on Shoes/Windows.

I installed the typhoeus gem with draged in ethon in Ruby 2.3.3/Windows, It didn't build anything native so it must be using FFI. I built Shoes (win7), including the libcurl-4.dll I built yesterday and cshoes.exe -c Manage gems ->show local -> load ethon provided this useful error message:

ethon-cobbler

If those were all the libcurls it searched for, it wouldn't find libcurl-4.dll. It appears it is the ethon gem that is doing the ffi since it's the gem requiring the ffi binary gem (which Shoes includes). That will need fixing. I haven't looked at the ethon code yet but it could be a simple fix. Or not.

Shoes progress (the widget) or the Shoes.Setup gem handling or download does not have a smooth gui update. Part of that is Ruby, and part of that is Shoes. (and part of it is the OS). I suspect it could be improved which some heuristics, Typheous won't be worse in that dimension - assuming the callback/yield works as documented.

Contributor

ccoupe commented Aug 10, 2017

Oops! I forgot I told download.rb to not verify ssl. So, openssl certificates is likely to still be wrong on Shoes/Windows.

I installed the typhoeus gem with draged in ethon in Ruby 2.3.3/Windows, It didn't build anything native so it must be using FFI. I built Shoes (win7), including the libcurl-4.dll I built yesterday and cshoes.exe -c Manage gems ->show local -> load ethon provided this useful error message:

ethon-cobbler

If those were all the libcurls it searched for, it wouldn't find libcurl-4.dll. It appears it is the ethon gem that is doing the ffi since it's the gem requiring the ffi binary gem (which Shoes includes). That will need fixing. I haven't looked at the ethon code yet but it could be a simple fix. Or not.

Shoes progress (the widget) or the Shoes.Setup gem handling or download does not have a smooth gui update. Part of that is Ruby, and part of that is Shoes. (and part of it is the OS). I suspect it could be improved which some heuristics, Typheous won't be worse in that dimension - assuming the callback/yield works as documented.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Aug 10, 2017

Contributor

ethon/curls/settings.rb: ffi_lib ['libcurl', 'libcurl.so.4'] - seems like place to explore.

Contributor

ccoupe commented Aug 10, 2017

ethon/curls/settings.rb: ffi_lib ['libcurl', 'libcurl.so.4'] - seems like place to explore.

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Aug 10, 2017

Collaborator

Please, read the first post. It contains all the instructions to make it work without having to change code.

Collaborator

BackOrder commented Aug 10, 2017

Please, read the first post. It contains all the instructions to make it work without having to change code.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Aug 10, 2017

Contributor

@BackOrder - always good to be reminded. You installed curl (the app) and changed the PATH in order to get a working test harness. However, Shoes users shouldn't have to do that for basic functionality We only need libcurl*dll inside the shoes app and the gems in the ruby. Code changes will be required to ethon/typheous - it could be an override/monkey patch we can do in Shoes or there is some upper level setting to add to that array passed into ffi_lib.

Contributor

ccoupe commented Aug 10, 2017

@BackOrder - always good to be reminded. You installed curl (the app) and changed the PATH in order to get a working test harness. However, Shoes users shouldn't have to do that for basic functionality We only need libcurl*dll inside the shoes app and the gems in the ruby. Code changes will be required to ethon/typheous - it could be an override/monkey patch we can do in Shoes or there is some upper level setting to add to that array passed into ffi_lib.

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Aug 10, 2017

Collaborator

This is just something to get you started. I suspect it should work if the proper library is in Shoes directory.

Collaborator

BackOrder commented Aug 10, 2017

This is just something to get you started. I suspect it should work if the proper library is in Shoes directory.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Aug 11, 2017

Contributor

Setting this gets ethon to work. (xwin7)

ccoupe@bronco:~/Projects/gems/rb23/built/win7/ethon-0.10.1/lib/ethon/curls$ more settings.rb
module Ethon
  module Curl
    callback :callback, [:pointer, :size_t, :size_t, :pointer], :size_t
    callback :debug_callback, [:pointer, :debug_info_type, :pointer, :size_t, :p
ointer], :int
    ffi_lib_flags :now, :global
    ffi_lib ['libcurl', 'libcurl-4', 'libcurl.so.4']
  end
end

No sure how that can be monkey patched - i believe it's being call by the 'require'
On osx we have

ccoupe@mini:/usr/lib$ ls -l *curl*
lrwxr-xr-x  1 root  wheel      15 Oct 20  2014 libcurl.3.dylib -> libcurl.4.dylib
-rwxr-xr-x  1 root  wheel  781616 Jul 23  2015 libcurl.4.dylib
lrwxr-xr-x  1 root  wheel      15 Oct 20  2014 libcurl.dylib -> libcurl.4.dylib

which would work with the original code - perhaps the xwin7 curl build can be told to name it libcurl.dll
instead of libcurl-4.dll. That would be best for all.

Contributor

ccoupe commented Aug 11, 2017

Setting this gets ethon to work. (xwin7)

ccoupe@bronco:~/Projects/gems/rb23/built/win7/ethon-0.10.1/lib/ethon/curls$ more settings.rb
module Ethon
  module Curl
    callback :callback, [:pointer, :size_t, :size_t, :pointer], :size_t
    callback :debug_callback, [:pointer, :debug_info_type, :pointer, :size_t, :p
ointer], :int
    ffi_lib_flags :now, :global
    ffi_lib ['libcurl', 'libcurl-4', 'libcurl.so.4']
  end
end

No sure how that can be monkey patched - i believe it's being call by the 'require'
On osx we have

ccoupe@mini:/usr/lib$ ls -l *curl*
lrwxr-xr-x  1 root  wheel      15 Oct 20  2014 libcurl.3.dylib -> libcurl.4.dylib
-rwxr-xr-x  1 root  wheel  781616 Jul 23  2015 libcurl.4.dylib
lrwxr-xr-x  1 root  wheel      15 Oct 20  2014 libcurl.dylib -> libcurl.4.dylib

which would work with the original code - perhaps the xwin7 curl build can be told to name it libcurl.dll
instead of libcurl-4.dll. That would be best for all.

@ccoupe ccoupe closed this Aug 11, 2017

@ccoupe ccoupe reopened this Aug 11, 2017

ccoupe added a commit that referenced this issue Aug 11, 2017

new branch 'curl' for #307
* libcurl rename in setup phase because ethon gem doesn't see
  the cross compiled libcurl-4.dll name.

@ccoupe ccoupe added the Normal label Aug 11, 2017

@ccoupe ccoupe added this to the 3.3.4 milestone Aug 11, 2017

@ccoupe ccoupe removed the Enhancement label Aug 11, 2017

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Aug 12, 2017

Contributor

Time for some benchmarks. On my cable modem, my website, 423KB file, (ymmv)

linux cmd line curl            - 478KB/s
shoes download              - 423KB/s
shoes with progress bar  -   12KB/s

Typheous won't be faster than bare curl and and has a noticeable FFI pause which would skew this test even more - it's a small file. Clearly the problem is the progress bar and not so much the native library. What about a larger file? 10.9MB

linux cmd line curl            - 18 seconds
shoes download              - 14 seconds 

Shoes Wins ! Beware of benchmarks! Yes, I did check twice. People complain about net/http or read complaints about net/http. This was Ruby 2.3.4. Perhaps they fixed net/http? Has anyone checked?

Why is download with progress bar so damn slow? I don't know, I can only guess. Switching the back end library may not be a useful path to pursue.

Contributor

ccoupe commented Aug 12, 2017

Time for some benchmarks. On my cable modem, my website, 423KB file, (ymmv)

linux cmd line curl            - 478KB/s
shoes download              - 423KB/s
shoes with progress bar  -   12KB/s

Typheous won't be faster than bare curl and and has a noticeable FFI pause which would skew this test even more - it's a small file. Clearly the problem is the progress bar and not so much the native library. What about a larger file? 10.9MB

linux cmd line curl            - 18 seconds
shoes download              - 14 seconds 

Shoes Wins ! Beware of benchmarks! Yes, I did check twice. People complain about net/http or read complaints about net/http. This was Ruby 2.3.4. Perhaps they fixed net/http? Has anyone checked?

Why is download with progress bar so damn slow? I don't know, I can only guess. Switching the back end library may not be a useful path to pursue.

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Aug 12, 2017

Collaborator

Why is download with progress bar so damn slow? I don't know, I can only guess. Switching the back end library may not be a useful path to pursue.

My understanding is that it creates a new thread and then Shoes juggles between Ruby green threads (for Shoes UI and code execution), GTK and the new download/progress thread. Unfortunately, GIL makes it so that only one thing at the time is executed.

Testing on a larger file resulted in freezing Shoes UI.

Collaborator

BackOrder commented Aug 12, 2017

Why is download with progress bar so damn slow? I don't know, I can only guess. Switching the back end library may not be a useful path to pursue.

My understanding is that it creates a new thread and then Shoes juggles between Ruby green threads (for Shoes UI and code execution), GTK and the new download/progress thread. Unfortunately, GIL makes it so that only one thing at the time is executed.

Testing on a larger file resulted in freezing Shoes UI.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Aug 13, 2017

Contributor

One mystery solved. The :pause arg is number of seconds to sleep between GUI updates. Removing that gets me 423KBs on the small file and 12 seconds on the big file. However, the gui flashes noticeably
A :pause => 0.02 works well - no flashing smooth animation and updates and 15 seconds. While ruby goes to sleep Gtk can keep up with its queue. Sadly this is platform specific tuning but tt can be done on any copy of Shoes you happen to have.

[update]
curl.exe on Windows (win7 vm) takes 8 to 13 seconds for the big file. (so the VM is negligible)
Shoes takes 243 seconds That is slow! . 110 seconds w/o a :pause => (no gui flashing either) which suggests something to me: The gtk loop/yield to ruby needs examination.

Contributor

ccoupe commented Aug 13, 2017

One mystery solved. The :pause arg is number of seconds to sleep between GUI updates. Removing that gets me 423KBs on the small file and 12 seconds on the big file. However, the gui flashes noticeably
A :pause => 0.02 works well - no flashing smooth animation and updates and 15 seconds. While ruby goes to sleep Gtk can keep up with its queue. Sadly this is platform specific tuning but tt can be done on any copy of Shoes you happen to have.

[update]
curl.exe on Windows (win7 vm) takes 8 to 13 seconds for the big file. (so the VM is negligible)
Shoes takes 243 seconds That is slow! . 110 seconds w/o a :pause => (no gui flashing either) which suggests something to me: The gtk loop/yield to ruby needs examination.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Aug 13, 2017

Contributor

Here's the script I'm using - in case I'm calculating transfer rates incorrectly or you want to compare.

Shoes.app do
  background "#eee"
  @list = stack do
    para "Enter a URL to download:", :margin => [10, 8, 10, 0]
    flow :margin => 10 do
      @url = edit_line :width => -120
      @url.text ='http://walkabout.mvmanila.com/public/share/Ytm-2.exe'
      st_time = 0
      end_time = 0
      totalsz = 0
      button "Download", :width => 120 do
        st_time = Time.now
        @list.append do
          stack do
            background "#eee".."#ccd"
            stack :margin => 10 do
              dld = nil
              para @url.text, " [", link("cancel") { dld.abort }, "]", :margin => 0
              d = inscription "Beginning transfer.", :margin => 0
              p = progress :width => 1.0, :height => 14
              dld = download @url.text, :save => File.basename(@url.text),
                :pause => 0.02,
                :progress => proc { |dl| 
                  d.text = "Transferred #{dl.transferred} of #{dl.length} bytes (#{dl.percent}%)"
                  p.fraction = dl.percent },
                :finish => proc { |dl|
                     end_time = Time.now
                     secs = (end_time.to_i - st_time.to_i)
                     kb = dl.length < 1024 ? 1: dl.length / 1024
                     d.text = "Download completed KB/s: #{kb/secs} #{secs}"
                  }
            end
          end
        end
      end
    end
  end
end
Contributor

ccoupe commented Aug 13, 2017

Here's the script I'm using - in case I'm calculating transfer rates incorrectly or you want to compare.

Shoes.app do
  background "#eee"
  @list = stack do
    para "Enter a URL to download:", :margin => [10, 8, 10, 0]
    flow :margin => 10 do
      @url = edit_line :width => -120
      @url.text ='http://walkabout.mvmanila.com/public/share/Ytm-2.exe'
      st_time = 0
      end_time = 0
      totalsz = 0
      button "Download", :width => 120 do
        st_time = Time.now
        @list.append do
          stack do
            background "#eee".."#ccd"
            stack :margin => 10 do
              dld = nil
              para @url.text, " [", link("cancel") { dld.abort }, "]", :margin => 0
              d = inscription "Beginning transfer.", :margin => 0
              p = progress :width => 1.0, :height => 14
              dld = download @url.text, :save => File.basename(@url.text),
                :pause => 0.02,
                :progress => proc { |dl| 
                  d.text = "Transferred #{dl.transferred} of #{dl.length} bytes (#{dl.percent}%)"
                  p.fraction = dl.percent },
                :finish => proc { |dl|
                     end_time = Time.now
                     secs = (end_time.to_i - st_time.to_i)
                     kb = dl.length < 1024 ? 1: dl.length / 1024
                     d.text = "Download completed KB/s: #{kb/secs} #{secs}"
                  }
            end
          end
        end
      end
    end
  end
end
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Aug 13, 2017

Contributor

I've got the Shoes Windows download to 90 secs (not as pathetic) by dropping the (Windows) gtk_idle timeout from 100ms to 5ms - see shoes/native/gtk.c(shoes_native_loop(). Much of Shoes/Windows seems snappier but I know enough to know I could be biased. The file open dialog (cobbler->Install Gempack->load) where it creates a file open dialog really does seem faster. Setting the idle to 1ms does not hurt anything but not real improvements.

This is a gtk timer that fires every given X ms and tells ruby to run it's threads - Linux gtk polls fd's which shoes/Windows doesn't do - unreliable if memory serves but that was back in Ruby 1.9.3 days.

Contributor

ccoupe commented Aug 13, 2017

I've got the Shoes Windows download to 90 secs (not as pathetic) by dropping the (Windows) gtk_idle timeout from 100ms to 5ms - see shoes/native/gtk.c(shoes_native_loop(). Much of Shoes/Windows seems snappier but I know enough to know I could be biased. The file open dialog (cobbler->Install Gempack->load) where it creates a file open dialog really does seem faster. Setting the idle to 1ms does not hurt anything but not real improvements.

This is a gtk timer that fires every given X ms and tells ruby to run it's threads - Linux gtk polls fd's which shoes/Windows doesn't do - unreliable if memory serves but that was back in Ruby 1.9.3 days.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Aug 14, 2017

Contributor

Testing on a larger file resulted in freezing Shoes UI.

I'm attempting to modify /lib/shoes/download.rb with typhoeus and I can see how that happens in the old code. The old code keep everything download in memory and on saved it to a file at the end of the transfer so anything larger than your physical memory is going to start paging into virtual memory and only gets worse.

Contributor

ccoupe commented Aug 14, 2017

Testing on a larger file resulted in freezing Shoes UI.

I'm attempting to modify /lib/shoes/download.rb with typhoeus and I can see how that happens in the old code. The old code keep everything download in memory and on saved it to a file at the end of the transfer so anything larger than your physical memory is going to start paging into virtual memory and only gets worse.

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Aug 14, 2017

Collaborator

You might want to test with something like an ISO file (Linux, FreeBSD, etc). Something at around 1 to 4 Gb would be a fairly good stress test.

https://download.freebsd.org/ftp/releases/amd64/amd64/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-dvd1.iso

Collaborator

BackOrder commented Aug 14, 2017

You might want to test with something like an ISO file (Linux, FreeBSD, etc). Something at around 1 to 4 Gb would be a fairly good stress test.

https://download.freebsd.org/ftp/releases/amd64/amd64/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-dvd1.iso

@ccoupe ccoupe added the Windows label Aug 14, 2017

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Aug 15, 2017

Contributor

I've got a version of lib/shoes/download.rb using Typhoeus and it's performance is less than stellar on Windows. 123KB/s and 91 secs to to download the 10.9Mb file. 4 or 5 times slower than curl. It's better that we started with so that's worth something. No need for ethon mods or typhoeus mods. I do like the code simplicity of this version of download.rb (and it fixes some bugs - the large .iso) so I tried it on Linux minlin - 700KB/s and 16 secs - Acceptable on Linux.

My understanding is that it creates a new thread and then Shoes juggles between Ruby green threads (for Shoes UI and code execution), GTK and the new download/progress thread.

Ruby hasn't used green threads since 1.9.x. Now days It Uses pthreads.- Native threads on Linux and OSX. Gtk uses pthreads (if built to do so and I try to do so). Ruby does have a thread pool (The GIL) that it manages which should not effect other pthreads (like GTK or Cocoa). Windows uses a pthreads emulator which is a dll (winpthreads.dll) included with shoes. Clearly there are still performance issues with Windows and Ruby - not my wheel house to fix.

As we might remember, Windows has always had an event loop performance problem - back to the days of Shoes Raisins/Policeman when one thread sucked up 100%. The change to Gtk for Windows helped with that problem but I suspect it's not optimal when it deals with socket IO - it uses a different method of scheduling for Windows vs Linux.

Contributor

ccoupe commented Aug 15, 2017

I've got a version of lib/shoes/download.rb using Typhoeus and it's performance is less than stellar on Windows. 123KB/s and 91 secs to to download the 10.9Mb file. 4 or 5 times slower than curl. It's better that we started with so that's worth something. No need for ethon mods or typhoeus mods. I do like the code simplicity of this version of download.rb (and it fixes some bugs - the large .iso) so I tried it on Linux minlin - 700KB/s and 16 secs - Acceptable on Linux.

My understanding is that it creates a new thread and then Shoes juggles between Ruby green threads (for Shoes UI and code execution), GTK and the new download/progress thread.

Ruby hasn't used green threads since 1.9.x. Now days It Uses pthreads.- Native threads on Linux and OSX. Gtk uses pthreads (if built to do so and I try to do so). Ruby does have a thread pool (The GIL) that it manages which should not effect other pthreads (like GTK or Cocoa). Windows uses a pthreads emulator which is a dll (winpthreads.dll) included with shoes. Clearly there are still performance issues with Windows and Ruby - not my wheel house to fix.

As we might remember, Windows has always had an event loop performance problem - back to the days of Shoes Raisins/Policeman when one thread sucked up 100%. The change to Gtk for Windows helped with that problem but I suspect it's not optimal when it deals with socket IO - it uses a different method of scheduling for Windows vs Linux.

ccoupe added a commit that referenced this issue Aug 15, 2017

for #307 - branch curl typhoeus can be used for download
* Performance is slightly better on Windows. No big deal on Linux.
* fixes an oversight - don't store the whole download in memory if
  save: <file> is given.
* much useless code in the download.rb

ccoupe added a commit that referenced this issue Aug 16, 2017

for #307 - add download test scripts
* not even close to the api in the manual - since shoes 3.2.0 (m3.rb)
* beware - m4.rb segfaults. gdb works.

ccoupe added a commit that referenced this issue Aug 16, 2017

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Aug 16, 2017

Contributor

The can of worms is fully open now! The old ruby, download.rb replacement was not compliant with the 3.0 download api as documented in the manual. There is a segfault in rbload.c for one on of the manual samples (m4) - that will be easier to fix and test than the speed problem with Windows I May have to revisit the select/poll of fd. Even with the new code/rake refactorings, that's a big pain to test since it affects everything.

Contributor

ccoupe commented Aug 16, 2017

The can of worms is fully open now! The old ruby, download.rb replacement was not compliant with the 3.0 download api as documented in the manual. There is a segfault in rbload.c for one on of the manual samples (m4) - that will be easier to fix and test than the speed problem with Windows I May have to revisit the select/poll of fd. Even with the new code/rake refactorings, that's a big pain to test since it affects everything.

ccoupe added a commit that referenced this issue Aug 19, 2017

for #307 branch curl -
* minlin is good - image and download
* not so good for other targets.

ccoupe added a commit that referenced this issue Aug 20, 2017

Branch curl, #307 - copy libcurl.dylib at build time.
* osx does not work on this branch.

ccoupe added a commit that referenced this issue Aug 22, 2017

branch rb23 - the one with/out libcurl - #307 is involved
* hack to fix image and download (open_uri) to use a
  specified ca certs file.
* does not fix OSX or Windows openssl and ssl certs - just hides it
* different solutions required to get openssl really working
  on osx and windows. Which would not be need if Tyhpoeus was adopted
  and working - that's being a realy PITA.

ccoupe added a commit that referenced this issue Aug 23, 2017

this hack gets https working for https #307 (partial)
* windows and osx need this - inspired by
  https://www.theguild.nl/ruby-and-ssl-certificate-validation/
* doesn't really fix the ssl issues in the ruby in Shoes windows
  and osx - that's a deps issue and much deeper
* also includes the libcurl shared library and the ethon and
  Typhoeus gems - it doesn't use them (but you can try them)
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Aug 24, 2017

Contributor

There's a Shoes 3.3.4 beta

Contributor

ccoupe commented Aug 24, 2017

There's a Shoes 3.3.4 beta

ccoupe added a commit that referenced this issue Aug 24, 2017

issue #307 - branch rb23
* include some fixes from branch curl make typhoeus available when
  require 'typhoeus'
* improves Windows performance on this task, and probably many others.

ccoupe added a commit that referenced this issue Aug 27, 2017

on #307, brach curl, target pi2 - copy the damn libcurl
* still fails to load typhoeus during ruby init. Very confusing.
  because x86_64_linux and xwin7 work.

ccoupe added a commit that referenced this issue Oct 24, 2017

for #377, #323 cache and no-cache are working, file and http
* using typhoenus would totally rewrite this rat maze. #307
* Did I mention this is ugly?

@ccoupe ccoupe modified the milestones: 3.3.4, 3.3.5 Nov 2, 2017

@dredknight dredknight added this to To Do in Performance issues Dec 31, 2017

@ccoupe ccoupe modified the milestones: 3.3.5, 3.3.6 Jan 13, 2018

@ccoupe ccoupe modified the milestones: 3.3.6, 3.3.7 Mar 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment