Loading history is slow => config option to limit number of lines? #759

Open
balinterdi opened this Issue Nov 22, 2012 · 11 comments

Comments

Projects
None yet
4 participants

I've just looked into why launching a rails console takes ~60 secs in a mid-sized Rails project and found the culprit was Pry loading its history.

Turned out my history file had ~17.000 lines and took about 40-50 seconds to load. This seems slow at first although I did not really dig into this. More importantly, I don't think a history file should keep as many elements by default as users will surely get bitten by this.

I think Pry.config.history.max_lines should have a sane default (e.g 100) and it should be configurable. I'll try to find the time to get a stab at this.

Owner

kyrylo commented Nov 22, 2012

~[master]% wc -l ~/.pry_history
65508 /home/curacao/.pry_history
~[master]% time pry -e 'exit'
pry -e 'exit'  1.50s user 0.14s system 96% cpu 1.690 total

Please, note that I have many plugins installed.

Owner

kyrylo commented Nov 22, 2012

Oh, and what is the version of your Pry? The latest available is 0.9.10. You can check your version from command line (pry -v) or directly within Pry (with help of pry-version command).

Owner

banister commented Nov 22, 2012

@balinterdi my history file is enormous and loads in no time, modern pry versions should load history quickly, very old versions of pry had a problem loading large files and were slow.

Interesting. I also have 0.9.10 installed and yet experienced the slow loading.

» bundle exec pry -v                                                                                                                                                    
Pry version 0.9.10 on Ruby 1.9.3
@ghost

ghost commented Nov 22, 2012

➜  ~P git:(master) ✗ cat ~/.pry_history | wc -l
    6676
➜  ~P git:(master) ✗ time pry -e 'exit'
pry -e 'exit'  0.31s user 0.09s system 68% cpu 0.581 total
➜  ~P git:(master) ✗ pry -v
Pry version 0.9.10 on Ruby 2.0.0➜  ~P git:(master) ✗ 

Maybe the bottleneck is somewhere else?

@ghost

ghost commented Nov 22, 2012

I still think it is a good idea to have a cutoff number, I rarely use all of the history available to me but I might use 10-20 lines of history. 100 is large(relatively speaking), and loads fast.

Owner

kyrylo commented Nov 23, 2012

@balinterdi, how did you determine it was Pry history that caused such slowness?

I profiled the below code by inserting print statements (ok, puts statements :) ) between each of the lines with a timestamp affiexed:

  def self.initial_session_setup

    return if !initial_session?

    # note these have to be loaded here rather than in pry_instance as
    # we only want them loaded once per entire Pry lifetime.
    load_rc if Pry.config.should_load_rc
    load_local_rc if Pry.config.should_load_local_rc
    load_plugins if Pry.config.should_load_plugins
    load_requires if Pry.config.should_load_requires
    load_history if Pry.config.history.should_load
    load_traps if Pry.config.should_trap_interrupts

    @initial_session = false
  end

and found that load_history took 40-50 seconds to load. Deleting ~/.pry_history and relaunching rails console brought it down to 8 secs (from the 50-60 seconds experienced before).

After some more poor man's profiling I've found that it is pushing to the Readline::HISTORY that is slow:

In history.rb:

    def load
      @loader.call do |line|
        @pusher.call(line.chomp)
        @history << line.chomp
      end
      @saved_lines = @original_lines = @history.length
    end

If I comment out the @pusher.call line then history is loaded in no time.

Readline::History is just an Enumerable object so I don't see any reason why pushing to it should be slow.

That said, is there a reason Pry loads the history lines one-by-one? Maybe we could cut down on startup time by loading all lines at once.

Regarding why others don't see the slowness I do, I suspect I might have an issue with the readline lib but I'm not sure how that works.

Owner

rf- commented Nov 23, 2012

Readline::HISTORY actually has some strange properties. For example, calling to_a on it when it gets big is very expensive (which is why we maintain a separate @history array). I've never seen pushing to it be slow, though, so this is an interesting problem.

@rf- rf- closed this Nov 23, 2012

@rf- rf- reopened this Nov 23, 2012

@rf- rf- modified the milestone: v1.0.0 Apr 29, 2014

@rf- rf- added the feature label Apr 29, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment