When using ido.el, find-file with tramp make directory and file cache into /.ido.last.
If some directory on remote server (i.e. /ssh:user@host:/foo) was saved in the file,
emacs always requires SSH password when start up to connect remote server.
Maybe ido.el has some function or variable to disable directory caching.