"mb scan" may hang, because rsync's "--timeout" option is not

[poeml]

                                                                                                                                 [          ]

Issue migrated (2015-06-05) from old issue tracker http://mirrorbrain.org/issues/issue12

Title    "mb scan" may hang, because rsync's "--timeout" option is not
             effective at all times
 Priority  bug                       Status            resolved
Superseder                          Nosy List          poeml
 Assigned  poeml                    Keywords           scanner
      To

msg31 (view) Author: poeml Date: 2009-10-08.11:50:06

rsync should never hang longer than 20 seconds when we probe for files -- since we run it with --timeout=20.

But this timeout doesn't seem to apply to the connect phase of rsync:

  \_ /usr/bin/python /usr/bin/mb scan -j 8 -a -d distribution/11.2-Milestone7/iso
            |   \_ sh -c { rsync -d --timeout=20 rsync://ftp.novell.co.jp/opensuse/distribution/11.2-Milestone7/iso

/tmp/mb_probefile_PChalR/ --list-only; } 2>&1
| _ rsync -d --timeout=20 rsync://ftp.novell.co.jp/opensuse/distribution/11.2-Milestone7/iso
/tmp/mb_probefile_PChalR/ --list-only

strace shows:

  connect(3, {sa_family=AF_INET6, sin6_port=htons(873), inet_pton(AF_INET6, "2001:278:101f:1::2", &sin6_addr), sin6_flowinfo=0,

sin6_scope_id=0}, 28

msg32 (view) Author: poeml Date: 2009-10-08.11:50:55

The impact is that this may cause the whole "mb scan" command to hang in the
initial phase, where probing for directories is done.

msg105 (view) Author: poeml Date: 2009-12-11.21:16:57

Incidentally, I discovered in the rsync man page that there's an additional timeout setting:

--contimeout=SECONDS set daemon connection timeout in seconds

...exactly what we need.

It was new in rsync 3.0.0.

Thus, we can use it, except on platforms that still have rsync 2.6. Debian 5.0 is such a case.

So it could be worthwhile to detect the rsync version in the beginning of a scan. The detection can be used
to also make sure that rsync is installed at all, and give a meaningful error message if it isn't.

% python -c "import commands; status, output = commands.getstatusoutput('rsync --version'); print status;
ver = output.splitlines()[0].split()[2]; print ver"
0
3.0.4

Something along these lines needs to be added to mb/testmirror.py.

msg129 (view) Author: poeml Date: 2010-02-10.04:31:22

I committed a fix to trunk (r7953).

http://svn.mirrorbrain.org/viewvc/mirrorbrain?view=revision&revision=7953

History
         Date         User  Action              Args
2010-02-10 04:41:58 poeml set    status: testing -> resolved
2010-02-10 04:31:22 poeml set    status: in-progress -> testing
                                   messages: + msg129
2009-12-11 21:16:58 poeml set    status: chatting -> in-progress
                                   messages: + msg105
2009-12-01 20:55:44 poeml set    keyword: + scanner
2009-10-08 11:50:55 poeml set    status: unread -> chatting
                                   messages: + msg32
2009-10-08 11:50:06 poeml create

(end of migrated issue)