What happens if I download more files while fscrawler is indexing?

[ScottCov]

on one server I do webscraping, downloading and on the other I am indexing the pdf files with fscrawler. I transfer files to the indexing server using rsync. At times there can be thousands of files that get transferred at once and it can take several hours. I touched all files on the first server immediately before my transfer but only a subset of the files ended up getting indexed. I am guessing that is because when fscrawler started running it queued up the files that had been transfered so far, indexed them and then searched for new files 15 min later however, since the files downloaded after fscrawler started indexing had the same time stamp as the ones it had already indexed it assumed they were old files and didn't index them? Just to be clear....all the files transferred with rsync had the same time stamp