Bulk imports failed with agitation.

[keith-turner]

After running another bulk import+agitation test 5 out of 1000 bulk imports failed with the following messages in the shell.

2019-07-16 21:18:30,537 [shell.Shell] ERROR: org.apache.accumulo.core.client.AccumuloException: hdfs://leader1:8020/tmp/bt/36/files: java.io.IOException: Filesystem closed
2019-07-16 23:39:50,466 [shell.Shell] ERROR: org.apache.accumulo.core.client.AccumuloException: hdfs://leader1:8020/tmp/bt/279/files: java.io.IOException: Filesystem closed
2019-07-17 02:52:02,848 [shell.Shell] ERROR: org.apache.accumulo.core.client.AccumuloException: hdfs://leader1:8020/tmp/bt/516/files: java.io.IOException: Filesystem closed
2019-07-17 03:04:18,888 [shell.Shell] ERROR: org.apache.accumulo.core.client.AccumuloException: hdfs://leader1:8020/tmp/bt/530/files: java.io.IOException: Filesystem closed
2019-07-17 10:05:27,626 [shell.Shell] ERROR: org.apache.accumulo.core.client.AccumuloException: hdfs://leader1:8020/tmp/bt/903/files: java.io.IOException: Filesystem closed

Unfortunately debug logging was not enabled so some critical information needed from the master logs was missing. However there was enough info to narrow this down to where I think the problem is the BulkImportMove Repo code that reads the rename mapping. I think this code encountered a hadoop file system closed exception that was not handled by the changes in #1245

Investigation revealed :

• No files had been moved for any of the dirs
• No blips were created
• In the master logs all of these operations failed immediately before a master restart.
• There were 103 cases of filesystem closed exception that were handled by the changes in fix #1231 prevent hadoop shutdown hook from failing FATE tx #1245