HADOOP-17454. [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0

[bgaborg]

Also fixes HADOOP-16995. ITestS3AConfiguration proxy tests failures when bucket probes == 0
The improvement should include the fix, ebcause the test would fail by default otherwise.

Change-Id: I9a7e4b5e6d4391ebba096c15e84461c038a2ec59

[bgaborg]

tested against ireland.

Testing: I had one failing test during the CLI run, but it was ok after a rerun:

[ERROR]   ITestS3ADeleteCost.testDeleteSingleFileInDir:110->AbstractS3ACostTest.verifyMetrics:360->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 operation returning after fs.delete(simpleFile) action_executor_acquired starting=0 current=0 diff=0, action_http_get_request starting=0 current=0 diff=0, action_http_head_request starting=4 current=5 diff=1, committer_bytes_committed starting=0 current=0 diff=0, committer_bytes_uploaded starting=0 current=0 diff=0, committer_commit_job starting=0 current=0 diff=0, committer_commits.failures starting=0 current=0 diff=0, committer_commits_aborted starting=0 current=0 diff=0, committer_commits_completed starting=0 current=0 diff=0, committer_commits_created starting=0 current=0 diff=0, committer_commits_reverted starting=0 current=0 diff=0, committer_jobs_completed starting=0 current=0 diff=0, committer_jobs_failed starting=0 current=0 diff=0, committer_magic_files_created starting=0 current=0 diff=0, committer_materialize_file starting=0 current=0 diff=0, committer_stage_file_upload starting=0 current=0 diff=0, committer_tasks_completed starting=0 current=0 diff=0, committer_tasks_failed starting=0 current=0 diff=0, delegation_token_issued starting=0 current=0 diff=0, directories_created starting=2 current=3 diff=1, directories_deleted starting=0 current=0 diff=0, fake_directories_created starting=0 current=0 diff=0, fake_directories_deleted starting=6 current=8 diff=2, files_copied starting=0 current=0 diff=0, files_copied_bytes starting=0 current=0 diff=0, files_created starting=1 current=1 diff=0, files_delete_rejected starting=0 current=0 diff=0, files_deleted starting=0 current=1 diff=1, ignored_errors starting=0 current=0 diff=0, multipart_instantiated starting=0 current=0 diff=0, multipart_upload_abort_under_path_invoked starting=0 current=0 diff=0, multipart_upload_aborted starting=0 current=0 diff=0, multipart_upload_completed starting=0 current=0 diff=0, multipart_upload_part_put starting=0 current=0 diff=0, multipart_upload_part_put_bytes starting=0 current=0 diff=0, multipart_upload_started starting=0 current=0 diff=0, object_bulk_delete_request starting=3 current=4 diff=1, object_continue_list_request starting=0 current=0 diff=0, object_copy_requests starting=0 current=0 diff=0, object_delete_objects starting=6 current=9 diff=3, object_delete_request starting=0 current=1 diff=1, object_list_request starting=5 current=6 diff=1, object_metadata_request starting=4 current=5 diff=1, object_multipart_aborted starting=0 current=0 diff=0, object_multipart_initiated starting=0 current=0 diff=0, object_put_bytes starting=0 current=0 diff=0, object_put_request starting=3 current=4 diff=1, object_put_request_completed starting=3 current=4 diff=1, object_select_requests starting=0 current=0 diff=0, op_copy_from_local_file starting=0 current=0 diff=0, op_create starting=1 current=1 diff=0, op_create_non_recursive starting=0 current=0 diff=0, op_delete starting=0 current=1 diff=1, op_exists starting=0 current=0 diff=0, op_get_delegation_token starting=0 current=0 diff=0, op_get_file_checksum starting=0 current=0 diff=0, op_get_file_status starting=2 current=2 diff=0, op_glob_status starting=0 current=0 diff=0, op_is_directory starting=0 current=0 diff=0, op_is_file starting=0 current=0 diff=0, op_list_files starting=0 current=0 diff=0, op_list_located_status starting=0 current=0 diff=0, op_list_status starting=0 current=0 diff=0, op_mkdirs starting=2 current=2 diff=0, op_open starting=0 current=0 diff=0, op_rename starting=0 current=0 diff=0, s3guard_metadatastore_authoritative_directories_updated starting=0 current=0 diff=0, s3guard_metadatastore_initialization starting=0 current=0 diff=0, s3guard_metadatastore_put_path_request starting=0 current=0 diff=0, s3guard_metadatastore_record_deletes starting=0 current=0 diff=0, s3guard_metadatastore_record_reads starting=0 current=0 diff=0, s3guard_metadatastore_record_writes starting=0 current=0 diff=0, s3guard_metadatastore_retry starting=0 current=0 diff=0, s3guard_metadatastore_throttled starting=0 current=0 diff=0, store_io_request starting=0 current=0 diff=0, store_io_retry starting=0 current=0 diff=0, store_io_throttled starting=0 current=0 diff=0, stream_aborted starting=0 current=0 diff=0, stream_read_bytes starting=0 current=0 diff=0, stream_read_bytes_backwards_on_seek starting=0 current=0 diff=0, stream_read_bytes_discarded_in_abort starting=0 current=0 diff=0, stream_read_bytes_discarded_in_close starting=0 current=0 diff=0, stream_read_close_operations starting=0 current=0 diff=0, stream_read_closed starting=0 current=0 diff=0, stream_read_exceptions starting=0 current=0 diff=0, stream_read_fully_operations starting=0 current=0 diff=0, stream_read_opened starting=0 current=0 diff=0, stream_read_operations starting=0 current=0 diff=0, stream_read_operations_incomplete starting=0 current=0 diff=0, stream_read_seek_backward_operations starting=0 current=0 diff=0, stream_read_seek_bytes_discarded starting=0 current=0 diff=0, stream_read_seek_bytes_skipped starting=0 current=0 diff=0, stream_read_seek_forward_operations starting=0 current=0 diff=0, stream_read_seek_operations starting=0 current=0 diff=0, stream_read_seek_policy_changed starting=0 current=0 diff=0, stream_read_total_bytes starting=0 current=0 diff=0, stream_read_version_mismatches starting=0 current=0 diff=0, stream_write_block_uploads starting=0 current=0 diff=0, stream_write_block_uploads_aborted starting=0 current=0 diff=0, stream_write_block_uploads_committed starting=0 current=0 diff=0, stream_write_bytes starting=0 current=0 diff=0, stream_write_exceptions starting=0 current=0 diff=0, stream_write_exceptions_completing_upload starting=0 current=0 diff=0, stream_write_queue_duration starting=0 current=0 diff=0, stream_write_total_data starting=0 current=0 diff=0, stream_write_total_time starting=0 current=0 diff=0: object_delete_objects expected:<2> but was:<3>

does not seem related

[steveloughran]

+1 pending successful Yetus build. I've turned off existence checks for a long time and makes things faster, especially creation of a new FS instance in test suites.

Do update the release notes though, to say "you'll get told of missing/unreadable bucket on first operation against it".

Test failure is going to be related to directory marker retention and/or metric counting.

Could you file a JIRA, assign to me and include the specific test settings you were using?

[bgaborg]

thanks Steve. created https://issues.apache.org/jira/browse/HADOOP-17455
please check the release notes for this issue on jira- I think I filled it the right way

[mukund-thakur]

We changed the default behaviour of bucket probe from 2 to 0 which was present from long time. Won't this cause problem for the downstream components which used to expect this exception ?

Good that we have added the release notes. Thanks Gabor.