S3FSCheckSuccess fails when __HIVE_DEFAULT_PARTITION__ exists

[acmiyaguchi]

telemetry-airflow/plugins/s3fs_check_success.py

Lines 9 to 19 in 9c14df6

	def check_s3fs_success(bucket, prefix, num_partitions):
	"""Check the s3 filesystem for the existence of `_SUCCESS` files in dataset partitions.
	:prefix: Bucket prefix of the table
	:num_partitions: Number of expected partitions
	"""
	s3 = boto3.resource("s3")
	objects = s3.Bucket(bucket).objects.filter(Prefix=prefix)
	success = set([obj.key for obj in objects if "_SUCCESS" in obj.key])
	return len(success) == num_partitions

This fails when there is a __HIVE_DEFAULT_PARTITION__ in the dataset.

$aws s3 ls s3://telemetry-parquet/main_summary/v4/submission_date_s3=20190105/ --recursive | grep _SUCCESS | wc -l

101

This should be modified so the number of partitions is the minimum required _SUCCESS partitions to be written, or an explicit list of partitions.