bug: New BATCH functionality fails if output folder is missing

[kgpayne]

Singer SDK Version

0.11.1

Python Version

3.10

Bug scope

Taps (catalog, state, stream maps, etc.)

Operating System

Linux

Description

Currently, if the folder specified as storage.root in batch_config does not exist, the tap will run and fail on writing the first batch. Preferably we could create this with Path.mkdir(parents=True, exists_ok=True) but at a minimum we should check it exists before fetching records in order to 'fail fast'.

Possible solutions:

1. Create the root if not exists before fetching any records, and fail if creation fails.
2. Check if root exists before fetching any records and fail if it doesn't (i.e. do not create).

Code

https://github.com/meltano/squared/actions/runs/3152108087/jobs/5126999218#step:8:149

[edgarrmondragon]

This might be as simple as using the keyword argument create=True when calling

sdk/singer_sdk/helpers/_batch.py

Lines 157 to 169 in 721b6f9

	@contextmanager
	def fs(self, **kwargs: Any) -> Generator[FS, None, None]:
	"""Get a filesystem object for the storage target.
	Args:
	kwargs: Additional arguments to pass ``f`.open_fs``.
	Returns:
	The filesystem object.
	"""
	filesystem = fs.open_fs(self.fs_url.geturl(), **kwargs)
	yield filesystem
	filesystem.close()

All the keyword arguments are passed to https://docs.pyfilesystem.org/en/latest/reference/opener.html#fs.opener.registry.Registry.open_fs