airbyte-workers: add /tmp emptyDir volume to connector pods

[tbcdns]

What

Some connectors (such as destination-s3) require to write some temporary data (generally to /tmp).
It is a good security practice to enforce read only root filesystem on Kubernetes pod, and, some productive Kubernetes clusters enforce that all pods run with read only root filesystem.
Therefore, in order to still allow connectors to write temporary data to /tmp with read only root fs, we must mount an emptyDir volume to /tmp.

The original PR was here: #9874 we decided to split it into 3 different PRs.

How

Simply creating a new emptyDir volume and mount it under /tmp.

🚨 User Impact 🚨

No user impact

[marcosmarxm]

Thanks @tbcdns

[davinchia]

We are going to hold off on merging this first as we have some internal discussions to do. FYI @sherifnada

[davinchia]

@tbcdns, after discussing with @ChristopheDuong, we agree this is useful and we want to merge this in.

let us run some test in the mean time before doing so. should merge this in the next couple of days.

were you able to successfully run a destination-s3 sync using this deployment?

[davinchia]

Where are you seeing that destination-s3 uses a local file? Do you mean destination-snowflake?

[tbcdns]

@davinchia , I faced it with destination-s3, using Parquet as data format.
I don't know all the details, but Parquet (and / or the S3 client) is using some hadoop libraries where the default hadoop tmp dir is /tmp/hadoop-${user.name}
My understanding is that some temporary data needs to be written to the hadoop tmp directory. If /tmp is not writable, I was facing some weird and not so obvious hadoop stack trace.

But, I can confirm that I was able to run a destination-s3 sync using this deployment.

[jrhizor]

@ChristopheDuong @davinchia Outside of this PR, should we be setting a maximum size limit for the tmp volume as part of the resource requirements? Maybe this is only worth doing in the future with ephemeral storage, but I am a bit worried about adding/encouraging support for disk space usage on nodes without limiting the size.

[davinchia]

@tbcdns thanks!

@jrhizor that's a good call. I'll create an issue so we don't forget. We can decide on this limit once @ChristopheDuong and the connector teams have decided on disk bugger limits.

[davinchia]

Issue created here: https://github.com/airbytehq/airbyte-internal-issues/issues/478