How to use Papermill operator directly with S3 for output?

[ahsanshah]

Hi,

I am using the airflow PaperMillOperator but I am unclear how to configure the "output_nb". Ideally I would like to directly upload to S3 as follows:

        input_nb=client.get_object("papermill","papermill_helloworld.ipynb"),
        output_nb=client.fput_object("papermill","/tmp/out-{{ execution_date }}-testout.ipynb",content_type="application/JSON",)

This does not work as it is expecting a file_path output for S3.

I also tried the standard method of saving to a file:

        input_nb=client.get_object("papermill","papermill_helloworld.ipynb"),
        output_nb="/tmp/testout.ipynb"

However, it cannot find such a file:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/testout.ipynb'

My question(s) are:

1. Can I not just direct upload to S3 via the papermill operator?
2. If no, any idea how to save/upload outside the Operator

Note that this is Airflow 2.0 on K8s and I am using a VIrtualEnv within the DAG. Thank you.

[potiuk]

I belive PaperMill has no S3 integration. That would be a new feture and t if you want to add it, feel free. Airflow is created by > 2200 contributors like you who missed some feature and added it as a contribution, giving back for the free software they got. Extending and building own operators is a first class citizen, so you can even extend PaperMill operator in the way you want without contributing it back, though it would be cool if you do contribute it back. You can use S3Hook to get the authorisation information for S3 for example (you can look at the code of other operators - some google operatos have some S3 integration for one)