-
-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Singularity Operator with Airflow #178
Comments
I'm not an expert with airflow, but I'd try for options having that be a list instead of a single string in a list. |
@vsoch Unfortunately, that didn't fix it. I tried |
I would next try testing locally with just Singularity Python and see if it works (outside of airflow). If it does, then it's something about the implementation in airflow that is not passing the option forward. |
@vsoch I think I found the culprit, or at least a clue. In the airflow plugin, the client is started outside the creation of the instance: https://github.com/apache/airflow/blob/b0b25910713dd39e0193bdcd95b2cfd9e3fed5e7/airflow/providers/singularity/operators/singularity.py#L150 This appears to not transfer over the options. I tested this outside of airflow by:
Produces
|
@SiftingSands I think you are right! The issue is that options and kwargs that are provided to the instance init are only honored given that start is run - since the airflow code doesn't run start at init, we run it later, and those options/args are saved but not used. I think they should be honored (and the bug is here with spython) so I opened a PR to try and fix the problem: Would you be able to test with airflow to see if the options are carried through? |
Confirming that #179 fixes the aforementioned issues with the Singularity plugin for Apache Airflow. Logs from the tasks in airflow indicate that options are now successfully passed over to instances at run-time. I haven't exhaustively tested every option, but I don't think that's necessary given your diagnosis. Thanks for the rapid response! |
Great! Let's get that PR merged and released. |
I'm putting this question here because the Singularity operator for Apache Airflow uses spython - didn't think I would get the attention on the airflow repo.
I followed the example DAG https://github.com/apache/airflow/tree/main/airflow/providers/singularity/example_dags and slightly modified the operator :
The container runs without an issue but the logs show the command operating on '/' instead of '/home/' corresponding to the defined option. I also observe similar behavior with other options that don't seem to translate over the the airflow runs (such as binds, --no-nv, --no-home). These options work fine outside of airflow as expected, so I don't think it's my singularity build. I'm probably missing something obvious, but I couldn't figure it out after a serious debugging effort.
Thanks!
The text was updated successfully, but these errors were encountered: