New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory usage #23
Comments
That'd be handled by the executor. Once we'll be on Yarn we'll get all the goodies that come with it. If someone was to write and use a MesosExecutor, they'd get whatever ships or can be plugged into Mesos. With Celery, we get some good stuff with "Celery Flower". Every environment also has some kind of ops system monitoring daemon, at Airbnb we use statsd and Datadog, we can do box level monitoring with that. There's minimal integration with statsd within Airflow at the moment, but we could instrument more things using this, but not everyone has statsd in their stack. Keep in mind that Airflow's scope is to mostly be an orchestrator, it tells other systems what to do. If you're planning on doing some large distributed computation that uses lots of memory, Airflow might not be the right platform. Instead, you may want to use a distributed system, or write some sort of service, and invoke it from Airflow. |
…-wepay Squashed commit of the following: commit d7ca49193f291a1b52540ce436412a6749a94e23 Author: Nikolay Stoynov <nikolays@wepay.com> Date: Wed Nov 6 10:35:46 2019 -0800 Applied [AIRFLOW-5553]
* add input and output facets * allow building from current branch * enable generating from local Signed-off-by: Julien Le Dem <julien@apache.org>
I think is quite linked with one of my previous questions, but just to be sure. Will you look at the memory usage of each tasks and output an awesome graph or things like that?
That could be very cool, I did not see any scheduler with a good ui doing that.
The text was updated successfully, but these errors were encountered: