Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory usage #23

Closed
afarbos opened this issue Jun 12, 2015 · 1 comment
Closed

Memory usage #23

afarbos opened this issue Jun 12, 2015 · 1 comment

Comments

@afarbos
Copy link

afarbos commented Jun 12, 2015

I think is quite linked with one of my previous questions, but just to be sure. Will you look at the memory usage of each tasks and output an awesome graph or things like that?

That could be very cool, I did not see any scheduler with a good ui doing that.

@mistercrunch
Copy link
Member

That'd be handled by the executor. Once we'll be on Yarn we'll get all the goodies that come with it. If someone was to write and use a MesosExecutor, they'd get whatever ships or can be plugged into Mesos.

With Celery, we get some good stuff with "Celery Flower". Every environment also has some kind of ops system monitoring daemon, at Airbnb we use statsd and Datadog, we can do box level monitoring with that.

There's minimal integration with statsd within Airflow at the moment, but we could instrument more things using this, but not everyone has statsd in their stack.

Keep in mind that Airflow's scope is to mostly be an orchestrator, it tells other systems what to do. If you're planning on doing some large distributed computation that uses lots of memory, Airflow might not be the right platform. Instead, you may want to use a distributed system, or write some sort of service, and invoke it from Airflow.

whynick1 pushed a commit to whynick1/incubator-airflow that referenced this issue Mar 31, 2020
…-wepay

Squashed commit of the following:

commit d7ca49193f291a1b52540ce436412a6749a94e23
Author: Nikolay Stoynov <nikolays@wepay.com>
Date:   Wed Nov 6 10:35:46 2019 -0800

    Applied [AIRFLOW-5553]
mobuchowski pushed a commit to mobuchowski/airflow that referenced this issue Jan 4, 2022
mobuchowski pushed a commit to mobuchowski/airflow that referenced this issue Jan 4, 2022
* add input and output facets
* allow building from current branch
* enable generating from local
Signed-off-by: Julien Le Dem <julien@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants