GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
Already on GitHub? Sign in to your account
Pre-outputs aren't deleted automatically anymore since we introduced the DAG capabilities for job flows. The reason for this is that
backend = get_backend(opts)
fs = backend.create_filesystem(opts)
gets executed at a point where 'opts' doesn't contain the 'hadoop' option anymore (since this option gets removed as part of running the iteration).