Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use chill.ClosureCleaner on all functions #267

Closed
johnynek opened this issue Jan 9, 2013 · 1 comment
Closed

Use chill.ClosureCleaner on all functions #267

johnynek opened this issue Jan 9, 2013 · 1 comment

Comments

@johnynek
Copy link
Collaborator

johnynek commented Jan 9, 2013

Scala captures a reference to the enclosing object always, even if the function doesn't reference that object.

When the job is serialized, this can dramatically increase the size, and cause some things to need to be serializable that don't really need to be.

A fix for this is to go into the few Cascading Operations we have, and call ClosureCleaner(fn) on all the functions passed into constructors.

This could potentially dramatically reduce this size of jobConfs and speed up submission times. Also, it could make it easier for users by reducing the number of cases where they need to hack around weird serialization errors.

@johnynek
Copy link
Collaborator Author

addressed in: #273

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant