Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete run_with_autoformat and run_without_autoformat functions #8731

Open
27 tasks
eyonland opened this issue May 22, 2024 · 2 comments
Open
27 tasks

Delete run_with_autoformat and run_without_autoformat functions #8731

eyonland opened this issue May 22, 2024 · 2 comments
Assignees

Comments

@eyonland
Copy link
Contributor

eyonland commented May 22, 2024

Problem:
We no longer want to support automatically formatting tensors (ie, changing there layouts or moving them off/on device implicitly). The user needs to be aware of the cost of doing these operations and at the moment this cost is being hidden. ttnn offers a to_layout C++ method that can be leveraged to do whatever formatting is necessary. The ask here is to identify any models that are currently using the functions below and will break if the following functions below no longer have this feature. These changes will require the models to import ttnn and use ttnn.layout wherever necessary. Subsequently, the following operations need to be update to nolonger use run_with_autoformat and run_without_autoformat.

@eyonland eyonland changed the title Remove dependencies on run_with_autoformat and run_without_autoformat Delete run_with_autoformat and run_without_autoformat functions May 22, 2024
@eyonland eyonland self-assigned this Jun 7, 2024
@eyonland eyonland added the master label Jun 7, 2024
@eyonland
Copy link
Contributor Author

Operations do not put the tensor back on device after a fallback : The merging of ttnn operations without the use of run_with_autoformat has hit some road blockers. The underlying issue is that ops like permute will fallback to running on host. They will take their tensor on device, pull it on host for the fallback, and then just leave it on host. This underlying issue with ops that are not able to perform entirely on device is being masked by tt_lib ops with run_with_autoformat. Getting rid of this means fundamentally that we have to go in and at least put the tensors back on device.

@bbradelTT
Copy link
Contributor

@eyonland what support is provided to handle odd shapes on device?

For example, outer() uses tensors of [1,1,M,1] and [1,1,1,N]. These tensors cannot be handled on device and need to be transformed into [1,1,M,32] and [1,1,32,N]. However, there is no functionality to handle this case properly outside of run_with_autoformat that I know of.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants