Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introducing WrappedFileSystem for transactions #41433

Merged
merged 2 commits into from Jul 17, 2020

Conversation

samikama
Copy link
Contributor

This is part of a large PR set of introducing Transactional FileSystems. The class will be updated when a follow up PR updates FileSystem class methods.

@google-ml-butler google-ml-butler bot added the size:M CL Change Size: Medium label Jul 15, 2020
mihaimaruseac
mihaimaruseac previously approved these changes Jul 16, 2020
@google-ml-butler google-ml-butler bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Jul 16, 2020
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Jul 16, 2020
@gbaned gbaned self-assigned this Jul 16, 2020
@gbaned gbaned added the comp:core issues related to core part of tensorflow label Jul 16, 2020
@gbaned gbaned added this to Assigned Reviewer in PR Queue via automation Jul 16, 2020
@mihaimaruseac
Copy link
Collaborator

Hmm, this fails to build with errors such as

...
./tensorflow/core/platform/file_system.h:422:30: error: 'tensorflow::WrappedFileSystem::Stat' hides overloaded virtual function [-Werror,-Woverloaded-virtual]
  virtual tensorflow::Status Stat(
                             ^
./tensorflow/core/platform/file_system.h:163:30: note: hidden overloaded virtual function 'tensorflow::FileSystem::Stat' declared here: different number of parameters (2 vs 3)
  virtual tensorflow::Status Stat(const string& fname,
                             ^
./tensorflow/core/platform/file_system.h:428:30: error: 'tensorflow::WrappedFileSystem::DeleteFile' hides overloaded virtual function [-Werror,-Woverloaded-virtual]
  virtual tensorflow::Status DeleteFile(
                             ^
./tensorflow/core/platform/file_system.h:167:30: note: hidden overloaded virtual function 'tensorflow::FileSystem::DeleteFile' declared here: different number of parameters (1 vs 2)
  virtual tensorflow::Status DeleteFile(const string& fname) = 0;
                             ^
...

@samikama
Copy link
Contributor Author

I guess this is because of the compiler flags -Werror,-Woverloaded-virtual . We want to hide the overloaded function here. Most of the time it is unintentional but we need to do that in order to be able split a big change set to smaller ones. It doesn't show up in public builds since this flag is not there. There are 2 ways I can think of to get around this compiler flag. First one is to comment arguments of the WrappedFileSystem and uncomment it when base class changes, which need to be applied to all other FileSystem derived classes as well. Or make a huge PR changing all at the same time. Which one do you prefer.

@mihaimaruseac
Copy link
Collaborator

You are right. Since we cannot change compile flags, let's try the first approach

PR Queue automation moved this from Assigned Reviewer to Reviewer Requested Changes Jul 16, 2020
@google-ml-butler google-ml-butler bot removed the ready to pull PR ready for merge process label Jul 16, 2020
@samikama
Copy link
Contributor Author

@mihaimaruseac Done. I am going to modify other PRs and open new ones with changes in remaining filesystems however we will need to do a big change in the end to uncomment all overrides and commented arguments at once.

PR Queue automation moved this from Reviewer Requested Changes to Approved by Reviewer Jul 16, 2020
@google-ml-butler google-ml-butler bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Jul 16, 2020
@mihaimaruseac
Copy link
Collaborator

Agree that we will need a big change to change API at one point but that should be easily done/reviewed since it's mostly a string replace everywhere.

@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Jul 16, 2020
@tensorflow-copybara tensorflow-copybara merged commit aca7118 into tensorflow:master Jul 17, 2020
PR Queue automation moved this from Approved by Reviewer to Merged Jul 17, 2020
@samikama samikama deleted the Transactions_part2 branch July 17, 2020 03:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes comp:core issues related to core part of tensorflow ready to pull PR ready for merge process size:M CL Change Size: Medium
Projects
PR Queue
  
Merged
Development

Successfully merging this pull request may close these issues.

None yet

6 participants