[ZEPPELIN-3758]. Convert old note file note.json to new style #3189
Conversation
zjffdu
force-pushed
the
ZEPPELIN-3758
branch
from
1f3a398
to
ad3b825
Compare
|
Also, since we are doing this; should there be a command line utility as well? which on run from command-line can convert from old to new format. |
|
Good idea, I can do it in a follow up PR. |
zjffdu
force-pushed
the
ZEPPELIN-3758
branch
from
ad3b825
to
5b59dcc
Compare
|
Many classes are old existing code which I just rename them. e.g. |
| OldNotebookRepo oldNotebookRepo = | ||
| PluginManager.get().loadOldNotebookRepo(newNotebookRepo.getClass().getCanonicalName()); | ||
| oldNotebookRepo.init(conf); | ||
| List<OldNoteInfo> oldNotesInfo = oldNotebookRepo.list(AuthenticationInfo.ANONYMOUS); |
There was a problem hiding this comment.
would accessing as anonymous here causes some old note to be skipped?
There was a problem hiding this comment.
No, anonymous can access all notes.
There was a problem hiding this comment.
ah, that sounds a bit counter - intuitive..
| note.setPath(note.getName()); | ||
| newNotebookRepo.save(note, AuthenticationInfo.ANONYMOUS); | ||
| if (deleteOld) { | ||
| oldNotebookRepo.remove(note.getId(), AuthenticationInfo.ANONYMOUS); |
| @@ -732,6 +732,8 @@ public String getZeppelinSearchTempPath() { | |||
| ZEPPELIN_NOTEBOOK_STORAGE("zeppelin.notebook.storage", | |||
| "org.apache.zeppelin.notebook.repo.GitNotebookRepo"), | |||
| ZEPPELIN_NOTEBOOK_ONE_WAY_SYNC("zeppelin.notebook.one.way.sync", false), | |||
| ZEPPELIN_NOTEBOOK_NEW_FORMAT_CONVERT("zeppelin.notebook.new_format.convert", false), | |||
| ZEPPELIN_NOTEBOOK_NEW_FORMAT_DELETE_OLD("zeppelin.notebook.new_format.delete_old", false), | |||
There was a problem hiding this comment.
If zeppelin.notebook.new_format.convert=true and zeppelin.notebook.new_format.delete_old=false, would Zeppelin try to convert same old notes many times, after each restart?
There was a problem hiding this comment.
That's right, user need to set zeppelin.notebook.new_format.convert to false when converting is done. so that next time zeppelin won't do the conversion again.
There was a problem hiding this comment.
If the user forgot to change it back, every restart would convert an old version of an old format (json) and overwrites it on a new version of a newer version note. I would go with the script, but it's not easy when you have 100 notes in HDFS :(
|
|
||
| @Override | ||
| public void save(final Note note, AuthenticationInfo subject) throws IOException { | ||
| this.fs.writeFile(note.toJson(), |
There was a problem hiding this comment.
It better be two-phase write approach.
Write to a temp file first, and then rename atomically.
Otherwise we may end up with partial writes, and lost note's content completely (in case if e.g. a file system was full).
See for example https://issues.apache.org/jira/browse/ZEPPELIN-3467
There was a problem hiding this comment.
It will write to a temp file first. See here https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/FileSystemStorage.java#L178
|
Does this work out of the box if the notes are in HDFS? I am facing a similar situation which I am guessing is related to this issue: |
What is this PR for?
This PR is follow up of ZEPPELIN-261, it is to convert old note file note.json to new style when user upgrading zeppelin before 0.9. 2 properties are introduced.
zeppelin.notebook.new_format.convert, by default it is false. When enabled, zeppelin will first find all the old note file note.json and then convert it into new style.zeppelin.notebook.new_format.delete_old, by default it is false, when setting true, zeppelin will delete the old note file after converting it into new style.What type of PR is it?
[Improvement]
Todos
What is the Jira issue?
How should this be tested?
Screenshots (if appropriate)
Questions: