Working Git repositories #491

amberin · 2019-03-13T13:09:53Z

OK, so, first of all... I'm not really a programmer, much less a Java programmer. I just really wanted this functionality myself. I used this as a kind of exercise to learn something about Java and app development.

I've probably made some horrible decisions here, but this code seems to work well for me (on a Samsung S7), and I'm hoping that someone more experienced will be able to suggest improvements if this is not acceptable. I'd also be willing to re-work things if I can get pointers in the right direction.

I believe the commit messages explain more or less all my changes.

This was the easiest way I could think of to let RepoPreferences connect a repo URI with a repo ID. I'm not sure if it's a good idea, as it required lots of changes to the SyncRepo creation calls. But they don't seem to have broken... yet. If we could store repository preferences in the database, instead of in an XML file, then RepoPreferences wouldn't have much need for the repo ID number anymore.

revision of a notebook, one for adding new notebooks, and one for deleting notebooks. I also renamed the variable repositoryPath to fileName, as I thought the name was misleading.

1. Create RepoPreferences in the new way, passing in the repo ID as a third argument. 2. Allow cloning into an existing but empty repository. 3. A working method for finding the latest revision of a notebook in the repo. 4. A working method for creating a new notebook. 5. A new method for deleting a notebook from the repository.

1. Use viewModel and RepoFactory, like the other repo activities. 2. Create RepoPreferences in the new way, with repoId as an argument. 3. Catch userData from RepoCreate which tells us the ID of a newly created repo. 4. Work around the file browser adding "file:" to the beginning of paths.

Hint to the user that the chosen local directory must be empty.

clone command fail. When JGit 5.3 or higher is released, we should switch to that. (The hard link issue seems to have been fixed in February 2019.)

amberin

Damn, I just realized the commit message of 553aa93 is wrong. I added the repo URI as a third argument to RepoPreferences, and the repo ID was added as a third argument to the SyncRepo build methods in RepoFactory. Oh well...

amberin

And in cabe012 the second change is to allow cloning into an empty folder, not repository...

colonelpanic8 · 2019-03-14T23:33:26Z

@amberin Thanks for picking this up. I think that @nevenz wanted me to fix a bunch of smaller things (#173 (comment)) before making the git functionality live. Did you fix all of those?

colonelpanic8 · 2019-03-14T23:36:22Z

app/src/main/java/com/orgzly/android/repos/GitRepo.java

-                file, fileName, synchronizer.getFileRevision(fileName, commit));
-        synchronizer.tryPushIfUpdated(commit);
-        return currentVersionedRook(Uri.EMPTY.buildUpon().appendPath(fileName).build());
+        synchronizer.addAndCommitNewFile(file, fileName);


Please do not make this change. I went through great pains to make it so that all of the operations here were safe. You've basically completely removed all of that safety.

You need to do what it actually says in the fixme.

@nevenz Can you clarify what you meant in this fixme? Is this information simply no longer stored at all? I'm not really sure what you mean by "get list from the remote". What we need is to be able to access that piece of meta data.

@IvanMalison I hear you, that's good feedback. But I found that this method is only used when a new notebook is created, and then I couldn't see neither the need for nor the possibility of checking for previous revisions of the new file.

I also had trouble understanding the fixme, fwiw...

FIXME: Removed current_versioned_rooks table, just get the list from remote

@IvanMalison I think I just meant get this data by querying the remote repository (doing an API call).

To avoid the extra call - this data is available in getBooksFromAllRepos. Interface could be changed so that VersionedRook is passed where needed.

But as @amberin said, I don't think that storeBook and retrieveBook are used for TwoWaySyncRepo repo when there is no matching remote notebook:

orgzly-android/app/src/main/java/com/orgzly/android/sync/SyncService.java

Lines 444 to 457 in cabe012

// FIXME: This is a pretty nasty hack that completely circumvents the existing code path

if (!namesake.getRooks().isEmpty()) {

VersionedRook rook = namesake.getRooks().get(0);

if (rook != null && namesake.getStatus() != BookSyncStatus.NO_CHANGE) {

Uri repoUri = rook.getRepoUri();

SyncRepo repo = dataRepository.getRepo(repoUri);

if (repo instanceof TwoWaySyncRepo) {

handleTwoWaySync(dataRepository, (TwoWaySyncRepo) repo, namesake);

return BookAction.forNow(

BookAction.Type.INFO,

namesake.getStatus().msg(repo.getUri().toString()));

}

}

}

My memory is hazy, but I'm quite sure I had a reason for writing the code and structuring it as I did. It should not be hard to do the right thing here.

But as @amberin said, I don't think that storeBook and retrieveBook are used for TwoWaySyncRepo repo when there is no matching remote notebook

I've just checked, and it seems to me that storeBook() only gets called 1) when there is no matching remote notebook, and 2) upon forced save. What risks do you see in those scenarios, @IvanMalison?

a) this is only true because this is a a twowaysync repo, right? storeBook is not an add file method.
b) I simply don't want any code paths at all that do things unconditionally. This makes it much harder to have anything go wrong, or for future changes to accidentally make something unexpected happen.
c) Doing the right thing here shouldn't be hard

How about this? https://github.com/orgzly/orgzly-android/pull/491/files#diff-6887dc2ebf7c9e65784782c2717ffcaeL127

colonelpanic8 · 2019-03-14T23:40:42Z

app/src/main/java/com/orgzly/android/repos/GitRepo.java

-        if (syncBackNeeded) {
-            writeBack = synchronizer.repoDirectoryFile(fileName);
-        }
+        writeBack = synchronizer.repoDirectoryFile(fileName);


Why was this change made?

I don't remember all the details, but I couldn't get the TwoWaySyncResult() constructor to run when writeBack was not set.

Please don't do this. Avoiding sync back when it is not needed makes syncing much more light weight and efficient in the case where many files are involved.

You are probably right. I am now unable to reproduce the issue that I was having when I made this change, so I will revert this.

FWIW, I am now also unable to create a situation where this syncBook() method is invoked without syncBackNeeded ending up true, so it isn't being invoked for books without changes.

(This was reverted in ccfb719)

amberin · 2019-03-15T00:03:47Z

@IvanMalison I definitely did not fix all of those issues -- I just got to the point where the functionality can actually be used. I'd be happy to fix more bugs and work more on it, but first I wanted to check if these changes are insane or not...

I'm not sure if the browser-related issues still apply. I'm testing on my Samsung, and it seems like their file browser is a bit special.

I believe the most demanding issue in that list (for me) is support for user/password auth. That is definitely not something I would know how to go about implementing.

colonelpanic8 · 2019-03-15T00:28:30Z

@IvanMalison I definitely did not fix all of those issues -- I just got to the point where the functionality can actually be used. I'd be happy to fix more bugs and work more on it, but first I wanted to check if these changes are insane or not...

The changes are definitely not in line with the what I originally intended with the code.

I believe the most demanding issue in that list (for me) is support for user/password auth. That is definitely not something I would know how to go about implementing.

That should not be very hard. JGit has user/password auth facilities:

https://www.codeaffine.com/2014/12/09/jgit-authentication/

they give this example:

CloneCommand cloneCommand = Git.cloneRepository();
cloneCommand.setURI( "https://example.com/repo.git" );
cloneCommand.setCredentialsProvider( new UsernamePasswordCredentialsProvider( "user", "password" ) );

Orgzly just needs to figure out how it wants to store this data.

colonelpanic8 · 2019-03-16T08:03:54Z

app/src/main/java/com/orgzly/android/repos/GitRepo.java

-    private VersionedRook currentVersionedRook(Uri uri) throws IOException {
-        RevCommit newCommit = synchronizer.currentHead();
-        return new VersionedRook(getUri(), uri, newCommit.name(), newCommit.getCommitTime()*1000);
+    private VersionedRook currentVersionedRook(Uri uri) {


What are all of these changes for? Why would the name attribute be used in the versioned rook object?

Because the mtime and revision hash of all notebooks was being set to that of the entire repo's latest commit. With this change, the mtime and revision hash shows correctly for each notebook. The previous situation also meant that after every commit, all unchanged notebooks would have their mtime and revision hash updated during the subsequent sync (to that of the latest commit), which produced the CONFLICT_LAST_SYNCED_ROOK_AND_LATEST_ROOK_ARE_DIFFERENT status.

I haven't changed which arguments get passed to the VersionedRook constructor. The name attribute is the commit hash which forms the revision ID of the rook.

colonelpanic8 · 2019-03-16T08:06:02Z

app/src/main/java/com/orgzly/android/repos/GitRepo.java

-        if (syncBackNeeded) {
-            writeBack = synchronizer.repoDirectoryFile(fileName);
-        }
+        writeBack = synchronizer.repoDirectoryFile(fileName);


Please don't do this. Avoiding sync back when it is not needed makes syncing much more light weight and efficient in the case where many files are involved.

Before this, Git repo settings were not saved properly when an existing repo was updated.

lfxgroove · 2019-05-09T17:29:15Z

@amberin if there's any help needed to make this reach it's goal i'd be happy to help out, programming wise or otherwise :)

colonelpanic8 · 2019-05-09T18:16:13Z

@lfxgroove Take a look at this comment on my original pull request: #173 (comment)

That's the work that needs to get done.

lfxgroove · 2019-05-10T14:49:45Z

List to keep track of what's done, taken from #173:
Issues or missing features to be resolved before enabling this in the app

Warning when syncing from Android Studio

Warning:WARNING: Dependency org.apache.httpcomponents:httpclient:4.1.3 is ignored for fdroidDebug as it may be conflicting with the internal version provided by Android.

Building fails when doing ./gradlew build (proguard-rules.pro needs updating probably)
Asking for Storage permission before entering Git repo creation, leaving user in repos list, see 596ff7f
Prepopulate directory (Orgzly/repos/git in storage root or similar)
On tapping checkmark (using git@bitbucket.org:nevenz/foo.git)

11-09 08:05:24.383  7914  7914 W System.err: java.io.IOException: Directory /storage/emulated/0/Download/git is not a git repository.
11-09 08:05:24.383  7914  7914 W System.err:    at com.orgzly.android.repos.GitRepo.ensureRepositoryExists(GitRepo.java:109)
11-09 08:05:24.384  7914  7914 W System.err:    at com.orgzly.android.repos.GitRepo.ensureRepositoryExists(GitRepo.java:73)
11-09 08:05:24.384  7914  7914 W System.err:    at com.orgzly.android.ui.fragments.GitRepoFragment$RepoCloneTask.doInBackground(GitRepoFragment.java:343)
11-09 08:05:24.384  7914  7914 W System.err:    at com.orgzly.android.ui.fragments.GitRepoFragment$RepoCloneTask.doInBackground(GitRepoFragment.java:312)

Had to delete selected (empty, created from browser) directory (can't use just browser, we have to type?)
Edit repo, change to non-existing URL, no check (dialog) this time, repo is accepted
Trying to sync gives "No repos configured" (with non-existing URL created above). Later discovered that the file from repository was downloaded at one point after all
"Unsupported repository type" trying to edit repo (non-existing URL)

11-09 08:15:50.019  7914  7914 W System.err: java.io.IOException: The file /storage/emulated/0/Documents/nevenz/foo2.git does not exist
11-09 08:15:50.019  7914  7914 W System.err:    at com.orgzly.android.repos.GitRepo.ensureRepositoryExists(GitRepo.java:105)
11-09 08:15:50.019  7914  7914 W System.err:    at com.orgzly.android.repos.GitRepo.ensureRepositoryExists(GitRepo.java:73)
11-09 08:15:50.019  7914  7914 W System.err:    at com.orgzly.android.repos.GitRepo.build(GitRepo.java:57)
11-09 08:15:50.019  7914  7914 W System.err:    at com.orgzly.android.repos.GitRepo.buildFromUri(GitRepo.java:53)
11-09 08:15:50.019  7914  7914 W System.err:    at com.orgzly.android.repos.RepoFactory.getFromUri(RepoFactory.java:49)
11-09 08:15:50.019  7914  7914 W System.err:    at com.orgzly.android.repos.RepoFactory.getFromUri(RepoFactory.java:23)
11-09 08:15:50.019  7914  7914 W System.err:    at com.orgzly.android.ui.ReposActivity.onRepoEditRequest(ReposActivity.java:188)
11-09 08:15:50.019  7914  7914 W System.err:    at com.orgzly.android.ui.fragments.ReposFragment.onListItemClick(ReposFragment.java:130)

Using https://nevenz@bitbucket.org/nevenz/foo.git

10-12 23:20:20.008  6287  6416 E AndroidRuntime: Caused by: java.lang.ClassCastException: org.eclipse.jgit.transport.TransportHttp cannot be cast to org.eclipse.jgit.transport.SshTransport
10-12 23:20:20.008  6287  6416 E AndroidRuntime:        at com.orgzly.android.git.GitSSHKeyTransportSetter$2.configure(GitSSHKeyTransportSetter.java:40)
10-12 23:20:20.008  6287  6416 E AndroidRuntime:        at org.eclipse.jgit.api.TransportCommand.configure(TransportCommand.java:138)
10-12 23:20:20.008  6287  6416 E AndroidRuntime:        at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:128)
10-12 23:20:20.008  6287  6416 E AndroidRuntime:        at org.eclipse.jgit.api.CloneCommand.fetch(CloneCommand.java:193)
10-12 23:20:20.008  6287  6416 E AndroidRuntime:        at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:133)
10-12 23:20:20.008  6287  6416 E AndroidRuntime:        at com.orgzly.android.repos.GitRepo.ensureRepositoryExists(GitRepo.java:91)
10-12 23:20:20.008  6287  6416 E AndroidRuntime:        at com.orgzly.android.repos.GitRepo.ensureRepositoryExists(GitRepo.java:73)
10-12 23:20:20.008  6287  6416 E AndroidRuntime:        at com.orgzly.android.ui.fragments.GitRepoFragment$RepoCloneTask.doInBackground(GitRepoFragment.java:343)
10-12 23:20:20.008  6287  6416 E AndroidRuntime:        at com.orgzly.android.ui.fragments.GitRepoFragment$RepoCloneTask.doInBackground(GitRepoFragment.java:312)
10-12 23:20:20.008  6287  6416 E AndroidRuntime:        at android.os.AsyncTask$2.call(AsyncTask.java:305)
10-12 23:20:20.008  6287  6416 E AndroidRuntime:        at java.util.concurrent.FutureTask.run(FutureTask.java:237)

Clicking checkmark immediately (all fields empty): Snackbar with javaj.io.IOException: Failed to clone repository , null
Entering "asd" as URL and pressing checkmark

java.io.IOException: Failed to clone repository asd, org.eclipse.jgit.errors.NoRemoteRepositoryException: asd: not found.

Error checking for all other fields
Allow login with user/pass

Minor stuff

colonelpanic8 · 2019-05-10T16:12:46Z

@lfxgroove Why are you tracking all of this here? Seems like #24 Would be the right place. I haven't looked at the most recent changes, but I'm not totally sold on all of the things @amberin is doing in this PR. If you resolver any of the issues @nevenz had it should probably be as part of a separate PR.

lfxgroove · 2019-05-11T05:30:19Z

@IvanMalison good question, don't really know, i'll post on that issue instead.

amberin added 6 commits March 13, 2019 12:45

Add three new methods to GitFileSynchronizer: one for finding the latest

c82c8af

revision of a notebook, one for adding new notebooks, and one for deleting notebooks. I also renamed the variable repositoryPath to fileName, as I thought the name was misleading.

Quality of life UI fixes to the Git repo activity.

e007863

Hint to the user that the chosen local directory must be empty.

Use a JGit version that does not rely on hard linking, since it made the

02004d6

clone command fail. When JGit 5.3 or higher is released, we should switch to that. (The hard link issue seems to have been fixed in February 2019.)

amberin commented Mar 13, 2019

View reviewed changes

amberin changed the title ~~Working Git repos~~ Working Git repositories Mar 13, 2019

amberin mentioned this pull request Mar 14, 2019

git synchroniztion #24

Open

colonelpanic8 reviewed Mar 14, 2019

View reviewed changes

colonelpanic8 suggested changes Mar 16, 2019

View reviewed changes

amberin added 5 commits March 16, 2019 23:43

Reverting an unnecessary change in commit cabe012.

ccfb719

Trying to add some safety to GitRepo.storeBook()

93dc366

Reverting an unintentional addition in the last commit.

7fe2746

One more sanity check added to storeBook().

ffb5a85

Make usecase.RepoUpdate() return the new repo ID.

935200c

Before this, Git repo settings were not saved properly when an existing repo was updated.

amberin mentioned this pull request May 17, 2019

Git sync base changes #543

Merged

33 tasks

amberin closed this May 30, 2019

amberin deleted the git_suggestions branch July 7, 2022 11:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Working Git repositories #491

Working Git repositories #491

amberin commented Mar 13, 2019

amberin left a comment •

edited

Loading

amberin left a comment

colonelpanic8 commented Mar 14, 2019

colonelpanic8 Mar 14, 2019

colonelpanic8 Mar 14, 2019

amberin Mar 15, 2019

nevenz Mar 15, 2019

colonelpanic8 Mar 16, 2019

amberin Mar 17, 2019 •

edited

Loading

colonelpanic8 Mar 17, 2019

amberin Mar 19, 2019

colonelpanic8 Mar 14, 2019

amberin Mar 15, 2019

colonelpanic8 Mar 16, 2019

amberin Mar 16, 2019

amberin Mar 25, 2019

amberin commented Mar 15, 2019

colonelpanic8 commented Mar 15, 2019

colonelpanic8 Mar 16, 2019

amberin Mar 16, 2019

colonelpanic8 Mar 16, 2019

lfxgroove commented May 9, 2019

colonelpanic8 commented May 9, 2019 •

edited

Loading

lfxgroove commented May 10, 2019 •

edited

Loading

colonelpanic8 commented May 10, 2019

lfxgroove commented May 11, 2019

	// FIXME: This is a pretty nasty hack that completely circumvents the existing code path
	if (!namesake.getRooks().isEmpty()) {
	VersionedRook rook = namesake.getRooks().get(0);
	if (rook != null && namesake.getStatus() != BookSyncStatus.NO_CHANGE) {
	Uri repoUri = rook.getRepoUri();
	SyncRepo repo = dataRepository.getRepo(repoUri);
	if (repo instanceof TwoWaySyncRepo) {
	handleTwoWaySync(dataRepository, (TwoWaySyncRepo) repo, namesake);
	return BookAction.forNow(
	BookAction.Type.INFO,
	namesake.getStatus().msg(repo.getUri().toString()));
	}
	}
	}

Working Git repositories #491

Working Git repositories #491

Conversation

amberin commented Mar 13, 2019

amberin left a comment • edited Loading

Choose a reason for hiding this comment

amberin left a comment

Choose a reason for hiding this comment

colonelpanic8 commented Mar 14, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amberin Mar 17, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amberin commented Mar 15, 2019

colonelpanic8 commented Mar 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lfxgroove commented May 9, 2019

colonelpanic8 commented May 9, 2019 • edited Loading

lfxgroove commented May 10, 2019 • edited Loading

colonelpanic8 commented May 10, 2019

lfxgroove commented May 11, 2019

amberin left a comment •

edited

Loading

amberin Mar 17, 2019 •

edited

Loading

colonelpanic8 commented May 9, 2019 •

edited

Loading

lfxgroove commented May 10, 2019 •

edited

Loading