[Bug]: InternalError on iOS when Tasks is installed

[juanbretti]

Describe the bug

I get this error:

And the backup is not completed.
To solve the issue, I have to not install Tasks, and clone again the repository.

Relevant errors (if available)

InternalError: An internal error caused this command to fail. Please file a bug report at 
https://github.com/isomorphic-git/isomorphic-git/issues 
with this error message: SHA check failed! Expected 27eb43e79ec4b1236d2ba1b27e738f279420bd93, 
computed b77dOb17f624d487b84cc67d4510794aeb349819

Steps to reproduce

The steps I am doing:

• Created a new vault.
• Clone a repository from Github.
• The repository has .obsidian.
• Set depth to 1.

No issues so far. Then...

• Edit a note.
• Run Obsidian Git: Create backup
• Backup is complete

No issues so far. Then...

• Install the Tasks plugin (https://publish.obsidian.md/tasks/)
• Run Obsidian Git: Create backup
• I get the mentioned error
• Backup is not done

Expected Behavior

No response

Addition context

Obsidian: 1.4.4 (98) << From TestFligth
Also tested with the stable release from the AppStore.

iPhone 12
iOS: 16.4.1
Calendar: 1.5.10
Dataview: 0.5.55
Obsidian Git: 2.19.1

No other plugin installed on my phone.

Operating system

iOS

Installation Method

Other

Plugin version

2.19.1

[juandastic]

I am having the exact same issue

[GJRobert]

I'm having this issue on iOS, but I don't have the Tasks plugin installed, FYI.

[juanbretti]

I'm having this issue on iOS, but I don't have the Tasks plugin installed, FYI.

@GJRobert, could you list the plugins you have installed on your iOS?

[GJRobert]

I'm having this issue on iOS, but I don't have the Tasks plugin installed, FYI.

@GJRobert, could you list the plugins you have installed on your iOS?

Hi thanks. Here they are:

1. Front Matter Title
2. Obsidian Git

Very few because this is just a vault in its early stage for testing Git syncing between 3 systems.

[GJRobert]

I don't know the cause. All my commits on iPad cannot be pushed any more, this error message starts today (5/12 UTC+8) on my iPad.
It was doing okay yesterday.

[anpin]

I'm having exact same issue since yesterday on my iOS device. While trying to pinpoint the cause I was switching branches and were able to push and pull most of the commits, but then issue came back. The issue also crashes obsidian UI and becomes totally black and unresponsive until app restart, I can only prevent it by shutting down network with airplane mode.

[juandastic]

I couldn't solve it. I just copy all the files from my iPad to google drive and commit the changes from my PC, then I delete the vault and create a new one and clone the repo again.
I tried using Working Copy as an alternative but the feature to auto pull on startup and auto push every x minutes are time saving

[GJRobert]

After copying the vault to iCloud and got it to my MacBook, on MacBook the error message would read:

corrupt loose object 'dee91ecf0955b6a59ad486510a8fd4400dbbda9c'
fatal: unable to read dee91ecf0955b6a59ad486510a8fd4400dbbda9c

and I searched for "corrupt loose object" and found this bug #423

I did upgrade my iPad to iOS 16 recently, but this issue is occurring on my iPad 1 week after the upgrade. Not sure if iOS 16 is the cause.

And now I'm trying to look for how to fix "loose object" of a Git repo to see if I can push them normally.

[GJRobert]

Although I have researched and tried several ways to try to fix the corrupted vault repo (even with the one Linus Torvalds documented), I still can't correct it. So finally I re-cloned the repo and replaced the .git/ in my vault, and sync the vault back to iPad, re-set the auth information, and pushed with this rebuilt vault repo. Old unpushed commits are lost and combined as a new commit, but at least the vault on iOS is back working again.

I'll keep testing to see if iOS cause any problem. And perhaps everybody using Obsidian-git on iOS should watch not to modify files during pushing, or take care not having any disk issue to try to prevent this issue from happening again.

[dbischof90]

I see that @GJRobert is reporting here as well :)

@tamiroh there is an issue open at isomorphic-git/isomorphic-git#1760 that tracks the git side of things. Is it possible that files are hashed (not pushed, but hashed) while they are being modified? Could that lead to a broken state? If yes, then you could try to lock the file for the time (e.g., copy the currently active file to a temporary file/buffer, finish the git task and then replace the committed file with the buffer (which should be commited file plus the delta of typed stuff). Something there seems to be corrupting the commit.

I had the same problems but only another LaTeX-helper installed, not the tasks add-on.

[tamiroh]

Hmm...

@Vinzent03
How about the above issue?

[anpin]

After copying the vault to iCloud

@GJRobert have you been able to copy the .git folder? It is not showing up for me in the files on iOS, so I'm only able to copy latest version of notes files, without the commit history

[GJRobert]

@anpin iOS will surely copy the whole vault including .git for you, and you can check it on your Mac(Book). At least on my side.

[dbischof90]

After copying the vault to iCloud

@GJRobert have you been able to copy the .git folder? It is not showing up for me in the files on iOS, so I'm only able to copy latest version of notes files, without the commit history

I can access it via shell: https://forum.obsidian.md/t/mobile-ios-app-to-work-with-hidden-folder/25741/2

[anpin]

thanks @dbischof90, I'll try that. Got this issue again after recovering it and using for like 5 days :(

[dead-end]

The error message

SHA check failed! Expected
27eb43e79ec4b1236d2ba1b27e738f279420bd93, computed 
b77dOb17f624d487b84cc67d4510794aeb349819

is a cryptic way to say that the .git/index file is broken. The file consists of 3 parts:

1. fixed size header
2. list of staging informations for each file
3. hash which is computed over 1. and 2.

The error message says that the hash over the first two parts is not correct.

Isomorphic-git has the class GitIndexManager.js which maintances the .git/index
file. The class ensures that the file has a proper format, the hash is always
computed when the list in 2. changed and has a lock to ensure that
concurrent writes that could corrupt the file can never occur. In isomorphic-git
the .git/index is never directy accessed, only through the GitIndexManager class.

In the obsidian-git repository there is a file: myAdapter.ts which contains the
following code that overwrites the .git/index:

    async saveAndClear(): Promise<void> {
        if (this.index !== undefined) {
            await this.adapter.writeBinary(
                this.plugin.gitManager.getVaultPath(this.gitDir + "/index"),
                this.index,
                {
                    ctime: this.indexctime,
                    mtime: this.indexmtime,
                }
            );
        }
        this.clearIndex();
    }

Why do you do this?

The issue in isomorohic-git: isomorphic-git/isomorphic-git#1760

[devzero]

I have this issue as well, but can still replicate it without the Tasks plugins. I sync from multiple desktops and an iOS phone and starting in may, I can no longer commit changes from the iOS phone. This is a huge issue for me.

[juanbretti]

I have this issue as well, but can still replicate it without the Tasks plugins. I sync from multiple desktops and an iOS phone and starting in may, I can no longer commit changes from the iOS phone. This is a huge issue for me.

My personal solution, limiting the number of plugins on iOS.
Alternatively, maybe you could .gitignore all the plugins on iOS.

[devzero]

Even with no plugins other than git installed, with my repo, this happens

[juanbretti]

Even with no plugins other than git installed, with my repo, this happens

I am now at the same stage. I am not able to sync anymore.
Any ideas?

[Vinzent03]

@dead-end The reason for me to cache the .git/index file is a HUGE performance gain. Without that caching layer, it would reread the index file for every minor action, which is very slow. Maybe there is some issue resulting in a broken index file. But I can't reproduce your issue, so I don't know what exactly to fix. I'm very sure it has nothing to do with any specific plugin installed, but rather the files count. Though I think the tasks plugin is huge and should therefore not be committed.

[juanbretti]

[...] Though I think the tasks plugin is huge and should therefore not be committed.

For me, it works with version Obsidian Git 2.19.1 and Dataview and Calendar plugin installed.
But if I try to update to Obsidian Git 2.20.4, it breaks.
So, to make it work again, I had to downgraded to 2.19.1. Could it be something related the version?

I have 122 .md and 1 .jpg files. A total of 1.70 MiB.

Do you have any ideas to try?
Thanks!

[Vinzent03]

From a quick look at the CHANGELOG, I can't find any change that could affect this, so I doubt the plugin version is responsible. I guess it's a more race condition or similar and thus hard to reproduce. Though once the index is broken, it's not that easy to recover.

[juanbretti]

Is there any trick to re-create the index on the mobile (iOS)?
Or I must restart the git repository from GitHub?

[dead-end]

@Vinzent03 As I said in the note above, the GitIndexManager.js has a lock, that ensures that concurrent writing to the index results in a valid index file. You delegate the writing to the saveAndClear function, which pushes the write outside the lock context and with this concurrent writes are possible. As far as I understand the implementation you pull before you commit. Both can update the index. If you look at the first image in this thread you see

1. commit
2. pull
3. error

which has the wrong order. I think this is a race problem, but I have no evidence jet.

With your implementation you only save the reading of the index file, but the file has to be parsed each time. A better way to improve performance is to use the isomorphic git cache:

https://isomorphic-git.org/docs/en/cache

Create a global cache object, pass this to the function calls and flush the cache from time to time. You get better performance because you cache pared objects and you relay on the official api.

[Vinzent03]

I'm aware of the cache object and pretty sure I've tested that as well, but will look into that again.

[Vinzent03]

I've just tried it and it still reads the index file for every file, so using a cache does not fix it. It's only useful if I call a isogit function many times, but methods like statusMatrix use their own cache if none is provided.

[dead-end]

@Vinzent03 I think the intended way to use the cache is to create one (global) cache object and to pass it to all function calls. If this does not work, create an issue at isomorphic git with some example code and we will fix it. This is better than implementing a cache for yourself.

[maakbaas]

I have the same with:

• a clean vault
• a succesfully cloned repo with a single md file
• no other plugins

Anything I can do, or is the plugin basically broken?

[juanbretti]

Anything I can do, or is the plugin basically broken?

For me, now, it is working on iOS with version 2.19.1.
But I'm not sure if it is a stable condition.

If I try to update, it breaks.

[tl87]

I'm also having this issue and I've tried with a clean new vault, with only two plugins (Excalidraw and Obsidian-Git).

Obsidian-Git version: 2.20.5
iPad iOS version: 16.5.1

Im not sure how to resolve this issue and I'm hoping there will be a solution soon, because this plugin is awesome :-)

[GJRobert]

At least make your vault sync properly again, people, you don't have to stuck here. Good luck!
(me 2 months ago)

So finally I re-cloned the repo and replaced the .git/ in my vault, and sync the vault back to iPad, re-set the auth information, and pushed with this rebuilt vault repo. Old unpushed commits are lost and combined as a new commit, but at least the vault on iOS is back working again.

[tl87]

Does any one know a way to export the .obsidian folder from your iPad ? I might have an idea.

[narangkay]

@Vinzent03 what do you think about disabling the optimisation just for iOS? This is a really painful experience which results in having to reclone the entire repo every ~few days.

[GJRobert]

Does any one know a way to export the .obsidian folder from your iPad ? I might have an idea.

@tl87 Mine: copying the vault to iCloud and got it synced to my MacBook. .obsidian included.

[UzverUA]

Yeah, there is no other way to fix this bug but completely delete vault and clone it from git.

For those who have many devices - I strongly advice to rename your .obsidian folders(settings -> about) for each device specifically and backup those to your repo as well. So in this case all I had to do is to only clone my vault, open it from obsidian, go to settings->about, change name to ".obsidian-ios" and hit "Relaunch" and I'm up and running.
But don't forget, that entering in vault will still create hidden in iOS. obsidian folder. So either remove it from shell with rm -r .obsidian or sync it and remove from other devices, or just use gitignore with .obsidian in it.

[dbischof90]

Yeah, there is no other way to fix this bug but completely delete vault and clone it from git.

For those who have many devices - I strongly advice to rename your .obsidian folders(settings -> about) for each device specifically and backup those to your repo as well. So in this case all I had to do is to only clone my vault, open it from obsidian, go to settings->about, change name to ".obsidian-ios" and hit "Relaunch" and I'm up and running.

But don't forget, that entering in vault will still create hidden in iOS. obsidian folder. So either remove it from shell with rm -r .obsidian or sync it and remove from other devices, or just use gitignore with .obsidian in it.

This sounds random and unsatisfactory but if that's the current solution... could you elaborate what the problem here is? What is so special about this folder that it brings down a git repo?

[UzverUA]

Yeah, there is no other way to fix this bug but completely delete vault and clone it from git.
For those who have many devices - I strongly advice to rename your .obsidian folders(settings -> about) for each device specifically and backup those to your repo as well. So in this case all I had to do is to only clone my vault, open it from obsidian, go to settings->about, change name to ".obsidian-ios" and hit "Relaunch" and I'm up and running.
But don't forget, that entering in vault will still create hidden in iOS. obsidian folder. So either remove it from shell with rm -r .obsidian or sync it and remove from other devices, or just use gitignore with .obsidian in it.

This sounds random and unsatisfactory but if that's the current solution... could you elaborate what the problem here is? What is so special about this folder that it brings down a git repo?

I didn't say that .obsidian folder brings down git repo. My comment was written in addition to working temporary fix by completely deleting problematic vault folder and cloning it from git. To make such actions a bit easier and faster, so you won't have to move your .obsidian folder in terminal back and forth every time this error occurs.
And in general it seems to me like good idea to store settings of all devices in git. It's pretty handy, for example, when I have to add css snippets to my iOS devices - I can do that from my PC, instead of writing bunch of commands on iOS terminal app.

[tl87]

I found that you can compress the Obsidian folder in the Files app on iOS, then move it to a computer and merge it with a repo. Then you can reclone the repo in Obsidian on the iOS device, but then Obsidian on the iOS device, starts complaining about merge error, which is unsupported. Even with a clean clone from the git repo, this keeps happening.

You can always delete the plugin and use working-copy app on the iOS device to keep Obisidan synced.

[dbischof90]

Yeah, there is no other way to fix this bug but completely delete vault and clone it from git.
For those who have many devices - I strongly advice to rename your .obsidian folders(settings -> about) for each device specifically and backup those to your repo as well. So in this case all I had to do is to only clone my vault, open it from obsidian, go to settings->about, change name to ".obsidian-ios" and hit "Relaunch" and I'm up and running.
But don't forget, that entering in vault will still create hidden in iOS. obsidian folder. So either remove it from shell with rm -r .obsidian or sync it and remove from other devices, or just use gitignore with .obsidian in it.

This sounds random and unsatisfactory but if that's the current solution... could you elaborate what the problem here is? What is so special about this folder that it brings down a git repo?

I didn't say that .obsidian folder brings down git repo. My comment was written in addition to working temporary fix by completely deleting problematic vault folder and cloning it from git. To make such actions a bit easier and faster, so you won't have to move your .obsidian folder in terminal back and forth every time this error occurs. And in general it seems to me like good idea to store settings of all devices in git. It's pretty handy, for example, when I have to add css snippets to my iOS devices - I can do that from my PC, instead of writing bunch of commands on iOS terminal app.

That makes absolute sense! I was just wondering whether it's the .obsidian folder for some reason and that this would be so severe that it is essentially a won't-fix. I have only little experience with JS and even less with iOS app developement so I'm not gonna be of much help here - just some food for thought for someone who knows the codebase a bit better.

So it seems like we get an incorrect index written at some times, hence next time we check the index, the SHA can't be recalculated correctly and we get therefore an inconsistent state. I'm not sure if the authors of the articles that I scanned through use the terms "parallel" and "concurrent" correct but I read a few times that in iOS applications, async function calls can indeed be called in parallel - which would mean that while you'd await the termination of the index write, another task could make progress and modify the filesystem. If the index creation task is still not done, that could theoretically lead to this situation.

1. Is my understanding correct? Does Apple's JS engine execute async tasks on a threadpool (so really in parallel) potentially?
2. Is that even a problematic scenario? I think that situation could be tested in Python even, having 30 processes that randomly modify, create and delete files and an 31st one which creates commits on a local repo every few seconds. Then again, Python calls the reference implementation, not the reimplementation.
3. If yes to both, would a potential fix then not "just" involve an actual RWLock to enforce pausing other tasks? That would mean a performance regression, true, but well hashing a folder structure is a maybe not something you should do while you write into it...

[dbischof90]

A second thought: I just read this comment here and that could apply to us here as well: https://stackoverflow.com/a/1115956/5058041

So if we don't find out and accept that for now something fucky goes on a fix could look like:

1. backup the index file if SHA check fails and delete it
2. git reset --keep
3. Stage/Commit again
4. If successful: Delete backup and done, if not: delete 'new index file', put old in place again and show the above message

[UzverUA]

A second thought: I just read this comment here and that could apply to us here as well: https://stackoverflow.com/a/1115956/5058041

So if we don't find out and accept that for now something fucky goes on a fix could look like:

1. backup the index file if SHA check fails and delete it
2. git reset --keep
3. Stage/Commit again
4. If successful: Delete backup and done, if not: delete 'new index file', put old in place again and show the above message

So it happened again. But I wasn't able to do what you wrote, because a-shell's lg2, seems like, doesn't support such command. Or I don't know) But, I tried to search for another solution, because I made changes in vault that I didn't want to lose. I tried so many things, that I'm not even sure that this is the way to go, but, at least, it looks like lg2 push --force origin fixed that problem, till next time:)
Not perfect solution, but, at least, much faster than anything there currently is. I checked files that was pushed, and they were not corrupted.

P.S. I got that error right after installing Minimal theme, Minimal Settings plugin and Style Settings plugin.