Doing LFS file transfers to Atlassian BitBucket and attempting to run a diff and see no diffs

[marylouwho]

Greetings to all who venture here in hopes of assisting. I feel the need to apologize ahead of time if I have done something stupid. I have two (not huge) versions of a piece of software, it appears to me that I get them uploaded into BitBucket as all the commits and pushes threw no errors and the git log response I thought was good too. When I go to run the diff between my version 38 and 43, it does not return anything as if they are the same, they are not. I can provide this information from git log response:
commit 7c1b4d76fc230f1818d86e8d4f33665b97a6a159 (HEAD -> main, origin/main)
Author: Mary Lou Forsyth MaryLou.Forsyth@weidmann-group.com
Date: Thu May 16 15:02:52 2024 -0400
Version 43: TabLink_API.py
commit b8696357af4e7bf5727f3892d625799e80b4b159
Merge: 19c1721 d7af517
Author: Mary Lou Forsyth MaryLou.Forsyth@weidmann-group.com
Date: Thu May 16 14:45:31 2024 -0400
Merge branch 'main' of https://bitbucket.org/weidmann-it-dev/tablink_repository
Tablink API V38 being committed on 5-16-2024
commit 19c1721a27e21d31404b8a3dc4cc93024ff560f9
Author: Mary Lou Forsyth MaryLou.Forsyth@weidmann-group.com
Date: Thu May 16 14:13:24 2024 -0400
Version 38: TabLink_API.py
commit d7af517c745f06cb58edcbcdf3b8f3e40b418435
Author: MaryLou.Forsyth marylou.forsyth@weidmann-group.comDate: Thu May 16 18:04:13 2024 +0000
Initial commit

But when I do these (the first one should have worked) I get nothing (the second was just a stab):
git diff 19c1721a27e21d31404b8a3dc4cc93024ff560f9 7c1b4d76fc230f1818d86e8d4f33665b97a6a159 -- Tablink_API_V38.py Tablink_API_V43.py

M:\Git2_Repository>
M:\Git2_Repository>git diff b8696357af4e7bf5727f3892d625799e80b4b159 7c1b4d76fc230f1818d86e8d4f33665b97a6a159 -- Tablink_API_V38.py Tablink_API_V43.py

M:\Git2_Repository>

Here are my steps:
git init
git lfs install
git lfs track "Tablink_API_V38.py"
git add .gitattributes
git remote add origin https://bitbucket.org/weidmann-it-dev/tablink_repository.git
git checkout -b main
git add .
git status
git commit -m "Version 38: TabLink_API.py"
git fetch origin
git pull origin main --allow-unrelated-histories
git push origin main
git remote -v
git status
git lfs track "Tablink_API_V43.py"
git add .
git commit -m "Version 43: TabLink_API.py"
git fetch origin
git pull origin main --allow-unrelated-histories
git push origin main

So, I have been a long time user of BitBucket where its been great to just upload your python file and save it, when you wanted a new version just hit edit, delete the old and paste in the new version, then look at the diffs. I want to embrace this LFS process but I am not seeing where I am going so far off track. If anyone can give me a hint or point me to more documentation I have no problem plowing through docs to get the right answer, please help if you can and bless you!

[chrisd8088]

Hey, I'm sorry you're having trouble. I'm not entirely sure what issue you're experiencing, but it sounds like you want to manage some *.py files with Git LFS, and ideally also see the differences between difference copies or versions of those files.

I suppose the first caution I should give is that when you successfully manage files with Git LFS, when you run git diff on those files, you'll only see the difference in the Git LFS pointer files, because that's all Git is now recording. The actual file content is stored in Git LFS objects, which the pointer files reference, so you'd normally have to check out both versions and use a plain Unix diff command rather than git diff to see the differences. I'll try to provide an example below which might help clarify that point.

Also, in terms of your workflow, you're creating Git attributes entries with git lfs track and committing those changes (git add .gitattributes and git commit), but I don't see when you're actually creating the *.py files you want to manage with Git LFS. If those files already exist in your repository's history, you won't be able to convert them to Git LFS objects just by running git lfs track. Instead, you'd need to run git lfs migrate import to rewrite your Git repository's commit history, converting existing files into Git LFS objects. Note that this is a "destructive" process -- it typically changes your Git history, so you want to be sure you need Git LFS and that you understand the consequences before proceeding.

May I ask, why are you looking to use Git LFS in the first place? Are there limits on the size of the files you can push into your repository? If so, are there simpler ways you could divide up your *.py files so as to stay under that limit, so you could avoid using Git LFS entirely?

---

Let me start with a simple example where you are creating new files:

$ git init test
$ cd test
$ git lfs track '*.py'
Tracking "*.py"
$ git add .gitattributes
$ git commit -m attrs
$ cat .gitattributes
*.py filter=lfs diff=lfs merge=lfs -text

At this point, any new files you create which match the *.py pattern will be converted into Git LFS objects.

$ echo foo >foo.py
$ echo bar >bar.py
$ git add *.py
$ git commit -m foobar
$ git lfs ls-files
7d865e959b * bar.py
b5bb9d8014 * foo.py

Now, if you examine what Git has stored for those files, you'll see the content has been replaced with Git LFS pointers, and that's what git diff is generally going to show you:

$ git show HEAD:foo.py
version https://git-lfs.github.com/spec/v1
oid sha256:b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
size 4

$ git show HEAD:bar.py
version https://git-lfs.github.com/spec/v1
oid sha256:7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730
size 4

$ git diff --no-index -- foo.py bar.py
diff --git a/foo.py b/bar.py
index ad483a9..0161347 100644
--- a/foo.py
+++ b/bar.py
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
+oid sha256:7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730
 size 4

If you want to compare the files, you need to compare the copies in the working tree where Git LFS has replaced the pointer files with the actual content:

$ echo foo.py
foo
$ echo bar.py
bar
diff -u foo.py bar.py 
--- foo.py      2024-05-18 17:38:45
+++ bar.py      2024-05-18 17:35:13
@@ -1 +1 @@
-foo
+bar

If you change one of the files, and commit the change, that will alter the SHA256 OID value in the Git LFS pointer, and that's what git diff is going to show when you compare the Git revisions:

$ echo FOO >foo.py
$ git add foo.py
$ git commit -m FOO
$ git diff HEAD^
diff --git a/foo.py b/foo.py
index ad483a9..005f548 100644
--- a/foo.py
+++ b/foo.py
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
+oid sha256:1706b5c18e4358041b463995efc30f8f721766fab0e018d50d85978b46df013c
 size 4

So if you're expecting to use Git LFS and be able to compare file content, you'll usually need to do that "manually" and not with git diff, since Git now only knows about the pointer files, not the actual file content. The Git LFS client reads those pointers when you perform a Git checkout, fetches the actual file data, and puts it into your working tree.

---

Now I suspect that what you may be hoping to do is to use Git LFS with some pre-existing files in your Git history. As I noted, for that, you would want to run git lfs migrate import; the full documentation explains the various options for that command. But as I also noted, that will change your Git history. Quoting from the documentation:

When converting files to or from Git LFS, the git lfs migrate command will only make changes to your local repository and working copy, never any remotes. This is intentional as the import and export modes are generally "destructive" in the sense that they rewrite your Git history, changing commits and generating new commit SHAs. (The exception is the "no-rewrite" import sub-mode; see "IMPORT WITHOUT REWRITING HISTORY" for details.)

You should therefore always first commit or stash any uncommitted work before using the import or export modes, and then validate the result of the migration before pushing the changes to your remotes, for instance by running the info mode and by examining your rewritten commit history.

So you might do something like this, to rewrite your Git history and convert all existing .py files into Git LFS objects, on all branches (that's what the --everything option means):

$ git clone https://bitbucket.org/weidmann-it-dev/tablink_repository.git
$ cd tablink_repository
$ git lfs migrate import --everything --include '*.py'

But now your local Git repository will have a new commit history, different from your remote's. All *.py files will be Git LFS objects, both in HEAD and all prior commits, on all branches.

If you're really happy with the results, and want to replace the remote's history (for all branches), you can use git push --force --all origin, but please be absolutely, totally certain first!

[marylouwho]

chrisd, thank you so very much for taking your time to help me understand this! Your step by step was just what I needed, and you confirmed some questions as well. Perfect, you answered my issues and I thank you, bless you for your support!!