Add support for git_odb_foreach #185

Closed
book opened this Issue Mar 20, 2017 · 23 comments

Comments

Projects
None yet
2 participants
@book

book commented Mar 20, 2017

I'm trying to get the list of all digests in a repository (and their object type) using Git::Raw. Basically, it would be the equivalent of git cat-file --batch-check --batch-all-objects.

As you told me, it should be doable using git_odb_foreach. So I'm opening an issue to request this functionality in Git::Raw::Odb.

Thanks,

@jacquesg

This comment has been minimized.

Show comment
Hide comment
@jacquesg

jacquesg Mar 20, 2017

Owner

Added in da59b1a and ffa2437. Can you verify that these changes are sufficient?

Owner

jacquesg commented Mar 20, 2017

Added in da59b1a and ffa2437. Can you verify that these changes are sufficient?

@book

This comment has been minimized.

Show comment
Hide comment
@book

book Mar 20, 2017

Here are my various attempts, that mostly produced the expected results:

$ perl -MGit::Raw -E 'Git::Raw::Repository->open(".")->odb->foreach( sub { say $_[0]; return } )'
cc5c34f14d06c688b259a884f15b50f3643f5579
...

And with a slightly longer callback:

$ perl -MGit::Raw -E '
    my $r = Git::Raw::Repository->open(".");
    my @type = qw( bad commit tree blob tag );
    $r->odb->foreach( sub {
        my $id = shift;
        my $o = Git::Raw::Object->lookup( $r, $id );
        say "$id $type[$o->type]";
        return;
    } );
' 
cc5c34f14d06c688b259a884f15b50f3643f5579 commit
...

However, the following code produced an error:

$ perl -Mblib -MGit::Raw -E '
    my $r = Git::Raw::Repository->open(".");
    my @type = qw( bad commit tree blob tag );
    $r->odb->foreach( sub {
        my $id = shift;
        my $o = Git::Raw::Object->lookup( $r, $id );
        say $o->id, " ", $type[$o->type];
        return;
    } );
'
git_pack_foreach_entry callback returned -1

Wrapping the code in an eval block an printing the value of $@ gave:

id is not a valid Git::Raw::Object macro

when $o->id sounded like something I could do.

book commented Mar 20, 2017

Here are my various attempts, that mostly produced the expected results:

$ perl -MGit::Raw -E 'Git::Raw::Repository->open(".")->odb->foreach( sub { say $_[0]; return } )'
cc5c34f14d06c688b259a884f15b50f3643f5579
...

And with a slightly longer callback:

$ perl -MGit::Raw -E '
    my $r = Git::Raw::Repository->open(".");
    my @type = qw( bad commit tree blob tag );
    $r->odb->foreach( sub {
        my $id = shift;
        my $o = Git::Raw::Object->lookup( $r, $id );
        say "$id $type[$o->type]";
        return;
    } );
' 
cc5c34f14d06c688b259a884f15b50f3643f5579 commit
...

However, the following code produced an error:

$ perl -Mblib -MGit::Raw -E '
    my $r = Git::Raw::Repository->open(".");
    my @type = qw( bad commit tree blob tag );
    $r->odb->foreach( sub {
        my $id = shift;
        my $o = Git::Raw::Object->lookup( $r, $id );
        say $o->id, " ", $type[$o->type];
        return;
    } );
'
git_pack_foreach_entry callback returned -1

Wrapping the code in an eval block an printing the value of $@ gave:

id is not a valid Git::Raw::Object macro

when $o->id sounded like something I could do.

@jacquesg

This comment has been minimized.

Show comment
Hide comment
@jacquesg

jacquesg Mar 21, 2017

Owner

You need to return 0 in the callback. A Git::Raw::Object does not currently have an id, but I'll add it.

Owner

jacquesg commented Mar 21, 2017

You need to return 0 in the callback. A Git::Raw::Object does not currently have an id, but I'll add it.

@jacquesg

This comment has been minimized.

Show comment
Hide comment
@jacquesg

jacquesg Mar 21, 2017

Owner

Adding id in #186

Owner

jacquesg commented Mar 21, 2017

Adding id in #186

@jacquesg

This comment has been minimized.

Show comment
Hide comment
@jacquesg

jacquesg Mar 21, 2017

Owner

5c114d9 adds id

Owner

jacquesg commented Mar 21, 2017

5c114d9 adds id

@book

This comment has been minimized.

Show comment
Hide comment
@book

book Mar 21, 2017

It would also be useful to provide size, if it's easily available from the odb.

book commented Mar 21, 2017

It would also be useful to provide size, if it's easily available from the odb.

@book book referenced this issue in book/Git-Database Mar 21, 2017

Closed

Add Git::Raw::Repository backend #4

@jacquesg

This comment has been minimized.

Show comment
Hide comment
@jacquesg

jacquesg Mar 21, 2017

Owner

This isn't readily available nor do I think it makes sense. What you need I think is

if ($o->type == Git::Raw::Object->BLOB)
{
  my $blob = Git::Raw::Blob->lookup ($repo, $o->id);
  my $size = $blob->size;
  my $content = $blob->content;
}
Owner

jacquesg commented Mar 21, 2017

This isn't readily available nor do I think it makes sense. What you need I think is

if ($o->type == Git::Raw::Object->BLOB)
{
  my $blob = Git::Raw::Blob->lookup ($repo, $o->id);
  my $size = $blob->size;
  my $content = $blob->content;
}
@book

This comment has been minimized.

Show comment
Hide comment
@book

book Mar 21, 2017

size and content only exist for Git::Raw::Blob, but in the git object database, all object types have a size (and a raw content):

$ git cat-file --batch-check --batch-all-objects | head
0006f34f06abb6f95a27605053db1e48a196d822 tree 2253
0007c41dbca36ec635ab3db392e9fca005b44286 blob 49286
000d754d2f377fd7c21178aa74be1771baf17171 blob 2880
0018ebbf047d549ee2c8ad6cb8a6dc66540a97a5 blob 1190
001eb6d871742b6f41cb296e57d69d9a6e041db6 commit 255
0020a7762899f84ce5d54a7f2bc58a354c62bc2a blob 8497
0023f95ffdc839888f5d9b9a9d54fd7de15ada15 blob 17660
0029c27da0fbcad9ff8b0959bd05d96e72ebced4 tag 147
002a59d751d49a2e5f8ccd4b93cebf172f7e9100 tree 304
002ce43f5a437c432c9216b6d009bb015b6d303f tree 34

book commented Mar 21, 2017

size and content only exist for Git::Raw::Blob, but in the git object database, all object types have a size (and a raw content):

$ git cat-file --batch-check --batch-all-objects | head
0006f34f06abb6f95a27605053db1e48a196d822 tree 2253
0007c41dbca36ec635ab3db392e9fca005b44286 blob 49286
000d754d2f377fd7c21178aa74be1771baf17171 blob 2880
0018ebbf047d549ee2c8ad6cb8a6dc66540a97a5 blob 1190
001eb6d871742b6f41cb296e57d69d9a6e041db6 commit 255
0020a7762899f84ce5d54a7f2bc58a354c62bc2a blob 8497
0023f95ffdc839888f5d9b9a9d54fd7de15ada15 blob 17660
0029c27da0fbcad9ff8b0959bd05d96e72ebced4 tag 147
002a59d751d49a2e5f8ccd4b93cebf172f7e9100 tree 304
002ce43f5a437c432c9216b6d009bb015b6d303f tree 34
@book

This comment has been minimized.

Show comment
Hide comment
@book

book Mar 21, 2017

Also, I was wondering why Git::Raw::Object->lookup didn't return a Perl obect of the right specialized class, instead of the generic (parent?) class.

book commented Mar 21, 2017

Also, I was wondering why Git::Raw::Object->lookup didn't return a Perl obect of the right specialized class, instead of the generic (parent?) class.

@jacquesg

This comment has been minimized.

Show comment
Hide comment
@jacquesg

jacquesg Mar 21, 2017

Owner

Git::Raw::Repository->lookup will return the specialized object.

Owner

jacquesg commented Mar 21, 2017

Git::Raw::Repository->lookup will return the specialized object.

@jacquesg

This comment has been minimized.

Show comment
Hide comment
@jacquesg

jacquesg Mar 21, 2017

Owner

There does seem to be a way to get the actual size (https://libgit2.github.com/libgit2/#HEAD/group/odb/git_odb_read).

This would require me to add a Git::Raw::Odb::Object

Owner

jacquesg commented Mar 21, 2017

There does seem to be a way to get the actual size (https://libgit2.github.com/libgit2/#HEAD/group/odb/git_odb_read).

This would require me to add a Git::Raw::Odb::Object

@jacquesg

This comment has been minimized.

Show comment
Hide comment
@jacquesg

jacquesg Mar 21, 2017

Owner

Can you try 61b9a50

Owner

jacquesg commented Mar 21, 2017

Can you try 61b9a50

@book

This comment has been minimized.

Show comment
Hide comment
@book

book Mar 21, 2017

Works perfectly! Thanks!

Does the Git::Raw::Odb::Object also give access to the raw object data?
(I assume this is https://libgit2.github.com/libgit2/#HEAD/group/odb/git_odb_object_data).

book commented Mar 21, 2017

Works perfectly! Thanks!

Does the Git::Raw::Odb::Object also give access to the raw object data?
(I assume this is https://libgit2.github.com/libgit2/#HEAD/group/odb/git_odb_object_data).

@jacquesg

This comment has been minimized.

Show comment
Hide comment
@jacquesg

jacquesg Mar 21, 2017

Owner

Added (45a4b89)

Owner

jacquesg commented Mar 21, 2017

Added (45a4b89)

@book

This comment has been minimized.

Show comment
Hide comment
@book

book Mar 21, 2017

data works fine, thanks.

book commented Mar 21, 2017

data works fine, thanks.

@book

This comment has been minimized.

Show comment
Hide comment
@book

book Mar 21, 2017

I noted that read in Git::Raw::Odb does not work with abbreviated digests. Having the same behaviour as lookup in Git::Raw::Repository would be nice.

book commented Mar 21, 2017

I noted that read in Git::Raw::Odb does not work with abbreviated digests. Having the same behaviour as lookup in Git::Raw::Repository would be nice.

@jacquesg

This comment has been minimized.

Show comment
Hide comment
@jacquesg

jacquesg Mar 22, 2017

Owner

Fixed (b155b90)

Owner

jacquesg commented Mar 22, 2017

Fixed (b155b90)

@book

This comment has been minimized.

Show comment
Hide comment
@book

book Mar 22, 2017

Thanks!

However, this last patch has an issue, when trying to obtain ambiguous digests:

$ git cat-file --batch-check --batch-all-objects
577ecc210a55a5da10552a4415c4cbb5e321039b blob 988
577eccaacd6343158463f9eaefa19dec78358437 blob 224
65237d4ff8bddefebb2d4801e796f94fa9a9bbeb tree 66
9f0363e979a368db9748fb93278ab91a2152aa71 commit 255
$ perl -MGit::Raw -le '$r=Git::Raw::Repository->open(".");print $r->lookup(shift)' 577ecc
Ambiguous SHA1 prefix - found multiple offsets for pack entry at -e line 1
$ perl -MGit::Raw -le '$r=Git::Raw::Repository->open(".");print $r->odb->read(shift)' 577ecc
Git::Raw::Odb::Object=SCALAR(0x55af77bdeb30)
Segmentation fault

book commented Mar 22, 2017

Thanks!

However, this last patch has an issue, when trying to obtain ambiguous digests:

$ git cat-file --batch-check --batch-all-objects
577ecc210a55a5da10552a4415c4cbb5e321039b blob 988
577eccaacd6343158463f9eaefa19dec78358437 blob 224
65237d4ff8bddefebb2d4801e796f94fa9a9bbeb tree 66
9f0363e979a368db9748fb93278ab91a2152aa71 commit 255
$ perl -MGit::Raw -le '$r=Git::Raw::Repository->open(".");print $r->lookup(shift)' 577ecc
Ambiguous SHA1 prefix - found multiple offsets for pack entry at -e line 1
$ perl -MGit::Raw -le '$r=Git::Raw::Repository->open(".");print $r->odb->read(shift)' 577ecc
Git::Raw::Odb::Object=SCALAR(0x55af77bdeb30)
Segmentation fault
@jacquesg

This comment has been minimized.

Show comment
Hide comment
@jacquesg

jacquesg Mar 22, 2017

Owner

Yep, I missed a line :(

Owner

jacquesg commented Mar 22, 2017

Yep, I missed a line :(

@jacquesg

This comment has been minimized.

Show comment
Hide comment
@jacquesg

jacquesg Mar 22, 2017

Owner

Fixed (fa99da8)

Owner

jacquesg commented Mar 22, 2017

Fixed (fa99da8)

@book

This comment has been minimized.

Show comment
Hide comment
@book

book Mar 22, 2017

Everything works perfectly now! Thanks for being so reactive.

I'll now wait for the next release. :-)

book commented Mar 22, 2017

Everything works perfectly now! Thanks for being so reactive.

I'll now wait for the next release. :-)

@book book closed this Mar 22, 2017

@jacquesg

This comment has been minimized.

Show comment
Hide comment
@jacquesg

jacquesg Mar 22, 2017

Owner

I'll create a release within the next few minutes

Owner

jacquesg commented Mar 22, 2017

I'll create a release within the next few minutes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment