[GSOC][RFC] ref-filter: add %(rest) atom #976

adlternative · 2021-06-08T13:12:17Z

In order to let git cat-file --batch reuse ref-filter logic,
the atom %(rest) and %(raw) are added to ref-filter.

Change from last version:

Let --format=%(raw) re-support --perl.
Correct some errors in the test.
Add the reason why strbuf is not used
to replace atom_value's member s in the
commit message.

cc: Ævar Arnfjörð Bjarmason avarab@gmail.com
cc: Junio C Hamano gitster@pobox.com
cc: Christian Couder christian.couder@gmail.com
cc: Hariom Verma hariom18599@gmail.com
cc: Bagas Sanjaya bagasdotme@gmail.com

adlternative · 2021-06-08T14:36:56Z

/submit

gitgitgadget · 2021-06-08T14:38:27Z

Submitted as pull.976.git.1623163082.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git pr-976/adlternative/ref-filter-texconv-filters-v2-v1

To fetch this version to local tag pr-976/adlternative/ref-filter-texconv-filters-v2-v1:

git fetch --no-tags https://github.com/gitgitgadget/git tag pr-976/adlternative/ref-filter-texconv-filters-v2-v1

gitgitgadget · 2021-06-09T03:16:24Z

This branch is now known as zh/ref-filter-raw-data.

gitgitgadget · 2021-06-09T03:16:25Z

This patch series was integrated into seen via git@c281d03.

gitgitgadget · 2021-06-09T03:40:43Z

On the Git mailing list, Junio C Hamano wrote (reply to this):

"ZheNing Hu via GitGitGadget" <gitgitgadget@gmail.com> writes:

> In order to let git cat-file --batch reuse ref-filter logic, in this
> version, the atom %(rest) is added to ref-filter.
>
> Change from last version:
>
>  1. Deleted two commit about %(raw:textconv) and %(raw:filters), their
>     construction will be after letting cat-file --batch uses ref-filter
>     logic.
>  2. Exchange the order of the definition of "cmp_fn" and the declaration of
>     "a_size" and "b_size" in cmp_ref_sorting(), to avoid breaking
>     -Wdecl-after-stmt.
>  3. Delete --rest option, "git for-each-ref" family will reject %(rest) by
>     default.
>  4. Add GPG prefix to some %(raw) tests.
>
> ZheNing Hu (4):
>   [GSOC] ref-filter: add obj-type check in grab contents
>   [GSOC] ref-filter: add %(raw) atom
>   [GSOC] ref-filter: use non-const ref_format in *_atom_parser()
>   [GSOC] ref-filter: add %(rest) atom

Thanks, will replace.

gitgitgadget · 2021-06-09T03:58:40Z

On the Git mailing list, ZheNing Hu wrote (reply to this):

Junio C Hamano <gitster@pobox.com> 于2021年6月9日周三 上午11:39写道：
>
> "ZheNing Hu via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > In order to let git cat-file --batch reuse ref-filter logic, in this
> > version, the atom %(rest) is added to ref-filter.
> >
> > Change from last version:
> >
> >  1. Deleted two commit about %(raw:textconv) and %(raw:filters), their
> >     construction will be after letting cat-file --batch uses ref-filter
> >     logic.
> >  2. Exchange the order of the definition of "cmp_fn" and the declaration of
> >     "a_size" and "b_size" in cmp_ref_sorting(), to avoid breaking
> >     -Wdecl-after-stmt.
> >  3. Delete --rest option, "git for-each-ref" family will reject %(rest) by
> >     default.
> >  4. Add GPG prefix to some %(raw) tests.
> >
> > ZheNing Hu (4):
> >   [GSOC] ref-filter: add obj-type check in grab contents
> >   [GSOC] ref-filter: add %(raw) atom
> >   [GSOC] ref-filter: use non-const ref_format in *_atom_parser()
> >   [GSOC] ref-filter: add %(rest) atom
>
> Thanks, will replace.

It’s worth mentioning that this patch still fails the github CI test. [1]

t6300-for-each-ref.sh:
274: Verify sorts with raw
275: Verify sorts with raw:size
278: empty raw refs with %(if)

There is no such breakage on my local machine.

[1] https://github.com/gitgitgadget/git/pull/976/checks?check_run_id=2774963705

gitgitgadget · 2021-06-09T07:29:32Z

On the Git mailing list, Junio C Hamano wrote (reply to this):

ZheNing Hu <adlternative@gmail.com> writes:

> It’s worth mentioning that this patch still fails the github CI test. [1]
>
> t6300-for-each-ref.sh:
> 274: Verify sorts with raw
> 275: Verify sorts with raw:size
> 278: empty raw refs with %(if)
>
> There is no such breakage on my local machine.
>
> [1] https://github.com/gitgitgadget/git/pull/976/checks?check_run_id=2774963705

It's one of the two 32-bit jobs, I think.  If the other one fails
the same way, that means the code does not work on 32-bit systems,
most likely something in your code assumes certain types are 64-bit
when they aren't.

adlternative · 2021-06-09T16:10:54Z

/submit

gitgitgadget · 2021-06-09T16:12:28Z

Submitted as pull.976.v2.git.1623255121.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git pr-976/adlternative/ref-filter-texconv-filters-v2-v2

To fetch this version to local tag pr-976/adlternative/ref-filter-texconv-filters-v2-v2:

git fetch --no-tags https://github.com/gitgitgadget/git tag pr-976/adlternative/ref-filter-texconv-filters-v2-v2

gitgitgadget · 2021-06-10T06:54:57Z

There was a status update in the "Cooking" section about the branch zh/ref-filter-raw-data on the Git mailing list:

Prepare the "ref-filter" machinery that drives the "--format"
option of "git for-each-ref" and its friends to be used in "git
cat-file --batch".

gitgitgadget · 2021-06-11T08:15:15Z

This patch series was integrated into seen via git@59a0f86.

gitgitgadget · 2021-06-14T05:46:08Z

This patch series was integrated into seen via git@881fcbd.

gitgitgadget · 2021-06-14T07:17:03Z

This patch series was integrated into seen via git@f6bbc11.

gitgitgadget · 2021-06-15T08:07:08Z

This patch series was integrated into seen via git@c5de0bc.

gitgitgadget · 2021-06-15T08:22:49Z

There was a status update in the "Cooking" section about the branch zh/ref-filter-raw-data on the Git mailing list:

Prepare the "ref-filter" machinery that drives the "--format"
option of "git for-each-ref" and its friends to be used in "git
cat-file --batch".

gitgitgadget · 2021-06-16T07:00:43Z

This patch series was integrated into seen via git@97d0d54.

gitgitgadget · 2021-06-17T03:10:17Z

There was a status update in the "Cooking" section about the branch zh/ref-filter-raw-data on the Git mailing list:

Prepare the "ref-filter" machinery that drives the "--format"
option of "git for-each-ref" and its friends to be used in "git
cat-file --batch".

This conflicts with the other topic on "cat-file --batch" by the
same author.

Only tag and commit objects use `grab_sub_body_contents()` to grab object contents in the current codebase. We want to teach the function to also handle blobs and trees to get their raw data, without parsing a blob (whose contents looks like a commit or a tag) incorrectly as a commit or a tag. Skip the block of code that is specific to handling commits and tags early when the given object is of a wrong type to help later addition to handle other types of objects in this function. Mentored-by: Christian Couder <christian.couder@gmail.com> Mentored-by: Hariom Verma <hariom18599@gmail.com> Helped-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: ZheNing Hu <adlternative@gmail.com>

Add new formatting option `%(raw)`, which will print the raw object data without any changes. It will help further to migrate all cat-file formatting logic from cat-file to ref-filter. The raw data of blob, tree objects may contain '\0', but most of the logic in `ref-filter` depends on the output of the atom being text (specifically, no embedded NULs in it). E.g. `quote_formatting()` use `strbuf_addstr()` or `*._quote_buf()` add the data to the buffer. The raw data of a tree object is `100644 one\0...`, only the `100644 one` will be added to the buffer, which is incorrect. Therefore, we need to find a way to record the length of the atom_value's member `s`. Although strbuf can already record the string and its length, if we want to replace the type of atom_value's member `s` with strbuf, many places in ref-filter that are filled with dynamically allocated mermory in `v->s` are not easy to replace. At the same time, we need to check if `v->s == NULL` in populate_value(), and strbuf cannot easily distinguish NULL and empty strings, but c-style "const char *" can do it. So add a new member in `struct atom_value`: `s_size`, which can record raw object size, it can help us add raw object data to the buffer or compare two buffers which contain raw object data. Beyond, `--format=%(raw)` cannot be used with `--python`, `--shell`, `--tcl`, `--perl` because if our binary raw data is passed to a variable in the host language, the host language may not support arbitrary binary data in the variables of its string type. Mentored-by: Christian Couder <christian.couder@gmail.com> Mentored-by: Hariom Verma <hariom18599@gmail.com> Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Helped-by: Felipe Contreras <felipe.contreras@gmail.com> Helped-by: Phillip Wood <phillip.wood@dunelm.org.uk> Helped-by: Junio C Hamano <gitster@pobox.com> Based-on-patch-by: Olga Telezhnaya <olyatelezhnaya@gmail.com> Signed-off-by: ZheNing Hu <adlternative@gmail.com>

Because the perl language can handle binary data correctly, add the function perl_quote_buf_with_len(), which can specify the length of the data and prevent the data from being truncated at '\0' to help `--format="%(raw)"` re-support `--perl`. Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: ZheNing Hu <adlternative@gmail.com>

Use non-const ref_format in *_atom_parser(), which can help us modify the members of ref_format in *_atom_parser(). Mentored-by: Christian Couder <christian.couder@gmail.com> Mentored-by: Hariom Verma <hariom18599@gmail.com> Signed-off-by: ZheNing Hu <adlternative@gmail.com>

In order to let "cat-file --batch=%(rest)" use the ref-filter interface, add %(rest) atom for ref-filter. "git for-each-ref", "git branch", "git tag" and "git verify-tag" will reject %(rest) by default. Mentored-by: Christian Couder <christian.couder@gmail.com> Mentored-by: Hariom Verma <hariom18599@gmail.com> Signed-off-by: ZheNing Hu <adlternative@gmail.com>

gitgitgadget · 2021-06-30T22:59:41Z

There was a status update in the "Cooking" section about the branch zh/ref-filter-raw-data on the Git mailing list:

Prepare the "ref-filter" machinery that drives the "--format"
option of "git for-each-ref" and its friends to be used in "git
cat-file --batch".

This conflicts with the other topic on "cat-file --batch" by the
same author.

gitgitgadget · 2021-07-01T22:02:30Z

This patch series was integrated into seen via git@3c525b5.

gitgitgadget · 2021-07-02T21:33:19Z

This patch series was integrated into seen via git@ce080dd.

gitgitgadget · 2021-07-03T00:35:52Z

This patch series was integrated into seen via git@c8db213.

gitgitgadget · 2021-08-04T21:48:29Z

This patch series was integrated into seen via git@01d69c2.

gitgitgadget · 2021-08-04T22:21:01Z

This patch series was integrated into seen via git@e8a6112.

gitgitgadget · 2021-08-04T22:21:02Z

This patch series was integrated into next via git@4c4529d.

gitgitgadget · 2021-08-05T23:52:09Z

This patch series was integrated into seen via git@72ee1a2.

gitgitgadget · 2021-08-06T22:22:14Z

This patch series was integrated into seen via git@30b8edb.

gitgitgadget · 2021-08-06T23:21:17Z

This patch series was integrated into seen via git@0588156.

gitgitgadget · 2021-08-07T00:21:17Z

There was a status update in the "Cooking" section about the branch zh/ref-filter-raw-data on the Git mailing list:

Prepare the "ref-filter" machinery that drives the "--format"
option of "git for-each-ref" and its friends to be used in "git
cat-file --batch".

Will cook in 'next'.

gitgitgadget · 2021-08-09T20:15:03Z

This patch series was integrated into seen via git@3cd6276.

gitgitgadget · 2021-08-09T22:06:47Z

This patch series was integrated into seen via git@076856e.

gitgitgadget · 2021-08-11T01:15:48Z

There was a status update in the "Cooking" section about the branch zh/ref-filter-raw-data on the Git mailing list:

Prepare the "ref-filter" machinery that drives the "--format"
option of "git for-each-ref" and its friends to be used in "git
cat-file --batch".

Will cook in 'next'.

gitgitgadget · 2021-08-11T18:58:19Z

This patch series was integrated into seen via git@d342a15.

gitgitgadget · 2021-08-11T20:30:52Z

This patch series was integrated into seen via git@18426b1.

gitgitgadget · 2021-08-12T22:58:32Z

There was a status update in the "Cooking" section about the branch zh/ref-filter-raw-data on the Git mailing list:

Prepare the "ref-filter" machinery that drives the "--format"
option of "git for-each-ref" and its friends to be used in "git
cat-file --batch".

Will cook in 'next'.

gitgitgadget · 2021-08-15T21:20:54Z

This patch series was integrated into seen via git@4c4529d.

gitgitgadget · 2021-08-15T22:01:03Z

This patch series was integrated into seen via git@aa00842.

gitgitgadget · 2021-08-16T21:12:20Z

This patch series was integrated into seen via git@884fcb3.

gitgitgadget · 2021-08-16T23:09:25Z

There was a status update in the "Cooking" section about the branch zh/ref-filter-raw-data on the Git mailing list:

Prepare the "ref-filter" machinery that drives the "--format"
option of "git for-each-ref" and its friends to be used in "git
cat-file --batch".

Will cook in 'next'.

gitgitgadget · 2021-08-19T21:20:11Z

This patch series was integrated into seen via git@1bd338f.

gitgitgadget · 2021-08-20T00:13:51Z

This patch series was integrated into seen via git@50faa16.

gitgitgadget · 2021-08-20T17:49:27Z

There was a status update in the "Cooking" section about the branch zh/ref-filter-raw-data on the Git mailing list:

Prepare the "ref-filter" machinery that drives the "--format"
option of "git for-each-ref" and its friends to be used in "git
cat-file --batch".

Will cook in 'next'.

gitgitgadget · 2021-08-23T21:48:05Z

This patch series was integrated into seen via git@1b66e8e.

gitgitgadget · 2021-08-23T21:48:06Z

This patch series was integrated into next via git@5c06fb4.

gitgitgadget · 2021-08-23T21:59:06Z

There was a status update in the "Cooking" section about the branch zh/ref-filter-raw-data on the Git mailing list:

Originally merged to 'next' on 2021-08-04

Prepare the "ref-filter" machinery that drives the "--format"
option of "git for-each-ref" and its friends to be used in "git
cat-file --batch".

Will merge to 'master'.

gitgitgadget · 2021-08-24T23:32:30Z

This patch series was integrated into seen via git@bda891e.

gitgitgadget · 2021-08-24T23:32:30Z

This patch series was integrated into next via git@bda891e.

gitgitgadget · 2021-08-24T23:32:31Z

This patch series was integrated into master via git@bda891e.

gitgitgadget · 2021-08-24T23:32:33Z

Closed via bda891e.

adlternative force-pushed the ref-filter-texconv-filters-v2 branch from ce1c28a to 5e95292 Compare June 8, 2021 14:12

gitgitgadget bot added the seen label Jun 9, 2021

adlternative force-pushed the ref-filter-texconv-filters-v2 branch from 5e95292 to 5a5b5f7 Compare June 9, 2021 15:21

adlternative added 5 commits June 19, 2021 12:08

adlternative force-pushed the ref-filter-texconv-filters-v2 branch from 5a5b5f7 to 39a0d93 Compare June 19, 2021 05:33

gitgitgadget bot added the next label Aug 4, 2021

gitgitgadget bot added the master label Aug 24, 2021

gitgitgadget bot closed this Aug 24, 2021

[GSOC][RFC] ref-filter: add %(rest) atom #976

[GSOC][RFC] ref-filter: add %(rest) atom #976

Conversation

adlternative commented Jun 8, 2021 • edited Loading

adlternative commented Jun 8, 2021

gitgitgadget bot commented Jun 8, 2021

gitgitgadget bot commented Jun 9, 2021

gitgitgadget bot commented Jun 9, 2021

gitgitgadget bot commented Jun 9, 2021

gitgitgadget bot commented Jun 9, 2021

gitgitgadget bot commented Jun 9, 2021

adlternative commented Jun 9, 2021

gitgitgadget bot commented Jun 9, 2021

gitgitgadget bot commented Jun 10, 2021

gitgitgadget bot commented Jun 11, 2021

gitgitgadget bot commented Jun 14, 2021

gitgitgadget bot commented Jun 14, 2021

gitgitgadget bot commented Jun 15, 2021

gitgitgadget bot commented Jun 15, 2021

gitgitgadget bot commented Jun 16, 2021

gitgitgadget bot commented Jun 17, 2021

gitgitgadget bot commented Jun 30, 2021

gitgitgadget bot commented Jul 1, 2021

gitgitgadget bot commented Jul 2, 2021

gitgitgadget bot commented Jul 3, 2021

gitgitgadget bot commented Aug 4, 2021

gitgitgadget bot commented Aug 4, 2021

gitgitgadget bot commented Aug 4, 2021

gitgitgadget bot commented Aug 5, 2021

gitgitgadget bot commented Aug 6, 2021

gitgitgadget bot commented Aug 6, 2021

gitgitgadget bot commented Aug 7, 2021

gitgitgadget bot commented Aug 9, 2021

gitgitgadget bot commented Aug 9, 2021

gitgitgadget bot commented Aug 11, 2021

gitgitgadget bot commented Aug 11, 2021

gitgitgadget bot commented Aug 11, 2021

gitgitgadget bot commented Aug 12, 2021

gitgitgadget bot commented Aug 15, 2021

gitgitgadget bot commented Aug 15, 2021

gitgitgadget bot commented Aug 16, 2021

gitgitgadget bot commented Aug 16, 2021

gitgitgadget bot commented Aug 19, 2021

gitgitgadget bot commented Aug 20, 2021

gitgitgadget bot commented Aug 20, 2021

gitgitgadget bot commented Aug 23, 2021

gitgitgadget bot commented Aug 23, 2021

gitgitgadget bot commented Aug 23, 2021

gitgitgadget bot commented Aug 24, 2021

gitgitgadget bot commented Aug 24, 2021

gitgitgadget bot commented Aug 24, 2021

gitgitgadget bot commented Aug 24, 2021

adlternative commented Jun 8, 2021 •

edited

Loading