Full implementation of standard signature syntax in container signatures #305

jcharlet · 2019-11-07T11:06:24Z

These classes can parse PRONOM and container signature syntax, producing an Abstract Syntax Tree (AST). A compiler for the AST produces ByteSequence, SubSequence and SideFragment objects that can be used in DROID to search for those signatures.

solves #237

- currently splits subsequences as it goes - not sure if this is the right approach. The parser should probably just represent the expression passed in as a single parse tree. The compiler should worry about how to split this up.

the results of compilation. These either return immutable objects, or make defensive copies.

* There are four PRONOM signatures which currently cannot be parsed, as they use unofficial syntax. Waiting for TNA to respond on what they want to do / support for the PRONOM language officially.

* This is used to test that the PRONOM parser is capable of reading known good and real-world signatures.

* There are four PRONOM signatures which currently cannot be parsed, as they use unofficial syntax. Waiting for TNA to respond on what they want to do / support for the PRONOM language officially.

* Some of these are failing at present as they aren't yet supported by the parser. - string literals in ranges ['a'-'z'] - arbitrary sets of bytes [2227] - the & bitwise operator [&01] - ranges use both hyphens and colons to separate them in different signatures.

* also some code cleanup.

…sigs. * Haven't validated that it is correct yet, only that it can process all of them without error.

jcharlet · 2019-11-07T11:16:55Z

@nishihatapalmer the demo for rebasing:
Your commits:

My simple rebase command: worked right away without any conflict:
from feature branch
git rebase master
equivalent to git pull --rebase master

The result:

I think it's better for maintenance because when we merge back onto master, we don't have crazy branches all other the place. you can also merge with rebase on github and have a clean history without even needing a merge commit.

jcharlet · 2019-11-07T11:18:55Z

if you want, I'll let you do it and push to upstream to practice :) . You will have to force push it to override the history.

git checkout droidsyntaxparser
git rebase master
git push upstream droidsyntaxparser -f

I'll start fixing the checkstyle errors for travis and add documentation, to better understand your work .

* Use a strategy pattern to encapsulate different strategies on what kinds of elements can appear in anchor sequences. * PRONOM can only support bytes in anchors. * DROID can support anything in anchors, but it can lead to performance problems if sets or bitmasks are too big, so we limit the size. * If we can't find an anchoring sequence for DROID given the size limits, we remove all restrictions on size and look again. The advantage of this approach is that we get strict compliance for PRONOM, but have a two-fold strategy for DROID - one which will probably give better performance than the PRONOM strategy, and a fallback position for signatures for which no anchoring sequence could be found (there are three signatures which fall into this category at present - the [&01] signatures with no other bytes in them.

- currently splits subsequences as it goes - not sure if this is the right approach. The parser should probably just represent the expression passed in as a single parse tree. The compiler should worry about how to split this up.

the results of compilation. These either return immutable objects, or make defensive copies.

* There are four PRONOM signatures which currently cannot be parsed, as they use unofficial syntax. Waiting for TNA to respond on what they want to do / support for the PRONOM language officially.

* This is used to test that the PRONOM parser is capable of reading known good and real-world signatures.

* There are four PRONOM signatures which currently cannot be parsed, as they use unofficial syntax. Waiting for TNA to respond on what they want to do / support for the PRONOM language officially.

* Some of these are failing at present as they aren't yet supported by the parser. - string literals in ranges ['a'-'z'] - arbitrary sets of bytes [2227] - the & bitwise operator [&01] - ranges use both hyphens and colons to separate them in different signatures.

nishihatapalmer · 2019-11-26T15:16:02Z

This doesn't touch on the differences between binary and container syntax. DROID itself doesn't care, but PRONOM won't understand container syntax.

We should probably link to a description of the syntax somewhere, and the fact that you can use all of the syntax in either binary or container signatures if you like (but PRONOM won't be able to parse them for binary signatures if you want to submit them to TNA). Of course, nothing stops TNA using sigtool to rewrite a container signatures syntax in binary compatible format.

jcharlet · 2019-11-26T17:03:00Z

Aaaaallright sorry @nishihatapalmer for the confusion, some things are indeed a bit clearer, and I better understood your previous comment. Thanks for taking the time to explain!

We should probably link to a description of the syntax somewhere,

I mentioned in README.md "PRONOM Syntax also provides details on the regular expression syntax supported by DROID.", which I am going to improve with a relative link on github to the .md file. Isn't it sufficient?

and the fact that you can use all of the syntax in either binary or container signatures if you like (but PRONOM won't be able to parse them for binary signatures if you want to submit them to TNA). Of course, nothing stops TNA using sigtool to rewrite a container signatures syntax in binary compatible format.

what about adding the following paragraph, after what you offered to put in Signatures management and before the introduction to sigtool "To work further on signatures, we provide Sigtool [...]"

The full syntax can be used in either binary or container signatures. However, if you use the new syntax, to submit them to TNA and get those signatures included in PRONOM registry, you will need to compile those signatures for PRONOM using Sigtool.

nishihatapalmer · 2019-11-26T20:59:50Z

I'm going to be pedantic here, but it's kind of important.

Only binary signatures which are to be submitted to TNA need to be in binary format.

Container signatures submitted to TNA can - and should - use the full syntax, since PRONOM doesn't compile those, and it allows things which binary signatures don't support.

nishihatapalmer · 2019-11-26T21:10:21Z

And if you don't intend to submit local binary signatures to TNA, you can use the full container syntax in the Sequence attribute of a binary signature ByteSequence.

jcharlet · 2019-11-27T16:16:46Z

@nishihatapalmer could you please refine the README? I won't be as clear as you can be, I'd rather let you do it if you have some time, and that will save us from going back and forth here.

nishihatapalmer · 2019-11-27T21:02:50Z

Sure, but may be a few days before I'll have time.

…

On Wed, 27 Nov 2019, 16:16 Jeremie Charlet, ***@***.***> wrote: @nishihatapalmer <https://github.com/nishihatapalmer> could you please refine the README? I won't be as clear as you can be, I'd rather let you do it if you have some time, and that will save us from going back and forth here. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#305?email_source=notifications&email_token=ABBY4JHT7JCGKKO3KLXVTT3QV2MO7A5CNFSM4JKE6MBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFKAORQ#issuecomment-559155014>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBY4JCXJNJBHMR26EETXLLQV2MO7ANCNFSM4JKE6MBA> .

* explanation of what kinds of signature exist. * explanation of sigtool capabilities * explanation of simpler XML format.

nishihatapalmer · 2019-11-28T13:56:12Z

OK, I've posted some updates to the README.md file in my droidsyntaxparser branch. I'm tempted to simplify it a bit - remove the "Types" and "Syntax" sections (or considerably shorten them, or move them into another markdown file). The real new capabilities are "sigtool" and "Simpler signature XML". I kind of feel that putting in a quick education on what kinds of signatures exist in the middle of a high level overview of what's in the DROID repo is the wrong place for this - but telling users about sigtool and simpler signature XML is OK, as they're big and important changes.

…

On Wed, 27 Nov 2019 at 21:02, Matt Palmer ***@***.***> wrote: Sure, but may be a few days before I'll have time. On Wed, 27 Nov 2019, 16:16 Jeremie Charlet, ***@***.***> wrote: > @nishihatapalmer <https://github.com/nishihatapalmer> could you please > refine the README? I won't be as clear as you can be, I'd rather let you do > it if you have some time, and that will save us from going back and forth > here. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#305?email_source=notifications&email_token=ABBY4JHT7JCGKKO3KLXVTT3QV2MO7A5CNFSM4JKE6MBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFKAORQ#issuecomment-559155014>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABBY4JCXJNJBHMR26EETXLLQV2MO7ANCNFSM4JKE6MBA> > . >

jcharlet · 2019-11-28T14:53:39Z

thanks for those changes!

what about

renaming PRONOM Syntax to signature syntax.
moving types and syntax section in that signature syntax readme.
add a paragraph below Since version 6.5, DROID adds some new capabilities to support developing and testing signatures. :

Signature Syntax provides details on the types of signatures and regular expression syntax supported by DROID.

nishihatapalmer · 2019-11-28T14:59:26Z

Yup, those sound perfect.

…

On Thu, 28 Nov 2019, 14:53 Jeremie Charlet, ***@***.***> wrote: thanks for those changes! what about - renaming PRONOM Syntax to signature syntax. - moving types and syntax section in that signature syntax readme. - add a paragraph below Since version 6.5, DROID adds some new capabilities to support developing and testing signatures. : Signature Syntax provides details on the types of signatures and regular expression syntax supported by DROID. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#305?email_source=notifications&email_token=ABBY4JBJO6FSFBY3UE7PV3DQV7LPJA5CNFSM4JKE6MBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFM2J2I#issuecomment-559523049>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBY4JCUSYJPP2UQXZVSLNLQV7LPJANCNFSM4JKE6MBA> .

nishihatapalmer · 2019-11-28T15:03:35Z

As long as we retain a link to the signature syntax MD from the readme

…

On Thu, 28 Nov 2019, 14:59 Matt Palmer, ***@***.***> wrote: Yup, those sound perfect. On Thu, 28 Nov 2019, 14:53 Jeremie Charlet, ***@***.***> wrote: > thanks for those changes! > > what about > > - renaming PRONOM Syntax to signature syntax. > - moving types and syntax section in that signature syntax readme. > - add a paragraph below Since version 6.5, DROID adds some new > capabilities to support developing and testing signatures. : > > Signature Syntax provides details on the types of signatures and regular > expression syntax supported by DROID. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#305?email_source=notifications&email_token=ABBY4JBJO6FSFBY3UE7PV3DQV7LPJA5CNFSM4JKE6MBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFM2J2I#issuecomment-559523049>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABBY4JCUSYJPP2UQXZVSLNLQV7LPJANCNFSM4JKE6MBA> > . >

nishihatapalmer · 2019-11-28T15:06:27Z

I'm not sure the current link is actually working come to think of it...

…

On Thu, 28 Nov 2019, 15:03 Matt Palmer, ***@***.***> wrote: As long as we retain a link to the signature syntax MD from the readme On Thu, 28 Nov 2019, 14:59 Matt Palmer, ***@***.***> wrote: > Yup, those sound perfect. > > On Thu, 28 Nov 2019, 14:53 Jeremie Charlet, ***@***.***> > wrote: > >> thanks for those changes! >> >> what about >> >> - renaming PRONOM Syntax to signature syntax. >> - moving types and syntax section in that signature syntax readme. >> - add a paragraph below Since version 6.5, DROID adds some new >> capabilities to support developing and testing signatures. : >> >> Signature Syntax provides details on the types of signatures and regular >> expression syntax supported by DROID. >> >> — >> You are receiving this because you were mentioned. >> Reply to this email directly, view it on GitHub >> <#305?email_source=notifications&email_token=ABBY4JBJO6FSFBY3UE7PV3DQV7LPJA5CNFSM4JKE6MBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFM2J2I#issuecomment-559523049>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/ABBY4JCUSYJPP2UQXZVSLNLQV7LPJANCNFSM4JKE6MBA> >> . >> >

jcharlet · 2019-11-28T15:13:41Z

fixed https://github.com/digital-preservation/droid/blob/droidsyntaxparser/README.md

I'm not sure the current link is actually working come to think of it...
…
On Thu, 28 Nov 2019, 15:03 Matt Palmer, @.> wrote: As long as we retain a link to the signature syntax MD from the readme On Thu, 28 Nov 2019, 14:59 Matt Palmer, @.> wrote: > Yup, those sound perfect. > > On Thu, 28 Nov 2019, 14:53 Jeremie Charlet, @.***> > wrote: > >> thanks for those changes! >> >> what about >> >> - renaming PRONOM Syntax to signature syntax. >> - moving types and syntax section in that signature syntax readme. >> - add a paragraph below Since version 6.5, DROID adds some new >> capabilities to support developing and testing signatures. : >> >> Signature Syntax provides details on the types of signatures and regular >> expression syntax supported by DROID. >> >> — >> You are receiving this because you were mentioned. >> Reply to this email directly, view it on GitHub >> <#305?email_source=notifications&email_token=ABBY4JBJO6FSFBY3UE7PV3DQV7LPJA5CNFSM4JKE6MBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFM2J2I#issuecomment-559523049>, >> or unsubscribe >> https://github.com/notifications/unsubscribe-auth/ABBY4JCUSYJPP2UQXZVSLNLQV7LPJANCNFSM4JKE6MBA >> . >> >

…section in README to signature syntax doc

jcharlet · 2019-11-28T15:21:20Z

done!

nishihatapalmer · 2019-11-28T15:48:25Z

Looks great By the way, why does the standard non jre version of DROID have a Unix suffix on the filename? It should be completely platform independent.

…

On Thu, 28 Nov 2019, 15:21 Jeremie Charlet, ***@***.***> wrote: done! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#305?email_source=notifications&email_token=ABBY4JEQ6473OAYMOSOXESTQV7OXDA5CNFSM4JKE6MBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFM4QIQ#issuecomment-559532066>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBY4JHEAC2VLIZCEUDJOGTQV7OXDANCNFSM4JKE6MBA> .

nishihatapalmer · 2019-11-28T15:52:52Z

One more comment, the signature syntax file now talks about the sigtool "below". It's not below now as that's still in the original readme. Maybe change to a link back to the readme or the using sigtool documentation?

…

On Thu, 28 Nov 2019, 15:48 Matt Palmer, ***@***.***> wrote: Looks great By the way, why does the standard non jre version of DROID have a Unix suffix on the filename? It should be completely platform independent. On Thu, 28 Nov 2019, 15:21 Jeremie Charlet, ***@***.***> wrote: > done! > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#305?email_source=notifications&email_token=ABBY4JEQ6473OAYMOSOXESTQV7OXDA5CNFSM4JKE6MBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFM4QIQ#issuecomment-559532066>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABBY4JHEAC2VLIZCEUDJOGTQV7OXDANCNFSM4JKE6MBA> > . >

jcharlet · 2019-11-28T16:01:11Z

Looks great By the way, why does the standard non jre version of DROID have a Unix suffix on the filename? It should be completely platform independent.

mmh it used to because we had a windows version with jre and a unix version without jre.
But shouldn't be the case anymore, since we recently put back the windows stuff in the bundle without jre and thus removed the unix suffix.
just checked on master and droidsyntaxparser on the core repo here, and it looks good to me https://github.com/digital-preservation/droid/blob/master/droid-binary/assembly.xml

maybe I missed something, where did you see that?

jcharlet · 2019-11-28T16:05:58Z

One more comment, the signature syntax file now talks about the sigtool "below". It's not below now as that's still in the original readme. Maybe change to a link back to the readme or the using sigtool documentation?

sorry for that.. fixed

nishihatapalmer added 16 commits September 1, 2019 13:25

Initial work on parser and compiler for DROID syntax - unfinished!

4b2ea2b

First version of ByteSequenceParser.

1fa3036

- currently splits subsequences as it goes - not sure if this is the right approach. The parser should probably just represent the expression passed in as a single parse tree. The compiler should worry about how to split this up.

First commit for anchor and compiler, some changes in parser.

8782c2b

Add safe accessor methods for internal objects to allow testing

0e43d2a

the results of compilation. These either return immutable objects, or make defensive copies.

First complete support for official PRONOM syntax (tested).

f4a4909

* There are four PRONOM signatures which currently cannot be parsed, as they use unofficial syntax. Waiting for TNA to respond on what they want to do / support for the PRONOM language officially.

Add a list of all known PRONOM ByteSequences for signature v91.

bf60212

* This is used to test that the PRONOM parser is capable of reading known good and real-world signatures.

Tests for first complete support for official PRONOM syntax.

42b05c5

* There are four PRONOM signatures which currently cannot be parsed, as they use unofficial syntax. Waiting for TNA to respond on what they want to do / support for the PRONOM language officially.

container signature sequences.

d9b7ca0

Add support for both colon and hyphen separating ranges.

5cfac49

* also some code cleanup.

Add support for arbitrary byte sets and the & operator.

873bdaf

Modify tests to support arbitrary sets.

18224d8

Allow whitespace inside sets.

97bb02b

First ByteSequenceCompiler that can compile all PRONOM and container …

d6a2e27

…sigs. * Haven't validated that it is correct yet, only that it can process all of them without error.

Some additional tests that give coverage of some error conditions.

1ca4426

Initial description of combined PRONOM and container syntax.

5f0da3b

This was referenced Nov 7, 2019

Droidsyntaxparser #304

Closed

full implementation of standard signature syntax in container signatures #237

Closed

nishihatapalmer added 10 commits November 7, 2019 11:34

Initial work on parser and compiler for DROID syntax - unfinished!

8bd3163

First version of ByteSequenceParser.

e895a03

- currently splits subsequences as it goes - not sure if this is the right approach. The parser should probably just represent the expression passed in as a single parse tree. The compiler should worry about how to split this up.

First commit for anchor and compiler, some changes in parser.

51ac70b

Add safe accessor methods for internal objects to allow testing

697411e

the results of compilation. These either return immutable objects, or make defensive copies.

First complete support for official PRONOM syntax (tested).

4162fc6

* There are four PRONOM signatures which currently cannot be parsed, as they use unofficial syntax. Waiting for TNA to respond on what they want to do / support for the PRONOM language officially.

Add a list of all known PRONOM ByteSequences for signature v91.

052a736

* This is used to test that the PRONOM parser is capable of reading known good and real-world signatures.

Tests for first complete support for official PRONOM syntax.

c069071

* There are four PRONOM signatures which currently cannot be parsed, as they use unofficial syntax. Waiting for TNA to respond on what they want to do / support for the PRONOM language officially.

container signature sequences.

9fbfd36

upd: improve README clarity, added links to other documentations

941958f

jcharlet marked this pull request as ready for review November 26, 2019 17:26

nishihatapalmer added 2 commits November 28, 2019 13:24

Update readme to be clearer.

ff92507

* explanation of what kinds of signature exist. * explanation of sigtool capabilities * explanation of simpler XML format.

Minor fixes to XML example.

d6493b1

fix: readme link to sig tool documentation

cf6b206

upd: rename PRONOM Syntax to signature syntax, move types and syntax …

93e9fde

…section in README to signature syntax doc

Fix Signature syntax.md with link to sigtool description

9497459

Merge branch 'master' into droidsyntaxparser

a5d2c38

jcharlet merged commit df8c1ca into master Dec 6, 2019

jcharlet deleted the droidsyntaxparser branch December 6, 2019 10:58

jcharlet added the enhancement label Jan 2, 2020

jcharlet changed the title ~~PR: full implementation of standard signature syntax in container signatures~~ Full implementation of standard signature syntax in container signatures Feb 12, 2020

jcharlet assigned nishihatapalmer Feb 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Full implementation of standard signature syntax in container signatures #305

Full implementation of standard signature syntax in container signatures #305

jcharlet commented Nov 7, 2019

jcharlet commented Nov 7, 2019

jcharlet commented Nov 7, 2019 •

edited

nishihatapalmer commented Nov 26, 2019

jcharlet commented Nov 26, 2019

nishihatapalmer commented Nov 26, 2019 •

edited

nishihatapalmer commented Nov 26, 2019 •

edited

jcharlet commented Nov 27, 2019

nishihatapalmer commented Nov 27, 2019 via email

nishihatapalmer commented Nov 28, 2019 via email

jcharlet commented Nov 28, 2019

nishihatapalmer commented Nov 28, 2019 via email

nishihatapalmer commented Nov 28, 2019 via email

nishihatapalmer commented Nov 28, 2019 via email

jcharlet commented Nov 28, 2019 •

edited

jcharlet commented Nov 28, 2019

nishihatapalmer commented Nov 28, 2019 via email

nishihatapalmer commented Nov 28, 2019 via email

jcharlet commented Nov 28, 2019 •

edited

jcharlet commented Nov 28, 2019

Full implementation of standard signature syntax in container signatures #305

Full implementation of standard signature syntax in container signatures #305

Conversation

jcharlet commented Nov 7, 2019

jcharlet commented Nov 7, 2019

jcharlet commented Nov 7, 2019 • edited

nishihatapalmer commented Nov 26, 2019

jcharlet commented Nov 26, 2019

nishihatapalmer commented Nov 26, 2019 • edited

nishihatapalmer commented Nov 26, 2019 • edited

jcharlet commented Nov 27, 2019

nishihatapalmer commented Nov 27, 2019 via email

nishihatapalmer commented Nov 28, 2019 via email

jcharlet commented Nov 28, 2019

nishihatapalmer commented Nov 28, 2019 via email

nishihatapalmer commented Nov 28, 2019 via email

nishihatapalmer commented Nov 28, 2019 via email

jcharlet commented Nov 28, 2019 • edited

jcharlet commented Nov 28, 2019

nishihatapalmer commented Nov 28, 2019 via email

nishihatapalmer commented Nov 28, 2019 via email

jcharlet commented Nov 28, 2019 • edited

jcharlet commented Nov 28, 2019

jcharlet commented Nov 7, 2019 •

edited

nishihatapalmer commented Nov 26, 2019 •

edited

nishihatapalmer commented Nov 26, 2019 •

edited

jcharlet commented Nov 28, 2019 •

edited

jcharlet commented Nov 28, 2019 •

edited