8252523: Add ASN.1 Formatter to work with test utility HexPrinter #268

RogerRiggs · 2020-09-20T13:54:02Z

JDK-8252523: Add ASN.1 Formatter to work with test utility HexPrinter

Debugging functions that utilize ASN.1, DER, and BER encoded streams is
difficult without test utilities to show the contents.
The ASN.1 formatter reads a stream and produces annotated output of the
tags, values, and structures.
When used with the test library jdk.test.lib.hexdump.HexPrinter the annotations are synchronized
with the hex formatted output.

Small changes to HexPrinter are included to improve the output readability.

Example decoding of a .pem certificate:

SEQUENCE [910]
  SEQUENCE [630]
    CONTEXT cons 0 [3]
      BYTE 2, 
    BYTE 3, 
    SEQUENCE [13]
      OBJECT ID  [9] 1.2.840.113549.1.1.11 (SHA256withRSA) 
      NULL 
    SEQUENCE [76]
      SET [11]
        SEQUENCE [9]
          OBJECT ID  [3] 2.5.4.6 (CountryName) 
          'IN' 
      ...
      SET [16]
        SEQUENCE [14]
          OBJECT ID  [3] 2.5.4.3 (CommonName) 
          Client1 
    SEQUENCE [30]
      UTCTIME  [13] '150526221718Z' 
      UTCTIME  [13] '250523221718Z' 
    ...
    SEQUENCE [290]
      SEQUENCE [13]
        OBJECT ID  [9] 1.2.840.113549.1.1.1 (RSA) 
        NULL 
      BIT STRING  [271]
      CONTEXT cons 3 [123]
        SEQUENCE [121]
          SEQUENCE [9]
            OBJECT ID  [3] 2.5.29.19 (BasicConstraints) 
            OCTET STRING  [2] <Unprintable> 
          SEQUENCE [44]
            OBJECT ID  [9] 2.16.840.1.113730.1.13 
            OCTET STRING  [31] '..OpenSSL Generated Certificate' 
          SEQUENCE [29]
            OBJECT ID  [3] 2.5.29.14 (SubjectKeyID) 
            OCTET STRING  [22] <Unprintable> 
          SEQUENCE [31]
            OBJECT ID  [3] 2.5.29.35 (AuthorityKeyID) 
            OCTET STRING  [24] <Unprintable> 
      SEQUENCE [13]
        OBJECT ID  [9] 1.2.840.113549.1.1.11 (SHA256withRSA) 
        NULL 
      BIT STRING  [257]

When used with the HexPrinter test utility, the formatting of the
hexadecimal values is selected with the parameters to HexPrinter.

    0000: 30 82 03 8e                                     ; SEQUENCE [910]
    0004:             30 82 02 76                         ;   SEQUENCE [630]
    0008:                         a0 03                   ;     CONTEXT cons 0 [3]
    000a:                               02 01 02          ;       BYTE 2, 
    000d:                                        02 01 03 ;     BYTE 3, 
    0010: 30 0d                                           ;     SEQUENCE [13]
    0012:       06 09 2a 86 48 86 f7 0d 01 01 0b          ;       OBJECT ID  [9] 1.2.840.113549.1.1.11 (SHA256withRSA) 
    001d:                                        05 00    ;       NULL 
    001f:                                              30 ;     SEQUENCE [76]
    0020: 4c                                              ; 
    0021:    31 0b                                        ;       SET [11]
    0023:          30 09                                  ;         SEQUENCE [9]
    0025:                06 03 55 04 06                   ;           OBJECT ID  [3] 2.5.4.6 (CountryName) 
    002a:                               13 02 49 4e       ;           'IN' 
    
    ...                                                           ...
    
    005b:                                  31 10          ;       SET [16]
    005d:                                        30 0e    ;         SEQUENCE [14]
    005f:                                              06 ;           OBJECT ID  [3] 2.5.4.3 (CommonName) 
    0060: 03 55 04 03                                     ; 
    0064:             0c 07 43 6c 69 65 6e 74 31          ;           Client1 
    006d:                                        30 1e    ;     SEQUENCE [30]
    006f:                                              17 ;       UTCTIME  [13] '150526221718Z' 
    0070: 0d 31 35 30 35 32 36 32 32 31 37 31 38 5a       ; 
    007e:                                           17 0d ;       UTCTIME  [13] '250523221718Z' 
    0080: 32 35 30 35 32 33 32 32 31 37 31 38 5a          ; 
    
    ...                                                         ...
    
    00db:                                  30 82 01 22    ;     SEQUENCE [290]
    00df:                                              30 ;       SEQUENCE [13]
    00e0: 0d                                              ; 
    00e1:    06 09 2a 86 48 86 f7 0d 01 01 01             ;         OBJECT ID  [9] 1.2.840.113549.1.1.1 (RSA) 
    00ec:                                     05 00       ;         NULL 
    00ee:                                           03 82 ;       BIT STRING  [271]
    00f0: 01 0f 00 30 82 01 0a 02 82 01 01 00 d8 70 03 54 ; 
    
    ...
    
    01f0: 0a 2d f5 de 59 3e d9 5e 74 93 d2 45 02 03 01 00 ; 
    0200: 01                                              ; 
    0201:    a3 7b                                        ;       CONTEXT cons 3 [123]
    0203:          30 79                                  ;         SEQUENCE [121]
    0205:                30 09                            ;           SEQUENCE [9]
    0207:                      06 03 55 1d 13             ;             OBJECT ID  [3] 2.5.29.19 (BasicConstraints) 
    020c:                                     04 02 30 00 ;             OCTET STRING  [2] <Unprintable> 
    0210: 30 2c                                           ;           SEQUENCE [44]
    0212:       06 09 60 86 48 01 86 f8 42 01 0d          ;             OBJECT ID  [9] 2.16.840.1.113730.1.13 
    021d:                                        04 1f 16 ;             OCTET STRING  [31] '..OpenSSL Generated Certificate' 
    0220: 1d 4f 70 65 6e 53 53 4c 20 47 65 6e 65 72 61 74 ; 
    0230: 65 64 20 43 65 72 74 69 66 69 63 61 74 65       ; 
    023e:                                           30 1d ;           SEQUENCE [29]
    0240: 06 03 55 1d 0e                                  ;             OBJECT ID  [3] 2.5.29.14 (SubjectKeyID) 
    0245:                04 16 04 14 87 13 66 bc 7a 4d 8e ;             OCTET STRING  [22] <Unprintable> 
    0250: 98 e7 97 fb cc 56 41 27 c8 5e 4c b2 4d          ; 
    025d:                                        30 1f    ;           SEQUENCE [31]
    025f:                                              06 ;             OBJECT ID  [3] 2.5.29.35 (AuthorityKeyID) 
    0260: 03 55 1d 23                                     ; 
    0264:             04 18 30 16 80 14 1f 21 4f db 10 31 ;             OCTET STRING  [24] <Unprintable> 
    0270: d0 67 83 09 03 d3 cd fc 46 ec cf 1d 8b b4       ; 
    027e:                                           30 0d ;       SEQUENCE [13]
    0280: 06 09 2a 86 48 86 f7 0d 01 01 0b                ;         OBJECT ID  [9] 1.2.840.113549.1.1.11 (SHA256withRSA) 
    028b:                                  05 00          ;         NULL 
    028d:                                        03 82 01 ;       BIT STRING  [257]
    0290: 01 00 3e 2b 5f 32 aa f0 f7 52 2b ba f3 bb 07 ee ; 
    
    ...
    
    0390: 6d 94                                           ;

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8252523: Add ASN.1 Formatter to work with test utility HexPrinter

Reviewers

Weijun Wang (@wangweij - Reviewer)

Download

$ git fetch https://git.openjdk.java.net/jdk pull/268/head:pull/268
$ git checkout pull/268

bridgekeeper · 2020-09-20T13:55:47Z

👋 Welcome back rriggs! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2020-09-20T13:57:04Z

@RogerRiggs The following labels will be automatically applied to this pull request: core-libs security.

When this pull request is ready to be reviewed, an RFR email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label (add|remove) "label" command.

mlbridge · 2020-09-20T14:01:38Z

Webrevs

wangweij

Just some comments based on the output example.

test/lib/jdk/test/lib/hexdump/ASN1Formatter.java

…onstructed tags

…rge tag-values.

RogerRiggs · 2020-09-29T14:20:50Z

Are there any other comments or suggestions?

wangweij · 2020-09-29T14:27:29Z

Can you post an example output of that cert? What about indefinite length? (Ex: 0x24, (byte) 0x80, 4, 2, 'a', 'b', 4, 2, 'c', 'd', 0, 0)

RogerRiggs · 2020-09-29T14:49:49Z

For your example the output is:

0000: 24 80                                           ; UNIVERSAL CONSTRUCTED OCTET STRING [INDEFINITE]
0002:       04 02 61 62                               ;   OCTET STRING [2] 'ab' 
0006:                   04 02 63 64                   ; OCTET STRING [2] 'cd' 
000a:                               00 00             ; END-OF-CONTENT ```

It seems to have lost an indent, I'll take a look at that.

jnimeh · 2020-09-29T15:20:13Z

Also in that last example, it seems to suggest that the second octet string is nested within the first one since it sits at a second indent layer. They are both primitives completely covered by their two byte values so shouldn't they sit at the same indentation level? Or is the indentation not there to suggest nested substructures and is more for separation between elements? Or is this what you mean by "lost an indent"? Also, should the end of content be at the same indentation level as the initial indefinite length encoding?

…h tag-values, added test for indefinite length

RogerRiggs · 2020-09-29T19:20:42Z

Also in that last example, it seems to suggest that the second octet string is nested within the first one since it sits at a second indent layer. They are both primitives completely covered by their two byte values so shouldn't they sit at the same indentation level? Or is the indentation not there to suggest nested substructures and is more for separation between elements? Or is this what you mean by "lost an indent"? Also, should the end of content be at the same indentation level as the initial indefinite length encoding?

Yes, all of the enclosed items should be at the same indent level. (A bug as it turns out).
I chose to indent the END-OF-CONTENT line at the same level to terminate the list of tag-values at that level
All of the items enclosed are at the same level.

The updated output is:

0000: 24 80                                           ; UNIVERSAL CONSTRUCTED OCTET STRING [INDEFINITE]
0002:       04 02 61 62                               ;   OCTET STRING [2] 'ab' 
0006:                   04 02 63 64                   ;   OCTET STRING [2] 'cd' 
000a:                               00 00             ;   END-OF-CONTENT

jnimeh · 2020-09-29T19:35:42Z

Regarding the end-of-content identifier, that looks good. Thanks for fixing the indentation for the right-side ASN.1 interpretation of the bytes. My only remaining question is whether the corresponding hex dumps on the left should match the indentation levels as well. I don't have a strong opinion either way on that one but if you're indenting for each element at the same nest level it seems like that could potentially chew up a lot of horizontal space. Was the extra indentation for the second octet string done for readability?

RogerRiggs · 2020-09-29T19:40:46Z

Regarding the end-of-content identifier, that looks good. Thanks for fixing the indentation for the right-side ASN.1 interpretation of the bytes. My only remaining question is whether the corresponding hex dumps on the left should match the indentation levels as well. I don't have a strong opinion either way on that one but if you're indenting for each element at the same nest level it seems like that could potentially chew up a lot of horizontal space. Was the extra indentation for the second octet string done for readability?

Max had requested the current offset of the byte values, so it was easy to see where each new value started and to keep the offsets on a modulo boundary. The formatter on the right is largely decoupled from the hex value tabular form on the left while keeping the correspondence between the formatted items and the bytes.

jnimeh · 2020-09-29T19:44:24Z

Regarding the end-of-content identifier, that looks good. Thanks for fixing the indentation for the right-side ASN.1 interpretation of the bytes. My only remaining question is whether the corresponding hex dumps on the left should match the indentation levels as well. I don't have a strong opinion either way on that one but if you're indenting for each element at the same nest level it seems like that could potentially chew up a lot of horizontal space. Was the extra indentation for the second octet string done for readability?

Max had requested the current offset of the byte values, so it was easy to see where each new value started and to keep the offsets on a modulo boundary. The formatter on the right is largely decoupled from the hex value tabular form on the left while keeping the correspondence between the formatted items and the bytes.

If you and Max find that better from a visual perspective then that works for me. Thanks for clarifying that.

wangweij

LGTM.

openjdk · 2020-09-30T14:31:35Z

@RogerRiggs This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for more details.

After integration, the commit message for the final commit will be:

8252523: Add ASN.1 Formatter to work with test utility HexPrinter

Reviewed-by: weijun

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 142 new commits pushed to the master branch:

709cfe5: 8253815: Remove unused HeapRegionManager::_num_committed from SA
f80a606: 8253375: OSX build fails with Xcode 12.0 (12A7209)
04775f1: 8253768: Deleting unused pipe_class definitions in adl-file (x86_64.ad).
dc3a0f5: 8253183: Fragile memory barrier selection for some weak memory model platforms
8331e63: 8253778: ShenandoahSafepoint::is_at_shenandoah_safepoint should not access VMThread state from other threads
4c65365: 8253792: javax.swing.text.GapContent documentation typos
3078b5f: 6441211: Small Error in API at javax.swing.plaf.synth.Region
4622a18: 8253791: Issue with useAppleColor check in CSystemColors.m
ac02afe: 8253721: Flag -XX:AVX3Threshold does not accept Zero value
4d9f207: 7151826: [TEST_BUG] [macosx] The test javax/swing/JPopupMenu/4966112/bug4966112.java not for mac
... and 132 more: https://git.openjdk.java.net/jdk/compare/224a30f12ce27257af94e2607c53e5f59ac2bd0f...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

RogerRiggs · 2020-09-30T20:41:23Z

/integrate

openjdk · 2020-09-30T20:42:51Z

@RogerRiggs Since your change was applied there have been 152 commits pushed to the master branch:

06d8cf6: 8253812: Cleanup AbstractMemberWriter
424d7d6: 8252881: [JVMCI] ResolvedJavaType.resolveMethod fails in fastdebug when invoked with a constructor
2a406f3: 8138732: Rename @HotSpotIntrinsicCandidate to @IntrinsicCandidate and move it to the jdk.internal.vm.annotation package
4b16f8a: 8253872: ArgumentHandler must use the same delimiters as in jvmti_tools.cpp
4fb8c77: 8253733: Cleanup internal taglet API
e5ba020: 8253829: Wrong length compared in SSPI bridge
9e453d9: 8239090: Improve CPU feature support in VM_Version
8f7c9a7: 8252001: remove usage of PropertyResolvingWrapper in vmTestbase/nsk/jdi
8cf8e46: 8253700: spurious "extends Throwable" at end of Optional.orElseThrow method declaration
8b3d676: 8238737: remove DeoptimizeAllClassesRate from CTW library
... and 142 more: https://git.openjdk.java.net/jdk/compare/224a30f12ce27257af94e2607c53e5f59ac2bd0f...master

Your commit was automatically rebased without conflicts.

Pushed as commit 092c227.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Add ASN1 Formatter to work with HexPrinter

67df141

openjdk bot added security security-dev@openjdk.org core-libs core-libs-dev@openjdk.org labels Sep 20, 2020

RogerRiggs marked this pull request as ready for review September 20, 2020 13:57

openjdk bot added the rfr Pull request is ready for review label Sep 20, 2020

Small cleanups to javadoc and code

915dfe7

wangweij reviewed Sep 21, 2020

View reviewed changes

test/lib/jdk/test/lib/hexdump/ASN1Formatter.java Show resolved Hide resolved

test/lib/jdk/test/lib/hexdump/ASN1Formatter.java Outdated Show resolved Hide resolved

test/lib/jdk/test/lib/hexdump/ASN1Formatter.java Outdated Show resolved Hide resolved

RogerRiggs added 2 commits September 22, 2020 15:49

For octet strings, dropped output of bytes; improved tag output for c…

d56007b

…onstructed tags

Cleanup of indeterminate length tag values and correct skipping of la…

2511559

…rge tag-values.

Corrected reading of extended length bytes, depth of indefinite lengt…

91741d4

…h tag-values, added test for indefinite length

wangweij approved these changes Sep 29, 2020

View reviewed changes

RogerRiggs changed the title ~~8252523: Add ASN1 Formatter to work with HexPrinter~~ 8252523: Add ASN.1 Formatter to work with HexPrinter Sep 30, 2020

RogerRiggs changed the title ~~8252523: Add ASN.1 Formatter to work with HexPrinter~~ 8252523: Add ASN.1 Formatter to work with test utility HexPrinter Sep 30, 2020

openjdk bot added the ready Pull request is ready to be integrated label Sep 30, 2020

openjdk bot closed this Sep 30, 2020

openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated labels Sep 30, 2020

openjdk bot removed the rfr Pull request is ready for review label Sep 30, 2020

RogerRiggs deleted the 8252523-asn1-formatter branch October 8, 2020 20:38

zhengyu123 mentioned this pull request Nov 24, 2021

8277486: NMT: Cleanup ThreadStackTracker #6504

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8252523: Add ASN.1 Formatter to work with test utility HexPrinter #268

8252523: Add ASN.1 Formatter to work with test utility HexPrinter #268

RogerRiggs commented Sep 20, 2020 •

edited by openjdk bot

Loading

bridgekeeper bot commented Sep 20, 2020

openjdk bot commented Sep 20, 2020

mlbridge bot commented Sep 20, 2020 •

edited

Loading

wangweij left a comment

RogerRiggs commented Sep 29, 2020

wangweij commented Sep 29, 2020

RogerRiggs commented Sep 29, 2020

jnimeh commented Sep 29, 2020

RogerRiggs commented Sep 29, 2020

jnimeh commented Sep 29, 2020

RogerRiggs commented Sep 29, 2020

jnimeh commented Sep 29, 2020

wangweij left a comment

openjdk bot commented Sep 30, 2020

RogerRiggs commented Sep 30, 2020

openjdk bot commented Sep 30, 2020

8252523: Add ASN.1 Formatter to work with test utility HexPrinter #268

8252523: Add ASN.1 Formatter to work with test utility HexPrinter #268

Conversation

RogerRiggs commented Sep 20, 2020 • edited by openjdk bot Loading

JDK-8252523: Add ASN.1 Formatter to work with test utility HexPrinter

Progress

Issue

Reviewers

Download

bridgekeeper bot commented Sep 20, 2020

openjdk bot commented Sep 20, 2020

mlbridge bot commented Sep 20, 2020 • edited Loading

Webrevs

wangweij left a comment

Choose a reason for hiding this comment

RogerRiggs commented Sep 29, 2020

wangweij commented Sep 29, 2020

RogerRiggs commented Sep 29, 2020

jnimeh commented Sep 29, 2020

RogerRiggs commented Sep 29, 2020

jnimeh commented Sep 29, 2020

RogerRiggs commented Sep 29, 2020

jnimeh commented Sep 29, 2020

wangweij left a comment

Choose a reason for hiding this comment

openjdk bot commented Sep 30, 2020

RogerRiggs commented Sep 30, 2020

openjdk bot commented Sep 30, 2020

RogerRiggs commented Sep 20, 2020 •

edited by openjdk bot

Loading

mlbridge bot commented Sep 20, 2020 •

edited

Loading