Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hoon, dojo: initial doccords release #5873

Merged
merged 141 commits into from Jan 17, 2023
Merged

hoon, dojo: initial doccords release #5873

merged 141 commits into from Jan 17, 2023

Conversation

drbeefsupreme
Copy link
Contributor

@drbeefsupreme drbeefsupreme commented Jul 1, 2022

This is a draft since its still a WIP, so there's some code that is obviously sloppy and can be improved, but major features are all working and all tests are passing. A couple minor features still have work to be done.

Short story: you can now wrap an arbitrary hoon with %note hoon containing doccords. You can wrap an arbitrary spec with a %dict spec, containing doccords. Both end up compiled as %hint types.

Surfacing doccords for arbitrary hoons isn't currently implemented in the printing library, but you can print doccords for cores, chapters, faces, arms, arm products, default arm of a core built by an arm, and molds using # arm/chapter/core/face, and output values of molds using ?? *foo in dojo.

OTA is blocked by #5849 which 172af42 is a suggested patch for. I had tried cherry picking this commit, but a fresh fake ship wouldn't compile with it, so I'm probably missing other things from that branch for it to work just yet and so haven't tested the fix with this feature branch. The band-aid in #5849 still works if you'd like to test this out in the meantime.

Examples of doccords syntax can be found in /lib/dprint.hoon, /lib/deco.hoon/ and /tests/lib/dprint.hoon. There's a lot of room for change here - I'll work on an exhaustive list of syntax changes and update this PR soon. The biggest difference is that I plan to replace :< and :> with :: with some strict rules on what :: comments should be parsed as formal comments. This might end up being pretty difficult.

All planned features for the first release are essentially complete (#5768 #5769 #5770 #5771 #5773 #5774 #5775 #5776 #5777 #5778 #5779 #5780 #5781 #5782 #5783 #5784 #5785 #5838) - what's mostly left is arguing over syntax and making the code cleaner.


Some example usage of printing library (output is colorized in dojo - that section of the printing lib is still rough, it needs another look)

>  =dp -build-file %/lib/dprint/hoon
> # dp
.dp
(undocumented)

core containing doccords search and printing utilities
|dprint
(undocumented)

types used by doccords
|dprint-types
(undocumented)

.debug
(undocumented)
> # dp/dprint
|dprint
core containing doccords search and printing utilities
chapters:
|dprint|printing
contains arms using for printing doccords items

|dprint|searching
contains arms used for looking for docs inside of a type

compiled against:
types used by doccords
|dprint-types
(undocumented)

.debug
(undocumented)
> # dp/dprint/searching
|searching
contains arms used for looking for docs inside of a type

the entrypoint for finding docs within a type is +find-item-in-type.
arms:
+all-arm-docs
grabs the docs for an arm.

+arm-and-chapter-overviews
returns an overview for a cores arms and chapters

+arm-product-docs
returns 0, 1, or 2 whats for an arm

+arms-as-overview
translate a tome into an overview
...
> # dp/dprint/searching/find-item-in-type
+find-item-in-type
returns the item to print while searching through topic

this gate is optionally called recursively to find the path (topic) in the
type (sut). once it finds the correct part of the type, it switches to
+signify to describe that part of the type. recursion is turned off (for some
cases) when a hint type is found, in case it is wrapping a match.

product:

(undocumented)

default arm in core:

(undocumented)

Doccords from output value of a mold:

> ?? [*term *tape]
  [%cell <|ascii symbol|> <|utf8 string as list|>]

drbeefsupreme and others added 30 commits March 14, 2022 11:49
there's a stub for +* but its not working yet
this populates the $what in $tome
Too often when dealing with big types the compiler traces and other such
outputs become hard to read. Wrapping a type as $+(shorthand big-type)
will now print #shorthand in place of the type.
This changes the parser for +tall so that it looks before and after a
hoon for doccords, and then extracts a label for %brcn if it exists.
+wrap will be used to annotating most hoons, but this commit only covers
%brcn
this changes the parser to take any bar runes surrounded by formal
comments and wraps them with %note tags containing those comments
wraps the skin in tisface with a %help skin
initial commit for library for finding and printing doccords. has some
basic functionality for looking through a type and finding the docs
within it and printing them, but is mostly unfinished
if the product had its own docs, it wouldn't also get the docs for a
the default arm produced by the core. this fixes that. also misc style
fixes
also adds another mule to a play:ut call to avoid another crash that i'm
not sure yet why it is happening
adds the ability to find cores and chapters and produce an item from them
docs written above an arm are now distinguishable in the AST from
docs written above the product of the arm, by tagging docs written
above the arm with a %funk link
rewrites select-arm-docs so that it checks for nested hint types and
sees if the outermost help hint has a %funk link with the name of the
arm in order to tell that its an arm-doc
stupid loobeans tripping me up
I didn't know what I was doing before, I think this is the right way to
wrap tisfas with a %note hoon.
before this, it was grabbing the initial arm-doc from the AST rather
than the type. now %arm items have all 3 types of docs available. the
interface has been degraded somewhat though, as %arm items no longer
have a single docs field. more refactoring will be needed to figure out
the best way to do this.
@drbeefsupreme
Copy link
Contributor Author

The doccords don't show up using the old +* syntax either, but some archaeology I did suggests %made used to have a bigger role. So I'm not entirely ruling it out.

The fix in https://github.com/urbit/urbit/pull/5873/files#diff-94e19e683e69204abd61d7b5991a01c1db936a982c17b46f8b3d3825497b6d9fR8426-R8432 definitely has some unintended side-effects. If I run

> =a %-  ream
  '''
  |$  [item]
  ::    an item
  [thing=cord detail=item]
  '''
  > =b ~(open ap a)
  > =c ?>(?=([%brtr *] b) q.b)
  > ~(open ap c)

you end up with

> ~(open ap c)
[ %note
  p=[%help p=[cuff=~ crib=[summary='an item' details=~]]]
    q
  [ %brcl
      p
    [ %ktsg
        p
      [ %ktls
        p=[%bust p=%noun]
          q
        [ %note
          p=[%help p=[cuff=~ crib=[summary='an item' details=~]]]
            q
          [   p
            [ %ktts
              p=term=%thing
              q=[%tsgl p=[%$ p=6] q=[%wing p=~[%cord]]]
            ]
              q
            [ %ktts
              p=term=%detail
              q=[%tsgl p=[%$ p=6] q=[%wing p=~[%item]]]
            ]
          ]
        ]
      ]
    ]
      q
    [ %tsls
        p
      [ %note
        p=[%help p=[cuff=~ crib=[summary='an item' details=~]]]
          q
        [   p
          [ %ktts
            p=term=%thing
              q
            [ %cncl
              p=[%tsgr p=[%wing p=~[[%.y p=7]]] q=[%wing p=~[%cord]]]
              q=~[[%$ p=12]]
            ]
          ]
            q
          [ %note
            p=[%help p=[cuff=~ crib=[summary='an item' details=~]]]
              q
            [ %ktts
              p=term=%detail
                q
              [ %cncl
                  p
                [%tsgr p=[%wing p=~[[%.y p=7]]] q=[%wing p=~[%item]]]
                q=~[[%$ p=13]]
              ]
            ]
          ]
        ]
      ]
      q=[%tsls p=[%dtts p=[%$ p=14] q=[%$ p=2]] q=[%$ p=6]]
    ]

So we're getting identical %notes littered all over the expansion. Not good. I'd prefer the special-casing of %gist in |$ over this, but I think if I stare at this awhile longer a better idea will come out.

@joemfb
Copy link
Member

joemfb commented Dec 13, 2022

The additional %notes are because +decorate:ax only uses the annotation, it doesn't consume it. You also need to call +clear:ax in the right spot (and figure out where that is). It's a difficult convention in that core ...

@drbeefsupreme
Copy link
Contributor Author

drbeefsupreme commented Dec 14, 2022

I found something that works for the |$ issue, in the sense that the AST produced by the +open hack in 5fc3b3f#diff-94e19e683e69204abd61d7b5991a01c1db936a982c17b46f8b3d3825497b6d9fR8433-L8429 is reproduced here 1418e1d but making use of +decorate and +clear instead.

It also does not have the duplicate %notes mentioned above. Basically, the %gist wrapped by a %ktcl turns into a %note wrapping the +open:ap of %ktcl, just like it does on +$ arms.

~(open ap c) from my previous message returns

[ %note
  p=[%help p=[cuff=~ crib=[summary='an item' details=~]]]
    q
  [ %brcl
      p
    [ %ktsg
        p
      [ %ktls
        p=[%bust p=%noun]
          q
        [ p=[%ktts p=term=%thing q=[%tsgl p=[%$ p=6] q=[%wing p=~[%cord]]]]
          q=[%ktts p=term=%detail q=[%tsgl p=[%$ p=6] q=[%wing p=~[%item]]]]
        ]
      ]
    ]
      q
    [ %tsls
        p
      [   p
        [ %ktts
          p=term=%thing
            q
          [ %cncl
            p=[%tsgr p=[%wing p=~[[%.y p=7]]] q=[%wing p=~[%cord]]]
            q=~[[%$ p=12]]
          ]
        ]
          q
        [ %ktts
          p=term=%detail
            q
          [ %cncl
            p=[%tsgr p=[%wing p=~[[%.y p=7]]] q=[%wing p=~[%item]]]
            q=~[[%$ p=13]]
          ]
        ]
      ]
      q=[%tsls p=[%dtts p=[%$ p=14] q=[%$ p=2]] q=[%$ p=6]]
    ]
  ]
]

My main concern is with the effect +clear has on def. +clear nulls out nut, bug, and def. +decorate handles nut and bug, but not def. The only places def is used are in +spore and the %bchp case in +analyze, and here I am changing the ̶o̶n̶l̶y̶ one of two callsites to +spore (the other being in +example) with +spore:clear. But since +spore is what handles $~ (which sets def), which is ubiquitous, I'd think that if there was a real issue here then it wouldn't get past compiling hoon.hoon. spore:clear should be running +spore with def made null.

@joemfb

drbeefsupreme added 3 commits December 14, 2022 16:02
everything appears to work fine without them and I don't think it is any
less unclear what this gate is doing.
notes aren't just for doccords, of course
all this did was set .nut. while it could be used with doccords, it is
currently unused, and none of the other values in the sample of _ax are
set this way (bug, def, cox, hay, dom). i experimented a little bit with
trying to make use of this but it made things overall more unreadable,
and it wouldn't make sense to do it without doing the same for other
values of the sample. im guessing this is just an old style.
@drbeefsupreme
Copy link
Contributor Author

I'm currently looking around for hoon.hoon PRs that could be bundled together with this one. I started already but its getting late so I'll finish tomorrow.

Based on @joemfb 's comment here #5939 (review) I'm guessing that since doccords breaks some jet signatures, it also needs to be a Kelvin release? In which case #6064 should be included as well.

drbeefsupreme added 3 commits December 15, 2022 14:24
no idea how this ended up happening, but apparently it was my fault.
anytime a gate prints with a complicated sample or product type it is
frequently extremely long. 3 is probably too low of a cutoff number, but
ideally a future version will have verbosity settings that will help
control this.
drbeefsupreme pushed a commit that referenced this pull request Dec 16, 2022
|*  foo  bar is sugar for =+  foo  |@  ++  $  bar  --, and newbies find
the old style confusing. this switches out the |@ pattern for the |*
one, at least in layer <=4. the only ones remaining are +toad, +rune,
and +runo, which are already tweaked in #5873 so we omit them here.
making hoon.hoon more legible to doccords. also moving some things
around that seemed to be in the wrong place
@drbeefsupreme
Copy link
Contributor Author

Friendly poke to @Fang- about reviewing this. Since @joemfb is out for at least a couple weeks I think its all on you unless you call in backup.

I am feeling pretty confident with the hoon.hoon changes but nobody has yet to really look at /lib/dprint.hoon or the dojo.hoon changes closely. Those have the highest probability of needing more than minor changes done IMO (though to be clear I believe they're shippable as is, if we had to).

drbeefsupreme and others added 6 commits January 4, 2023 18:33
See #6052. This is completely different from the +* used at the top
of doors, and has almost entirely been replaced by |$. The exception is
the use of the `%made` spec, not present in `|$`. I do not see an
obvious way to change `|$` to use `%made` since this `+*` parser uses
the name of the arm in the `%made` structure, unless we change the
AST of |$.
@philipcmonk
Copy link
Contributor

philipcmonk commented Jan 14, 2023

I believe this is ready to go. I made the following changes:

Screenshot from 2023-01-13 17-46-38

@philipcmonk philipcmonk merged commit 86cf639 into next/arvo Jan 17, 2023
@philipcmonk philipcmonk deleted the jon/doccords branch January 17, 2023 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants