Some more massive documentation

Also allow --per-line to have a Callable, and rename --show-line-number to --show-item-number. And introduce the concept of an item producer in the runtime documentation.
lizmat · Sep 29, 2022 · ed5647a · ed5647a
1 parent d452899
commit ed5647a
Show file tree

Hide file tree

Showing 24 changed files with 520 additions and 203 deletions.
diff --git a/Changes b/Changes
@@ -5,6 +5,11 @@ Revision history for App-Rak
        out documentation for all arguments from the command line
      - Bump dependency on rak to get default for :dir/:file fix
      - Make sure --find-all is actually handled
+     - Support --per-line also with a Callable producer
+     - Properly stringify any Buf returned by a matcher as a list
+       of integers
+     - Rename --show-line-number to --show-item-number to make it
+       more in line with the concept of producers producing items
 
 0.0.99  2022-09-27T22:12:15+02:00
      - Add support for --unicode

diff --git a/META6.json b/META6.json
@@ -33,7 +33,7 @@
     "help/filesystem.txt",
     "help/general.txt",
     "help/haystack.txt",
-    "help/input.txt",
+    "help/item.txt",
     "help/listing.txt",
     "help/option.txt",
     "help/pattern.txt",

diff --git a/README.md b/README.md
@@ -320,7 +320,7 @@ $ rak foo --extensions=md,markdown
 $ rak foo --extensions=
 ```
 
-Predefined groups are `#raku`, `#perl`, `#cro`, `#text` `#c`, `#c++`, `#yaml`, `#ruby`, `#python`, `#html`, `#markdown`, `#json`, `#jsonl`, `#csv`, `#config` and `#text`.
+Predefined groups are `#raku`, `#perl`, `#cro`, `#text`, `#c`, `#c++`, `#yaml`, `#ruby`, `#python`, `#html`, `#markdown`, `#json`, `#jsonl`, `#csv`, `#config` and `#text`.
 
 The `--list-known-extensions` argument can be used to see which predefined groups of extensions are supported, and which extensions they cover.
 
@@ -871,6 +871,18 @@ Indicate whether matching should be done per file, rather than per line. If spec
 $ rak foo --per-file='*.lines(:!chomp).head(10).join'
 ```
 
+--per-line[=code]
+-----------------
+
+Indicate whether matching should be done per line. If specified as a flag, will read lines with the indicated `--encoding` and present each line to the matcher (which is actually the default if no action was specified).i
+
+Optionally takes a `Callable` specification: this will be given an `IO::Path` object of the file: that is then expected to produce lines that will be presented to the matcher.
+
+```bash
+# look for foo in only the last 10 lines of each file
+$ rak foo --per-line='*.lines.tail(10)'
+```
+
 --proximate=[N]
 ---------------
 

diff --git a/doc/App-Rak.rakudoc b/doc/App-Rak.rakudoc
@@ -1086,6 +1086,23 @@ $ rak foo --per-file='*.lines(:!chomp).head(10).join'
 
 =end code
 
+=head2 --per-line[=code]
+
+Indicate whether matching should be done per line.  If specified as a flag,
+will read lines with the indicated C<--encoding> and present each line to the
+matcher (which is actually the default if no action was specified).i
+
+Optionally takes a C<Callable> specification: this will be given an C<IO::Path>
+object of the file: that is then expected to produce lines that will be
+presented to the matcher.
+
+=begin code :lang<bash>
+
+# look for foo in only the last 10 lines of each file
+$ rak foo --per-line='*.lines.tail(10)'
+
+=end code
+
 =head2 --proximate=[N]
 
 Indicates whether matched lines should be grouped together that are within
@@ -1248,13 +1265,13 @@ if possible, instead of just the line.  Defaults to C<False>.
 
 Requires that the L<C<Git::Blame::File>|https://raku.land/zef:lizmat/Git::Blame::File> module is installed.
 
-=head2 --show-filename
+=head2 --show-item-number
 
-Flag.  Indicate whether filenames should be shown.  Defaults to C<True>.
+Flag.  Indicate whether item numbers should be shown.  Defaults to C<True>.
 
-=head2 --show-line-number
+=head2 --show-filename
 
-Flag.  Indicate whether line numbers should be shown.  Defaults to C<True>.
+Flag.  Indicate whether filenames should be shown.  Defaults to C<True>.
 
 =head2 --shell=invocation
 

diff --git a/lib/App/Rak.rakumod b/lib/App/Rak.rakumod
@@ -17,7 +17,7 @@ my constant BOFF = "\e[22m";  # BOLD OFF
 #- start of available options --------------------------------------------------
 #- Generated on 2022-09-27T22:55:07+02:00 by tools/makeOPTIONS.raku
 #- PLEASE DON'T CHANGE ANYTHING BELOW THIS LINE
-my str @options = <absolute accessed after-context allow-loose-escapes allow-loose-quotes allow-whitespace auto-diag backup batch before-context blame-per-file blame-per-line blocks break checkout context count-only created csv-per-line degree device-number dir dont-catch dryrun edit encoding eol escape exec extensions file file-separator-null files-from files-with-matches files-without-matches filesize find find-all formula frequencies gid group group-matches hard-links has-setgid has-setuid help highlight highlight-after highlight-before human ignorecase ignoremark inode invert-match is-empty is-executable is-group-executable is-group-readable is-group-writable is-owned-by-group is-owned-by-user is-owner-executable is-owner-readable is-owner-writable is-readable is-sticky is-symbolic-link is-world-executable is-world-readable is-world-writable is-writable json-per-elem json-per-file json-per-line keep-meta known-extensions list-custom-options list-expanded-options list-known-extensions matches-only max-matches-per-file meta-modified mode modified modify-files module only-first output-file pager paragraph-context passthru passthru-context paths paths-from pattern per-file per-line proximate rename-files quietly quote rak recurse-symlinked-dir recurse-unmatched-dir repository save sayer sep shell show-blame show-filename show-line-number silently smartcase stats stats-only strict summary-if-larger-than trim type uid under-version-control unicode unique user verbose version vimgrep with-line-endings>;
+my str @options = <absolute accessed after-context allow-loose-escapes allow-loose-quotes allow-whitespace auto-diag backup batch before-context blame-per-file blame-per-line blocks break checkout context count-only created csv-per-line degree device-number dir dont-catch dryrun edit encoding eol escape exec extensions file file-separator-null files-from files-with-matches files-without-matches filesize find find-all formula frequencies gid group group-matches hard-links has-setgid has-setuid help highlight highlight-after highlight-before human ignorecase ignoremark inode invert-match is-empty is-executable is-group-executable is-group-readable is-group-writable is-owned-by-group is-owned-by-user is-owner-executable is-owner-readable is-owner-writable is-readable is-sticky is-symbolic-link is-world-executable is-world-readable is-world-writable is-writable json-per-elem json-per-file json-per-line keep-meta known-extensions list-custom-options list-expanded-options list-known-extensions matches-only max-matches-per-file meta-modified mode modified modify-files module only-first output-file pager paragraph-context passthru passthru-context paths paths-from pattern per-file per-line proximate rename-files quietly quote rak recurse-symlinked-dir recurse-unmatched-dir repository save sayer sep shell show-blame show-filename show-item-number silently smartcase stats stats-only strict summary-if-larger-than trim type uid under-version-control unicode unique user verbose version vimgrep with-line-endings>;
 #- PLEASE DON'T CHANGE ANYTHING ABOVE THIS LINE
 #- end of available options ----------------------------------------------------
 
@@ -524,7 +524,7 @@ my sub HELP($text, @keys, :$verbose) {
     say "-" x $header.chars;
     say $writing-to-stdout
       ?? $text.lines.map({
-              !.starts-with(" ") && .ends-with(":") ?? BON ~ $_ ~ BOFF !! $_
+              !$++ || .starts-with('--') ?? BON ~ $_ ~ BOFF !! $_
          }).join("\n")
       !! $text;
 
@@ -632,8 +632,15 @@ my sub rak-results() {
     }
 
     # Set way to stringify paths
-    my &stringify :=
+    my &path-stringify :=
       IO::Path.^find_method($absolute ?? "absolute" !! "relative");
+    my sub stringify($value) {
+        IO::Path.ACCEPTS($value)
+          ?? path-stringify($value)
+          !! Buf.ACCEPTS($value)
+            ?? $value.List.Str
+            !! $value.Str
+    }
 
     # show the results!
     my int $seen;
@@ -646,19 +653,15 @@ my sub rak-results() {
             # Just listing paths
             if $key eq '<find>' {
                 for @$value.sort(*.fc) {
-                    sayer IO::Path.ACCEPTS($_)
-                      ?? line-post-proc stringify($_)
-                      !! line-post-proc .Str;
+                    sayer line-post-proc stringify($_);
                     last RESULT if ++$seen == $only-first;
                 }
             }
 
             # Looks like normal search result
             elsif Iterable.ACCEPTS($value) {
                 if $value -> @matches {
-                    my $source := IO::Path.ACCEPTS($key)
-                      ?? stringify($key)
-                      !! $key;
+                    my $source := stringify($key);
                     sayer $break if $has-break && $seen;
 
                     if PairContext.ACCEPTS(@matches.head) {
@@ -677,8 +680,8 @@ my sub rak-results() {
                                 }
                                 else {
                                     sayer $linenr ~ ':' ~ (.matched
-                                      ?? line-post-proc .value.Str
-                                      !! .value.Str
+                                      ?? line-post-proc stringify(.value)
+                                      !! stringify(.value)
                                     );
                                 }
                                 last RESULT if ++$seen == $stop-after;
@@ -701,8 +704,8 @@ my sub rak-results() {
                                     sayer $source
                                       ~ ':' ~ $linenr
                                       ~ ':' ~ (.matched
-                                      ?? line-post-proc .value.Str
-                                      !! .value.Str
+                                      ?? line-post-proc stringify(.value)
+                                      !! stringify(.value)
                                     );
                                 }
                                 last RESULT if ++$seen == $stop-after;
@@ -724,8 +727,8 @@ my sub rak-results() {
                                 else {
                                     sayer $linenr
                                       ~ ':' ~ (.matched
-                                      ?? line-post-proc .value.Str
-                                      !! .value.Str
+                                      ?? line-post-proc stringify(.value)
+                                      !! stringify(.value)
                                     );
                                 }
                                 last RESULT if ++$seen == $stop-after;
@@ -745,7 +748,7 @@ my sub rak-results() {
                                 sayer .Str for @$_;
                             }
                             else {
-                                sayer line-post-proc .Str;
+                                sayer line-post-proc stringify $_;
                             }
                             last RESULT if ++$seen == $stop-after;
                         }
@@ -761,7 +764,8 @@ my sub rak-results() {
                                 sayer "$source:$_" for @$_;
                             }
                             else {
-                                sayer $source ~ ':' ~ line-post-proc .Str;
+                                sayer $source
+                                  ~ ':' ~ line-post-proc stringify $_;
                             }
                             last RESULT if ++$seen == $stop-after;
                         }
@@ -771,20 +775,14 @@ my sub rak-results() {
 
             # looks like frequencies output
             else {
-                sayer $outer.value ~ ':' ~ $outer.key;
+                sayer "&stringify($outer.value):$outer.key()";
                 last RESULT if ++$seen == $stop-after;
             }
         }
 
-        # Only got filename, so show its path
-        elsif IO::Path.ACCEPTS($outer) {
-            sayer stringify($outer);
-            last RESULT if ++$seen == $only-first;
-        }
-
-        # Probably --unique
+        # anything else
         else {
-            sayer $outer.Str;
+            sayer stringify $outer;
             last RESULT if ++$seen == $stop-after;
         }
     }
@@ -1549,9 +1547,9 @@ my sub option-per-file($value --> Nil) {
 }
 
 my sub option-per-line($value --> Nil) {
-    Bool.ACCEPTS($value)
-      ?? set-action('per-line', $value)
-      !! meh "'--per-line' must be specified as a flag";
+    set-action 'per-line', Bool.ACCEPTS($value)
+      ?? $value
+      !! convert-to-simple-Callable($value);
 }
 
 my sub option-proximate($value --> Nil) {
@@ -1617,8 +1615,8 @@ my sub option-show-filename($value --> Nil) {
     set-listing-flag('show-filename', $value);
 }
 
-my sub option-show-line-number($value --> Nil) {
-    set-result-flag('show-line-number', $value);
+my sub option-show-item-number($value --> Nil) {
+    set-result-flag('show-item-number', $value);
 }
 
 my sub option-silently($value --> Nil) {
@@ -1871,13 +1869,13 @@ my sub move-result-options-to-rak(--> Nil) {
                 %rak<max-matches-per-source> := $max;
             }
 
-            with %result<show-line-number>:delete {
+            with %result<show-item-number>:delete {
                 %rak<omit-item-number> := True unless $_;
             }
 
             if %result<find>:delete {
                 %rak<find>             := True;
-                %rak<omit-item-number> := True;
+                %rak<omit-item-number> := True unless %result<frequencies>;
 
                 # Only interested in number of files
                 if %result<count-only>:delete {
@@ -2149,7 +2147,7 @@ my sub action-json-per-elem(--> Nil) {
     move-filesystem-options-to-rak;
     move-result-options-to-rak;
 
-    if %listing<show-line-number>:delete {
+    if %listing<show-item-number>:delete {
         # no action needed
     }
     elsif %listing<files-with-matches>:delete {
@@ -2187,7 +2185,7 @@ my sub action-json-per-line(--> Nil) {
     move-filesystem-options-to-rak;
     move-result-options-to-rak;
 
-    if %listing<show-line-number>:delete {
+    if %listing<show-item-number>:delete {
         # no action needed
     }
     elsif %listing<files-with-matches>:delete {
@@ -2339,6 +2337,10 @@ my sub action-per-line(--> Nil) {
     move-filesystem-options-to-rak;
     move-result-options-to-rak;
 
+    # The default in rak already does the right thing
+    %rak<produce-many> := $action
+      if $action.defined && !($action<> =:= True);
+
     run-rak;
     rak-results;
     rak-stats;

diff --git a/resources/help.txt b/resources/help.txt
@@ -1,3 +1,6 @@
+Arguments overview:
+‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒
+
 Pattern specification:
  foo                      literal string
  '/ << bar >> /'          Raku regex indicated by being bounded by / /
@@ -8,23 +11,23 @@ either as first argument, or as --pattern=foo option
 String search pattern modifiers:
  --ignorecase  Ignore distinction between upper, lower and title case letters
  --ignoremark  Only compare base characters, ignore additional marks
- --smartcase   As --ignorecase if pattern does NOT contain any uppercase
+ --smartcase   As --ignorecase if pattern does *NOT* contain any uppercase
  --type=words        Look for string as a word
  --type=starts-with  Look for string at start of a line
  --type=ends-with    Look for string at end of a line
  --type=contains     Look for string anywhere (default)
 
-Action to perform:
- --find            Interprete selected paths as the lines to inspect
- --json-per-file   Call Raku code with parsed JSON
- --json-per-elem   Call Raku code with top elements of parsed JSON
- --json-per-line   Call Raku code with parsed JSON
- --blame-per-file  Call Raku code with Git::Blame::File objects if possible
- --blame-per-line  Call Raku code with Git::Blame::Line objects if possible
- --csv-per-line    Interpret file as CSV and produce rows
- --unicode         Search the Unicode codepoint database
- --per-file        Inspect files as a whole file
- --per-line        Inspect files line by line (default)
+Item producers:
+ --per-line[=producer]  Produce lines for pattern matching
+ --per-file[=producer]  Produce whole file for pattern matching
+ --find            Produce absolute paths for pattern matching
+ --json-per-file   Interpret whole file as a single JSON for matching
+ --json-per-elem   Interpret whole file as JSON, match each top level element
+ --json-per-line   Interpret each line as JSON for pattern matching
+ --blame-per-file  Produce Git::Blame::File object for pattern matcing
+ --blame-per-line  Produce Git::Blame::Line objects for pattern matching
+ --csv-per-line    Produce rows from Text::CSV object for pattern matching
+ --unicode         Produce names of codepoints for pattern matching
 
 Code pattern helpers:
  --repository=lib      First look for any modules to load in the "lib" directory
@@ -105,7 +108,7 @@ Listing modifiers:
  --show-blame                Show git blame information if possible
  --show-filename             Show the filename in which match occurred
  --file-separator-null       Separate filenames with null bytes
- --show-line-number          Show the line number in which match occurred
+ --show-item-number          Show the item number in which match occurred
  --summary-if-larger-than=N  Summarize matching line if longer than N chars
  --trim                      Remove whitespace, true if no context
  --highlight                 Highlight matches if possible

diff --git a/resources/help/code.txt b/resources/help/code.txt
@@ -1,4 +1,5 @@
 Code pattern helpers:
+‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒
 
 --module=foo
 
@@ -7,6 +8,11 @@ code pattern.  Can be specfied more than once.  Defaults to
 no additional module loading.  Also known as -M when calling
 Raku itself.
 
+Example:
+# show all non-empty lines as space separated codepoints
+# using the experimental "pack" function
+$ rak --module='experimental :pack' '{ (pack "A*", $_).List }'
+
 --quietly
 
 Indicate whether warnings should be disabled when executing

diff --git a/resources/help/content.txt b/resources/help/content.txt
@@ -1,4 +1,5 @@
 Content modifiers:
+‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒
 
 --backup[=extension]
 

diff --git a/resources/help/debug.txt b/resources/help/debug.txt
@@ -1,4 +1,5 @@
 Debugging options:
+‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒
 
 --dont-catch
 

diff --git a/resources/help/examples.txt b/resources/help/examples.txt
@@ -1,4 +1,5 @@
 Examples:
+‒‒‒‒‒‒‒‒‒
 
 Find all occurrences of the pattern ‘patricia’ in a file
 

diff --git a/resources/help/filesystem.txt b/resources/help/filesystem.txt
@@ -1,4 +1,5 @@
 File system filters:
+‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒
 
 File system filters refine the inital selection of files as indicated by
 the haystack arguments.