Permalink
Browse files

Continue documenting. Remove --input and --output.

Removed the --input and --output flags.  Input files are named plainly
on the command line, like the rest of the UNIX tool chain.  Output
goes to standard output, and pipeline redirection can send it
elsewhere.

Also, it appears the documentation will continue until morale
improves.
  • Loading branch information...
1 parent 27fe0ec commit c3c9eb92db45c8e5ba490ace96f18931fae2bec3 @rcaputo committed May 24, 2011
View
50 README
@@ -6,8 +6,9 @@ NAME
App::PipeFilter
DESCRIPTION
- App::PipeFilter is a distribution of shell pipeline filters that mimic
- some of the standard UNIX tools but process structured data.
+ App::PipeFilter is a distribution of shell pipeline filters designed to
+ work with structured data like JSON rather than whitespace separated
+ fields.
For example, jcut is a simple version of cut(1) that understands JSON
objects rather than whitespace separated fields.
@@ -29,7 +30,7 @@ DESCRIPTION
The jsonpath filter supports more complex expressions using
JSON::Path’s variant of JSONPath.
- % curl −s 'http://api.duckduckgo.com/?q=poe&o=json' |
+ curl −s 'http://api.duckduckgo.com/?q=poe&o=json' |
jsonpath −o '$..Topics.*.FirstURL' −o '$..Topics.*.Text' |
grep −i perl |
jmap −i col0 −o url −i col1 −o title |
@@ -39,35 +40,42 @@ DESCRIPTION
url: http://duckduckgo.com/Perl_Object_Environment
DESIGN GOAL
- Follow the UNIX convention of one record per line of text. This
- ensures App::PipeFilter tools are compatible with many standard UNIX
- filters. In the examples above, jcut and jsonpath output is piped
- through sort(1), uniq(1) and grep(1).
+ App::PipeFilter utilities generally follow the UNIX convention of
+ printing one record per line of text. This maximizes compatibility
+ with UNIX utilities like sort(1), uniq(1) and grep(1).
PRO TIPS
- JSON isn’t particularly concise, so put grep(1) and other filters that
- eliminate objects as early as possible in pipelines.
+ JSON isn’t particularly concise, so put grep(1), jcut(1) and other
+ filters that eliminate data as early as possible in pipelines.
SEE ALSO
- jcutExtract one or more named fields from JSON input.
+ jcatconcatenate and print JSON files
- jmapRename one or more named fields from JSON input.
+ jcutcut out selected portions of each JSON object in a file
- json2yaml − Convert JSON input records to YAML output records. Some
- people may find YAML output to be more readable.
+ jmap − map input JSON fields to output JSON fields by renaming them
- jsonpath − Like jcut, but fields are described using JSON::Path’s
- variant of JSONPath.
+ json2yaml − convert files of JSON objects into a stream of YAML objects
- jsortSort JSON input on one or more key fields.
+ jsonpathuse JSON::Path to cut out selected portions of JSON objects
- myswl2jsonConvert mysql(1) batch output (−B) into JSON records.
+ jsortsort input files of JSON objects on key fields
- http://json.org/
+ mysql2json − convert mysql −B output to JSON object streams
- http://search.cpan.org/perldoc?JSON::Path
+ <http://json.org/>
- http://goessner.net/articles/JsonPath/
+ <http://search.cpan.org/perldoc?JSON::Path>
+
+ <http://goessner.net/articles/JsonPath/>
+
+BUGS
+ https://rt.cpan.org/Public/Dist/Display.html?Name=App−PipeFilter
+ <https://rt.cpan.org/Public/Dist/Display.html?Name=App‐PipeFilter>
+
+REPOSITORY
+ https://github.com/rcaputo/app−pipefilter
+ <https://github.com/rcaputo/app‐pipefilter>
COPYRIGHT and LICENSE
App::PipeFilter is Copyright 2011 by Rocco Caputo. All rights are
@@ -76,4 +84,4 @@ COPYRIGHT and LICENSE
-perl v5.10.0 2011‐05‐21 App::PipeFilter(3)
+perl v5.10.0 2011‐05‐23 App::PipeFilter(3)
View
111 TODO.otl
@@ -1,30 +1,81 @@
-[_] 30% Requirements for CPAN 1.000 release
- [X] 100% Support multiline JSON input using JSON::XS's incremental parser.
- [X] 100% Read input in large chunks, parsing incrementally.
- [X] 100% Rename jpath to jsonpath.
- Open the jpath name for JPath, which is something different.
- http://bluelinecity.com/software/jpath/
- [_] 0% Document the libraries.
- [_] 0% First draft.
- [_] 0% Document the new Reader roles.
- [_] 0% Go back and make sure the changes for Reader roles are properly documented.
- [_] 0% Revise examples since the code changed significantly.
- [_] 0% Document the new Opener roles.
- [_] 0% Document App::PipeFilter::Role::Opener::GenericIO.
- [_] 0% Document App::PipeFilter::Role::Opener::GenericInput.
- [_] 0% Document App::PipeFilter::Role::Opener::GenericOutput.
- [_] 0% Document the use of these Opener roles.
- [_] 0% Document the executables.
- [_] 0% Test the executables.
- [_] 0% Use command line test cases as examples.
- [_] 0% Document =head1 METHODS for each library.
- [_] 0% Test dzil.ini.
- [_] 0% Clean up after failed Pod::Weaver experiment.
-[_] 50% Additional features
- [X] 100% Allow -o to be JSON::Path descriptions.
- Implemented jsonpath that acts like jcut but using JSONPath.
- http://goessner.net/articles/JsonPath/
- [_] 0% A jcut implementation using JSON:Select.
- [_] 0% Find or write a Perl implementation or bindings for JSON::Select.
- http://jsonselect.org/
- [_] 0% Wrap a jsonselect app around it.
+[_] 61% Stable.
+ [_] 90% Requirements for CPAN 1.000 release
+ [X] 100% Support multiline JSON input using JSON::XS's incremental parser.
+ [X] 100% Read input in large chunks, parsing incrementally.
+ [X] 100% Rename jpath to jsonpath.
+ Open the jpath name for JPath, which is something different.
+ http://bluelinecity.com/software/jpath/
+ [X] 100% Document the libraries.
+ [X] 100% First draft utilities.
+ [X] 100% First draft modules.
+ [X] 100% lib/App/PipeFilter/Generic/Json.pm
+ [X] 100% lib/App/PipeFilter/Generic.pm
+ [X] 100% lib/App/PipeFilter/JsonCat.pm
+ [X] 100% lib/App/PipeFilter/JsonCut.pm
+ [X] 100% lib/App/PipeFilter/JsonMap.pm
+ [X] 100% lib/App/PipeFilter/JsonPath.pm
+ [X] 100% lib/App/PipeFilter/JsonSort.pm
+ [X] 100% lib/App/PipeFilter/JsonToYaml.pm
+ [X] 100% lib/App/PipeFilter/MysqlToJson.pm
+ [X] 100% lib/App/PipeFilter/Role/Flags/Standard.pm
+ [X] 100% lib/App/PipeFilter/Role/Input/Json.pm
+ [X] 100% lib/App/PipeFilter/Role/Opener/GenericInput.pm
+ [X] 100% lib/App/PipeFilter/Role/Opener/GenericIO.pm
+ [X] 100% lib/App/PipeFilter/Role/Opener/GenericOutput.pm
+ [X] 100% lib/App/PipeFilter/Role/Output/Json.pm
+ [X] 100% lib/App/PipeFilter/Role/Output/Yaml.pm
+ [X] 100% lib/App/PipeFilter/Role/Reader/LineByLine.pm
+ [X] 100% lib/App/PipeFilter/Role/Reader/Sysread.pm
+ [X] 100% lib/App/PipeFilter/Role/Transform/None.pm
+ [X] 100% lib/App/PipeFilter.pm
+ [X] 100% End of DESCRIPTION description of the classes & roles it uses.
+ [X] 100% lib/App/PipeFilter/Generic/Json.pm
+ [X] 100% lib/App/PipeFilter/Generic.pm
+ [X] 100% lib/App/PipeFilter/JsonCat.pm
+ [X] 100% lib/App/PipeFilter/JsonCut.pm
+ [X] 100% lib/App/PipeFilter/JsonMap.pm
+ [X] 100% lib/App/PipeFilter/JsonPath.pm
+ [X] 100% lib/App/PipeFilter/JsonSort.pm
+ [X] 100% lib/App/PipeFilter/JsonToYaml.pm
+ [X] 100% lib/App/PipeFilter/MysqlToJson.pm
+ [X] 100% lib/App/PipeFilter/Role/Flags/Standard.pm
+ [X] 100% lib/App/PipeFilter/Role/Opener/GenericIO.pm
+ [X] 100% lib/App/PipeFilter/Role/Output/Json.pm
+ [X] 100% lib/App/PipeFilter/Role/Output/Yaml.pm
+ [X] 100% lib/App/PipeFilter/Role/Reader/LineByLine.pm
+ [X] 100% lib/App/PipeFilter/Role/Reader/Sysread.pm
+ [X] 100% lib/App/PipeFilter/Role/Transform/None.pm
+ [X] 100% Remove ./bin from examples.
+ [X] 100% Document the new Reader roles.
+ [X] 100% Go back and make sure the changes for Reader roles are properly documented.
+ [X] 100% Revise examples since the code changed significantly.
+ [X] 100% Document the new Opener roles.
+ [X] 100% Document App::PipeFilter::Role::Opener::GenericIO.
+ [X] 100% Document App::PipeFilter::Role::Opener::GenericInput.
+ [X] 100% Document App::PipeFilter::Role::Opener::GenericOutput.
+ [X] 100% Document the use of these Opener roles.
+ [X] 100% Document the executables.
+ [X] 100% Test the executables.
+ [X] 100% jcat*
+ [X] 100% jcut*
+ [X] 100% jmap*
+ [X] 100% json2yaml*
+ [X] 100% jsonpath*
+ [X] 100% jsort*
+ [X] 100% mysql2json*
+ [X] 100% Use command line test cases as examples.
+ [X] 100% Document =head1 METHODS for each library.
+ [_] 0% Test dzil.ini.
+ [X] 100% Clean up after failed Pod::Weaver experiment.
+ [X] 100% Verify that --input and --output are gone.
+ [X] 100% Removed from documentation.
+ [X] 100% Removed from examples.
+ [_] 33% Additional features
+ [X] 100% Allow -o to be JSON::Path descriptions.
+ Implemented jsonpath that acts like jcut but using JSONPath.
+ http://goessner.net/articles/JsonPath/
+ [_] 0% A jcut implementation using JSON:Select.
+ [_] 0% Find or write a Perl implementation or bindings for JSON::Select.
+ http://jsonselect.org/
+ [_] 0% Wrap a jsonselect app around it.
+ [_] 0% Cookbook.
View
@@ -6,34 +6,28 @@ __END__
=head1 NAME
-jcat - catenate JSON streams
+jcat - concatenate and print JSON files
=head1 SYNOPSIS
- jcat < eg/sample.json
-
- jcat --input eg/sample.json --output new.json
-
- jcat --input eg/sample.json --output new.json --verbose
+ jcat [--verbose] [file ...]
=head1 DESCRIPTION
-jcat(1) reads files sequentially, writing them to standard output or a
-named file. JSON records are written one per line regardless of their
-input configuration.
-
-jcat(1) inherits its --input, --output and --verbose flags from
-L<App::PipeFilter::Role::Flags::Standard>. See that module for more
-detailed documentation.
+jcat(1) reads files of JSON objects sequentially, writing them to
+standard output one object per line. Multiline JSON objects are
+flattened to single lines for output.
=head1 SEE ALSO
-You may read this module's implementation in its entirety at
+You may read this utility's implementation in its entirety at
perldoc -m jcat
+L<App::PipeFilter::JsonCat> implements this utility.
+
L<App::PipeFilter> has top-level documentation including a table of
-contents for all the libraries and binaries included in the project.
+contents for all the libraries and utilities included in the project.
=head1 BUGS
View
@@ -10,33 +10,36 @@ jcut - cut out selected portions of each JSON object in a file
=head1 SYNOPSIS
- jcut -o field_1 [-o field_2 ...] < input.json
-
- jcut -o field --input eg/sample.json
-
- jcut -o field --input eg/sample.json --output new.json
+ jcut -o field_1 [-o field_2 ...] [--verbose] [file ...]
=head1 DESCRIPTION
jcut(1) cuts out selected portions of each JSON record and writes them
-to standard output or a named file. Multiple fields may be specified
-with additional -o flags.
-
-jcut(1) inherits its --input, --output and --verbose flags from
-L<App::PipeFilter::Role::Flags::Standard>.
-See that module for more detailed documentation.
-
-jcut(1) inherits its -o flag from L<App::PipeFilter::JsonCut>.
-See that module for more detailed documentation.
+to standard output. Multiple output fields may be specified, one per
+-o flag.
=head1 SEE ALSO
-You may read this module's implementation in its entirety at
+You may read this utility's implementation in its entirety at
perldoc -m jcut
+L<App::PipeFilter::JsonCut> implements this utility, including the
+behavior of the -o flag.
+
+The jsonpath(1) utility performs a similar function, but output fields
+are specified by JSON::Path expressions instead of simple field names.
+jsonpath(1) can therefore extract data from more complex JSON objects.
+
L<App::PipeFilter> has top-level documentation including a table of
-contents for all the libraries and binaries included in the project.
+contents for all the libraries and utilities included in the project.
+
+=head1 PRO TIPS
+
+JSON is relatively verbose compared to the whitespace-separated
+formats that UNIX tools usually deal with. It's often beneficial to
+jcut(1) the fields you need early in a pipeline chain and discard any
+extraneous data.
=head1 BUGS
View
@@ -10,31 +10,28 @@ jmap - map input JSON fields to output JSON fields by renaming them
=head1 SYNOPSIS
-Convert input fields "one" and "two" to uppercase versions on output:
-
- jmap -i one -o ONE -i two -o TWO < input.json
+ jmap -i original_name -o new_name [-i orig -o new ...] [file ...]
=head1 DESCRIPTION
-jmap(1) reads files sequentially, altering some or all of their field
-names before writing them to standard output or a named file. It
-behaves like jcat(1) but renames fields in the process.
-
-jmap(1) inherits its --input, --output and --verbose flags from
-L<App::PipeFilter::Role::Flags::Standard>.
-See that module for more detailed documentation.
+jmap(1) behaves like jcat(1) but renames fields in the process of
+concatenating input files. jmap(1) reads files sequentially, altering
+some or all of their field names before writing them to standard
+output. jmap(1) can rename multiple fields by repeating its -i and -o
+flags.
-jmap(1) inherits its -i and -o flags from L<App::PipeFilter::JsonMap>.
+The -i and -o flags' unique behavior is implemented in
+L<App::PipeFilter::JsonMap>.
See that module for more detailed documentation.
=head1 SEE ALSO
-You may read this module's implementation in its entirety at
+You may read this utility's implementation in its entirety at
perldoc -m jmap
L<App::PipeFilter> has top-level documentation including a table of
-contents for all the libraries and binaries included in the project.
+contents for all the libraries and utilities included in the project.
=head1 BUGS
View
@@ -6,32 +6,37 @@ __END__
=head1 NAME
-json2yaml - convert a stream of JSON objects into a stream of YAML
-objects
+json2yaml - convert files of JSON objects into a stream of YAML objects
=head1 SYNOPSIS
- json2yaml < eg/sample.json
-
- json2yaml --input eg/sample.json --output new.yaml
-
- json2yaml --input eg/sample.json --output new.yaml --verbose
+ json2yaml [--verbose] [file ...]
=head1 DESCRIPTION
json2yaml(1) reads files sequentially, writing them as YAML to
-standard output or a named file.
+standard output.
+
+=head1 PRO TIPS
+
+The YAML stream is vertical, with one field per line, whereas most of
+the JSON filters write entire objects per line.
-json2yaml(1) inherits its --input, --output and --verbose flags from
-L<App::PipeFilter::Role::Flags::Standard>. See that module for more
-detailed documentation.
+This filter is often used as the equivalent of mysql(1)'s \G command:
+
+ ego (\G) Send command to mysql server, display result vertically.
+
+The json_xs(1) utility can reformat JSON vertically as well, if you
+prefer JSON over YAML.
=head1 SEE ALSO
-You may read this module's implementation in its entirety at
+You may read this utility's implementation in its entirety at
perldoc -m json2yaml
+L<App::PipeFilter::JsonToYaml> implements this utility.
+
L<App::PipeFilter> has top-level documentation including a table of
contents for all the libraries and binaries included in the project.
Oops, something went wrong.

0 comments on commit c3c9eb9

Please sign in to comment.