Soliciting FAQ material

[johnkerl]

What I have so far:

• No output at all: record separator is CRLF but file data contains LF line endings
• Fields not being picked out: field separator doesn't match data, e.g. the file is TSV but FS is comma (default)
• mlr put '$y=string($x);$z=$y.$y' gives (error) on numeric data such as x=123 while mlr put '$z=string($x).string($x)' does not
• How to handle data in application log files, e.g. 2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [search] various sorts of data {& punctuation} hits=1 status=0 time=2.378

Any other roadblocks/stumpers/head-scratch-moments?

[johnkerl]

This will become a new page within http://johnkerl.org/miller/doc

[johnkerl]

http://johnkerl.org/miller/doc/faq.html

[jungle-boogie]

I would say chaining together several statements would be a good FAQ.

group by, count and give a sum on the last column.

for instance...
mlr --icsv --opprint count-distinct -f Status,Payment_Type then sort -nr count then stats1 -a sum -f Amount

This isn't working

[johnkerl]

@jungle-boogie please let me know how this helps:
http://johnkerl.org/miller/doc/faq.html#How_do_I_examine_then-chaining?

[jungle-boogie]

nice work!

What's the difference between icsv and csv?

[johnkerl]

icsv means csv input. ocsv means csv output. csv means both. more info at mlr -h. let me know if that could be clearer (i think it could be).

[helix84]

Hi, I just wanted to note that the background image containing text makes actual site text hard to read (not just on the FAQ page).

[jungle-boogie]

Hi @helix84,

What do you think would be a more suitable background?

[johnkerl]

@helix84 there are a few options here ... I can lighten the background image with content as-is; lighten the text layer of the image, or remove it entirely; or remove the entire background image. The latter would sadden my internal amateur graphic designer, but legilibility must win out for technical writing. ;)

[helix84]

I'm now looking at the site on a different screen so I may be mistaken, but it seems to me you lightened the background. It's much more legible now. I now see another issue - 3 scrollbars (menu, content, site).

[johnkerl]

No I haven't changed the background. Must be your screen which is why I particularly value your feedback -- it needs to be legible for everyone.

There is intended to be separate scrolling for the left navpane & the main body. But I'm far from being a CSS expert. It's imperfect for me & I'm tempted to have just a single scroll ... especially if it behaves worse for other folks ... can you include a screenshot so I can see what the issue is?

Thank you!! :)

[jungle-boogie]

Miller's filter will print matching records, but can it be used to exclude records from the output? Cut can exclude but I think that excludes the whole record, not specific data.

[johnkerl]

You can change your == to !=, < to >=, and so on. But adding a -x to negate
the sense of the filter (exclude rather than include) is a simple and
intuitive idea. Later this evening. :)

Miller's filter will print matching records, but can it be used to exclude
records from the output? Cut can exclude but I think that excludes the
whole record, not specific data.

—
Reply to this email directly or view it on GitHub
#74 (comment).

[johnkerl]

mlr filter -x: f6a6bd3 d7739da http://johnkerl.org/miller/doc/reference.html#filter

[jungle-boogie]

Hi @johnkerl,

excellent work on the exclude!

I may be missing something really obvious but it seems that I can't get the output in csv format, just opprint.

If you take the sample data from here:
http://johnkerl.org/miller/doc/faq.html#How_do_I_do_arithmetic_on_fields_with_currency_symbols?

Do this:
mlr --opprint filter -x '$3 == "Reason: Payment Stopped"' sample.csv, you get 7 records with 2 excluded--as expected.

And now this:
mlr --csv --rs lf filter -x '$3 == "Reason: Payment Stopped"' sample.csv

I don't get any data even though my file is ascii:
% file sample.csv
sample.csv: ASCII text

Any variation to include csv (icsv, csv and csvlite) also doesn't seem to give me any output.

% mlr --csv filter -v -x '$3 == "Reason: Payment Stopped"' sample.csv
== (operator):
3 (field_name).
Reason: Payment Stopped (literal).

Looks like the filter works correct, though.

[johnkerl]

When you do mlr --opprint then you're specifying pretty-print output format but not specifying the input format, so it defaults to DKVP. When the DKVP reader encounters a field of the form name=value then it uses that name-value pair, but absent the pair-separator, it'll use the positional index as the field name. So with DKVP input, all the fields are named 1, 2, 3, etc., there are ten records in the file, and it's appropriate to filter on $3.

When you specify CSV format, then the field names are taken from the CSV header line and the field values are taken from the subsequent data lines. With CSV format, there are nine records in the file, your third column is named Description, and it's appropriate to filter on $Description:

$ mlr --icsv --rs lf --opprint filter -x '$Description == "Reason: Payment Stopped"' sample.csv 
EventOccurred EventType    Description                               Status   PaymentType NameonAccount TransactionNumber Amount
10/1/2015     Charged Back Reason: Authorization Revoked By Customer Disputed Checking    John          1                 $230.36
10/1/2015     Charged Back Reason: Authorization Revoked By Customer Disputed Checking    Fred          2                 $32.25
10/1/2015     Charged Back Reason: Customer Advises Not Authorized   Disputed Checking    Bob           3                 $39.02
10/1/2015     Charged Back Reason: Authorization Revoked By Customer Disputed Checking    Alice         4                 $57.54
10/1/2015     Charged Back Reason: Authorization Revoked By Customer Disputed Checking    Jungle        5                 $230.36
10/2/2015     Charged Back Reason: Customer Advises Not Authorized   Disputed Checking    Joseph        7                 $188.19
10/2/2015     Charged Back Reason: Customer Advises Not Authorized   Disputed Checking    Joseph        8                 $188.19

[jungle-boogie]

That gave me a hint to what I was trying to do, I just didn't take it far enough.

This is what I was trying to accomplish:
mlr --rs lf --csv filter -x '$Description == "Reason: Payment Stopped"' sample.csv

[johnkerl]

No updates in a year and a half -- closing this as it's not an effective avenue for people to use.