Make single-shot to pipe / redirect to other commands

[adambhayes]

It would be nice if we could get a single snapshot of output by something like
mop --single > output.txt
This would be really handy for scripting alerts and actions based on the output.

[brandleesee]

Sure, go ahead!
(and thank you)

[joce]

@adambhayes I imagine you'd like something like

$ mop --single AAPL > output.txt

that would result in output.txt containing e.g.:

AAPL $160.07 -$4.25 -2.58% $165.54 $159.75 $166.15 $116.21 $182.94 89.46M 79.04M 26.61 $0.86 0.00% $2.61T - -

? (The last two dashes being the pre and post market. They would be there both during trading hours, or be replaced by the market change percent corresponding to the AH period, if any)

Would you also want

$ mop --single AAPL GOOG > output.txt

to give

AAPL $160.07 -$4.25 -2.58% $165.54 $159.75 $166.15 $116.21 $182.94 89.46M 79.04M 26.61 $0.86 0.00% $2.61T - -
GOOG $2551.70 -$36.35 -1.40% $2621.57 $2550.07 $2634.98 $2010.00 $3042.00 1.27M 1.57M 22.74 $0.00 0.00% $1.68T - -

?

I see this info as a bit dense. So maybe a way to select which info you want for your output?

e.g.

$ mop --single AAPL GOOG -LP > output.txt

(L for "last" and P for "change%") to give

AAPL $160.07 -2.58%
GOOG $2551.70 -1.40%

?

(We would need to come up with good switches for sure)

[brandleesee]

My 2c on this matter.

Maybe we put a -verbose flag to output all available jnfo per stock(s) (ie the dense version) and the individual switches for each column.

Also, please, I would need the readme.md be updated with list of flags/switches.

What do you think, makes sense?

[brandleesee]

github on mobile phone is a bit quirky sorry

[joce]

Thinking more about it, I think maybe it should provide a format string if you want something other than the full one. e.g. :

$ mop --single --format "MyQuote: %t %l. Change is %c."

would result in:

MyQuote: AAPL $164.00. Change is $1.29.

(We'd need a token for each of the columns, but I think it's doable)

[brandleesee]

As long as the commands are simple, then please go ahead.
In the sense that I do not wish to have the command to be complicated that users would not make use of such a feature.

[joce]

All right, suggestions, then:

usage could be

mop [options] [tickers]

--single | -S would output the ticker + current trading value for the given symbols

e.g.

$ mop --single AAPL GOOG
AAPL $164.85
GOOG $2690.39

used with the --verbose | -V, it would output the whole line (I'm quite unsure it should output the pre/post market variations however)

$ mop -SV AAPL GOOG
AAPL $164.85 $2.11 1.29% $163.84 $160.87 $165.04 $116.21 $182.94 90.79M 87.66M 27.40 $0.86 0.00% $2.69T - -0.30%
GOOG $2690.39 $36.92 1.39% $2670.51 $2635.30 $2707.78 $2010.00 $3042.00 1.31M 1.57M 23.97 $0.00 0.00% $1.77T - 0.00%

And finally, the most "advanced" option would be the --format="format string" (which should be mutually exclusive with the verbose one. We'd need to decide which one takes precedence if the user puts both. I would be inclined to say format takes precedence).

$ mop -S --format="MyQuote: %t %l. Change is %c." AAPL RIVN
MyQuote: AAPL $164.85. Change is $2.11.
MyQuote: RIVN $63.43. Change is -$0.28.

We'd need to figure out what to do if we start mop with out any options, but with tickers, e.g.

$ mop AAPL RIVN

I think we could start the full mop, with the symbols passed in that would override the ones in the .moprc file (without overwriting it).

Starting to think we should come up with actual specs for the CLI arguments.

[brandleesee]

I am in favour of all you suggested and am assuming that mop -verbose downloads all the tickers. This is one feature I would personally find helpful as it would serve as a report on all the tickers being observed.

[joce]

Sure. It makes sense if not specifying ticker symbols would use all the symbols specified in the .moprc file.

[adambhayes]

@brandleesee & @joce : I like your plan!

[adambhayes]

At some point, something more machine-readable as output might be helpful, like csv with the "$," "%," etc. removed from the values. Just a thought. But this is great.

[joce]

There could be a --raw option to strip any non-numerical symbols.

[joce]

@adambhayes So I thought about this long and hard. The tl;dr; is I don't think it would be the right thing for MOP to implement this.

The UNIX philosophy is to approach software from a minimalist and modular way, and there are already a number of existing tools that exist that can get you exactly where you want to go. Also know that the tools I'll be using are all available on Linux and Windows (and I would certainly assume macOS as well!).

Let me elaborate and demonstrate for your benefit.

To query the data, we'll be using curl. To the best of my knowledge, this is installed by default on pretty much all OSes.

So, for example, to query the quotes for Apple, Alphabet and Ford, do the following

$ curl https://query1.finance.yahoo.com/v7/finance/quote?symbols=AAPL,GOOG,F 2>/dev/null

You just have to change the ticker symbols you want in the URL (AAPL,GOOG,F in the example above) to get the complete list of what you need. The data you get will be a JSON stream.

Note that the 2>/dev/null at the end is to silence the error stream so that you only get the raw data spewed to the console.

Next, you'll want to transform this into a CSV file. This is a bit trickier and I had to dig around to find a proper way to do this. You'll want to install jq, a command line JSON processor. It's amazingly versatile, and it allows you to manipulate JSON data in anyway you want.

The tricky part, is to convert the data we got from curl and transform it in a CSV stream. After much digging, I found the following that does the transformation:

$ curl https://query1.finance.yahoo.com/v7/finance/quote?symbols=AAPL,GOOG,F 2>/dev/null > jq -r .quoteResponse.result | jq -r '(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | @csv'

Now, that's a mouthful, and tbh, I can't say I understand exactly what all of it does does... The first jq -r .quoteResponse.result if to get to the actual result of the stream that you get from curl, and the last part is the magical incantation the does the actual transformation to CSV.

However, I understand that you can do all kinds of amazing transformation to your JSON data that we could not even start to tackle should we be doing it in MOP.

Now, if we bring all of this in a script to ease of convenience, you get the following in a batch file (Windows):

@ECHO OFF

IF "%~1"=="" (
    ECHO You need to pass in one or more ticker symbols to get their quotes. 1>&2
    GOTO :eof
)

SET YHOO_FIN_ARGS=

FOR %%A IN (%*) DO call :MakeList %%A
goto YahooQuery

:MakeList
IF DEFINED YHOO_FIN_ARGS (
    SET YHOO_FIN_ARGS=%YHOO_FIN_ARGS%,%1
) ELSE (
    SET YHOO_FIN_ARGS=%1
)
goto :eof

:YahooQuery

curl https://query1.finance.yahoo.com/v7/finance/quote?symbols=%YHOO_FIN_ARGS% 2>nul | jq -r .quoteResponse.result | jq -r "(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | @csv"

SET YHOO_FIN_ARGS=

or the following bash shell script (Linux/macOS):

#!/bin/bash

if [[ $# -eq 0 ]] ; then
    >&2 echo 'You need to pass in one or more ticker symbols to get their quotes.'
    exit 0
fi

yhooFinArgs=""

for i in $@
do
    if [ "$yhooFinArgs" = "" ]; then
        yhooFinArgs=$i
    else
        yhooFinArgs=$yhooFinArgs","$i
    fi
done

curl https://query1.finance.yahoo.com/v7/finance/quote?symbols=$yhooFinArgs 2>/dev/null | jq -r .quoteResponse.result | jq -r '(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | @csv'

That you can use such as (if you name your script quote.cmd (Windows) or quote.sh (Linux/macOS) to get your quotes in a csv file:

$ ./quote.sh AAPL GOOG F > myquotes.csv

Now, this will create a CSV with all the fields provided by Yahoo, which can be a bit much. I found another great piece of software that allows you to manipulate CSV files from the command line. It's called Miller. It has a very comprehensive documentation so you can get a good feel of what you can do. But, for example, if you want to get the symbol, current price and market cap, you can go with:

$ mlr --csv cut -o -f symbol,regularMarketPrice,marketCap myquotes.csv
symbol,regularMarketPrice,marketCap
AAPL,176.465,2879802834944
GOOG,2825.56,1862784385024
F,17.04,68232425472

and if you want to have it pretty printed and the market cap formatted differently, you could go with

$  mlr --icsv --opprint cut -o -f symbol,regularMarketPrice,marketCap then put '$marketCap=fmtnum($marketCap/1000000,"%.2fM")' myquotes.csv
symbol regularMarketPrice marketCap
AAPL   176.465            2879802.83M
GOOG   2825.56            1862784.39M
F      17.04              68232.43M

So, there you have it. I think with the right tools in hand, I think you can go much further that anything we could come up with in MOP.

If @brandleesee is fine with this, I'd close this bug.

[brandleesee]

@joce Noting the very detailed post, would you consider writing an entry in the wiki section? (I would then insert a link in the README.md.)

At the moment the wiki is locked but if you are in agreement, I would unlock it and you could write a detailed article on how to go about installing/downloading the data.

In the meantime, you can close this thread.

@adambhayes should you wish to pursue your proposal, please feel free to reopen this thread and submit a PR for review. I definitely have no objection!

[adambhayes]

Hi, @joce and Brandon. I see what you mean, @joce. This does make sense, and I'm personally satisfied with your approach using curl and jq. None of us have tons of time now anyway, but it's a fine solution. On April 2, 2022 4:42:12 AM EDT, Brandon Lee Camilleri ***@***.***> wrote: ***@***.*** Noting the very detailed post, would you consider writing an entry in the wiki section? (I would then insert a link in the README.md.)

At the moment the wiki is locked but if you are in agreement, I would unlock it and you could write a detailed article on how to go about installing/downloading the data. In the meantime, you can close this thread.

***@***.*** should you wish to pursue your proposal, please feel free to reopen this thread and submit a PR for review. I definitely have no objection!

…

-- Reply to this email directly or view it on GitHub: #88 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[joce]

@joce Noting the very detailed post, would you consider writing an entry in the wiki section? (I would then insert a link in the README.md.)

I do not mind, though I'm not certain it has much to do with mop per se if we decide this is not a direction we wish to take.
I've already written a blog post (my first!) about it (https://jocelyn.legau.lt/posts/pulling-yahoo-finance-tickers-to-csv/), basically rehashing what I had written here. I could modify it a bit more to adapt it to a wiki post if you wish to have it here as well.

[adambhayes]

I know what you mean, @joce, about it being a little outside the scope, because it's something *not* being implemented. But, if there aren't any restrictions on what's in docs, it's a nice thing to have written down, because it takes time to figure out.

…

On April 4, 2022 1:18:29 PM EDT, Jocelyn Legault ***@***.***> wrote: > @joce Noting the very detailed post, would you consider writing an entry in the wiki section? (I would then insert a link in the README.md.) I do not mind, though I'm not certain it has mush to do with mop per se if we decide this is not a direction we wish to take. I've already written a blog post (my first!) about it (https://jocelyn.legau.lt/posts/pulling-yahoo-finance-tickers-to-csv/), basically rehashing what I had written here. I could modify it a bit more to adapt it to a wiki post if you wish to have it here as well. -- Reply to this email directly or view it on GitHub: #88 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[brandleesee]

Thank you for the blog's link, @joce. And yes, please, I would rather have a dedicated page on how to download data etc in mop's repository. The reason behind this is so that this repository has its own instructions just in case the blog no longer remains available for whatever reason. I am not forcing you to do it, mind you! But if you do, I would be grateful.

The wiki for mop repository is now available:
https://github.com/mop-tracker/mop/wiki

Thank you both @adambhayes and @joce for your contributions.

@joce you may wish to close this issue, too.

[brandleesee]

@joce would you be happy to share the markdown file for your article so I can shape it and then upload it to the wiki section? I do not mind doing this myself.

In the meantime, have placed a reference in the README.md file / e65c1bc

Am also closing this as I am considering the issue as partially resolved.

[SomwareHR]

I think even in current form, but as a CSV or TSV file it would be perfect.
We can more easy extract needed data from what we have than add data to what we don't have.