Skip to content

Contains an example written in Visual Basic.Net of using the TD Ameritrade Web API or polygon.io API to update end-of-day stock quotes in a Microsoft SQL Server database

Notifications You must be signed in to change notification settings

stplbn249683/just-puttering

Repository files navigation

Important update on 11/5/2023
This program is rapidly becoming obsolete because TD Ameritrade accounts are
being moved to Schwab. In fact, I can no longer use it myself. Schwab is
supposed to have or be developing a Schwab Developer Portal with a similar
trading API. I am still trying to figure out whether it is presently available
for use.  I am leaving the code here for the present in case it is helpful to
someone with some other application.

Update on 1/4/2024
I have grown impatient about waiting for Schab to make the Trading API
documentation available under the Schab Developer Portal and to approve
pending requests to use it.  I have programs for backtesting different
strategies and using those strategies to tell me when to buy or sell
stocks/ETFs that I cannot use without up-to-date stock historical data. So I
have added an option to the program to use the free version of the polygon.io
trading API to update the stock historical data in the database.

There are significant disadvantages to using the polygon.io API compared to
the TD Ameritrade API.  The free version allows access to only 2 years of
historical data so I cannot use it to add a full record of historical data
for a new stock.  However I can use it to keep a stock that is already in the
database up-to-date.  Also, the free version only allows 5 API calls per minute
so updating the historical data for 100 stocks would take 20 minutes (if
someone was using one of the paid versions then they would probably want to
change the 65 second delay that I built into the program after every 5 API
calls). The TD Ameritrade API allowed 120 API calls per minute.  The free
version does not return OHLC values for the current day until after midnight.
The TD Ameritrade API returned the current day's OHLC values after the market
close and even returned the OHLC values up to the current time if it was called
before market close.

Using the free version of the polyogn.io API is simpler than using the TD
Ameritrade API since it only needs the API key. The program assumes that the API
key is stored in an external text file.  The filename (incuding the path) can be
specified in the InitializeDefaults subroutine or in the external
GetStockData.ini text file (which is assumed to be located in the application
folder; the same file as the executable). I cannot check the TD Ameritrade
option any more since my account was moved to Schwab so I am hoping that these
changes have not inadvertently disabled it.

Update on 1/15/2024
I need the OHLC values for the current day before the market closes so I will
have time to run the historical data through my programs before the market
closes and decide whether to buy or sell stocks.  I could do this with the
TD Ameritrade API by downloading the data to the database before the market
closed and then downloading the data again after the market closed to update
the database to the actual closing values.

Since I can longer do that, I added an option to import the OHLC values for a
single day from a CSV file that has been exported from a Yahoo portfolio. I
made it a menu item because I did not want to clutter the screen with another
button. The "Current Price" column is used as the close. The Yahoo portfolio
needs to contain all the ticker symbols that are in the ticker symbol list file
and some data for the ticker symbol needs to be already present in the database
or an error is returned. This needs to be done carefully because it could
create a gap in the historical data if the data from the previous market day is
not present in the database. The data would also need to be overwritten after
the market close using another Yahoo file or a download from a trading API.
Also, since BRK.B for example is BRK-B, the program presently replaces the "."
in the ticker symbol with "-" before searching for the Yahoo name.  This is a
confusing issue since Schwab uses BRK/B so I don't know how I'll handle that
in the future.

The original description of the program is below:

This folder contains an example of using TD Ameritrade’s Web API (see developer.tdameritrade.com) to maintain and update a database of 
end-of-day stock prices.  I am writing this in case someone finds it helpful since I have combined information that I found from 
various sources.  I personally use the data from the database for the calculations that I do in some of my Excel workbooks.
Disclaimer: This example includes code that I developed for my own personal use and I have included it as a source of information.  
I am not recommending that anyone else use the code as written and I am not responsible for any consequences that result from doing so.  
This post reflects my current understanding and may contain errors.  Since the code was developed for my own use, I did not attempt 
to make it elegant or efficient.

I think that a TD Ameritrade account is required in order to use TD Ameritrade’s Web API. I am using Visual Studio.Net (Visual Basic.Net) 
for the program described here that accesses the Web API and the database.  The database is a Microsoft SQL Server database and I am 
using a Windows 10 computer.  The versions of Visual Studio.Net and Microsoft SQL Server that I am using are free downloads.  I also 
use the Newtonsoft JSON Nuget package to make it easier to decode the JSON that is returned by the Web API.  Also, I set up a local 
Web site (redirect URL) on my local computer to receive the responses from some of the TD Ameritrade Web APIs and Web pages that allow me to 
manually obtain a new refresh token.

This folder contains an image of the input form of the Visual Basic.Net program.  An access token is required in order to obtain data 
from the Web API.  A refresh token is required in order to obtain an access token. An authorization code is required in order to obtain 
the refresh token.   A consumer key is required in order to obtain the authorization code.  A new refresh token needs to be obtained 
every 3 months whereas a new access token only works for half an hour.

Although there is a button on the program’s input form labelled “Get Refresh Token”, I have not created any code for getting the refresh 
token.  Since a new refresh token is only required every 3 months and using the TD Ameritrade’s Web pages to obtain a new refresh token 
requires logging into a TD Ameritrade account to obtain a new authorization code, I thought it was safer from a security standpoint to 
just use TD Ameritrade’s Web pages.  This is a little tedious since the authorization code that is returned by one Web page has to the 
URL decoded and then entered into another Web page (I use a small Visual Basic.Net program to do the URL decode; this is just 
s2 = Uri.UnescapeDataString(s1)).  I saved the refresh token and consumer key in separate text files for use by the program.

In the program, the button labelled “Get Access Token” calls a subroutine that obtains a new access token for the TD Ameritrade API. 
The subroutine reads the consumer key and refresh token from separate text input files and stores the returned access token in a 
different text file.  The access token that is returned by the Web API is output to a text file for later use by the other subroutines.  
A new access token needs to be obtained first for the download subroutines to work unless it has been less than a half hour since the 
last access token was obtained.

The buttons that are labelled “Download historical data for database” call a subroutine that downloads the end-of-day stock prices for 
a list of ticker symbols.  The end-of-day stock price data is downloaded to a different CSV (comma separated value) file for each ticker 
symbol.  There are 2 buttons because I use 2 different lists of ticker symbols named “ticker_list.txt” and “ticker_list1.txt”.  The 
subroutine checks the “market_price” table in the SQL Server “market_data” database to see whether there is already end-of-day stock 
price data for that ticker symbol in the database.  If end-of-day stock price data already exists, then the subroutine just downloads 
the most recent data up to the current day.  The data is downloaded starting 4 days before the current day in case some recent values 
have changed. If there is no end-of-day stock price data for the ticker symbol in the database, then all of the data up to the current 
day is downloaded to a CSV file.  After downloading the data for 116 ticker symbols, the subroutine waits for one minute 20 seconds 
because the TD Ameritrade API returns an error if more than 120 requests are made in one minute (and I allowed some extra wiggle room).  
The inputs to the subroutine are the filenames of the text files for the consumer key, access token and ticker list, the name of the 
folder where the output CSV files will be stored, and the database data source name.  The database data source name is the name listed 
as the server name for the database engine under Microsoft SQL Server Management Studio.  I’m not sure that the date conversion to the 
number of ticks used by the TD Ameritrade Web APIs is exactly accurate but it is close enough to use for end-of-day stock prices.  
Since the end-of-day data is used, the program is normally executed after the stock market closes.

I use the subroutine UpdateTickerList to construct a new ticker_list.txt file by reading the ticker symbols for the setup sheets of
the Excel workbooks that are listed in the indicator_files.txt. So indicator_files.txt and the subroutine UpdateTickerList are probably 
not needed by someone else.

The buttons that are labelled “Update database” read the CSV file for each ticker symbol and add the end-of-day stock price data to the 
market_price table in the database.  The inputs to the subroutine are the name of the folder where the output CSV files will be stored 
and the database data source name.

There are also buttons labelled “Download fundamental data for database” that download fundamental data for each ticker symbol.  
There are corresponding buttons labelled “Update Database” that store the fundamental data in the “fundamentals” table in the database.  
I don’t currently make use of the fundamental data in my Excel workbooks and the subroutines are similar to subroutines that I described 
above so I’m not going to give a detailed explanation of these subroutines but I have included the code.

The user needs to specify the data source name, the folder locations and the text file names either in the InitializeDefaults subroutine 
or else in the external text file “GetStockData.ini” that is read by the ReadDefaults subroutine if it exists.  IMPORTANT: The program
deletes the files in the folders so that old files won't get mixed in with new files so the folders should not contain other files
that the user wants to save!

Below, I have included information about the structure of the database tables. I have also included an Excel VBA function that shows 
how I read the end-of-day stock prices from the database data into Microsoft Excel.  I normally use the last 120 market days in my 
Excel calculations so that errors in calculations (such as exponential moving averages) will have time to die out.

The Excel VBA function that I use to read the end-of-day stock prices from the database data into Microsoft Excel is below.

Function UpdateWorkSheetFromDatabase%(DataSource$, NumTickers%, tickers$(), NumRowsPerTicker%(), StartRow&, oSheet As Worksheet)
  UpdateWorkSheetFromDatabase = 0
  Dim cn As ADODB.Connection
  Dim rst As ADODB.Recordset
  Dim cmd As ADODB.Command
  Dim i%, j&, RowOffset&
  Dim date1$, open1#, high1#, low1#, close1#, volume1&
  Dim year1$, month1$, day1$, s1$, s2$, msg$
 
  ' Open the connection.
  Dim ConnectionString$
  Set cn = New ADODB.Connection
  ConnectionString = "Provider='SQLOLEDB';Data Source='" & DataSource & "';Initial Catalog='market_data';Integrated Security='SSPI';"
  cn.Open ConnectionString

  RowOffset = 0
  For i% = 1 To NumTickers%
    s1 = tickers(i)
    
    ' Set the command text.
    Set cmd = New ADODB.Command
    Set cmd.ActiveConnection = cn
    With cmd
      ' I want the BOTTOM records of the original table; fortunately I also want them in descending order so I don't need the line that I commented out
      '.CommandText = "SELECT * FROM (SELECT TOP " & Trim$(Str$(num%)) & " * FROM market_price t1 WHERE Ticker='" & s1 & "' ORDER BY t1.Date DESC) t2 ORDER BY t2.Date ASC"
     .CommandText = "Select Top " & Trim$(Str$(NumRowsPerTicker(i))) & " * from market_price where Ticker='" & s1 & "' Order By Date DESC"
     .CommandType = adCmdText
     .Execute
    End With
 
    ' Open the recordset.
    Set rst = New ADODB.Recordset
    Set rst.ActiveConnection = cn
    rst.Source = "market_price"
    rst.CursorType = adOpenStatic
    rst.LockType = adLockReadOnly
    rst.Open cmd
    Dim num_records&
    num_records = rst.RecordCount
    If num_records <> NumRowsPerTicker(i) Then
      rst.Close
      cn.Close
      Set cmd = Nothing
      Set rst = Nothing
      Set cn = Nothing
      msg = "Number of records <> " & Trim$(Str$(NumRowsPerTicker(i))) & " for ticker " & tickers(i)
      MsgBox msg
      Exit Function
    End If

    
    rst.MoveFirst
    For j = 1 To NumRowsPerTicker(i)
      date1 = Trim$(Str$(rst.Fields.Item("Date")))
      open1 = rst.Fields.Item("Open")
      high1 = rst.Fields.Item("High")
      low1 = rst.Fields.Item("Low")
      close1 = rst.Fields.Item("Close")
      volume1 = rst.Fields.Item("Volume")
      
      s2 = ""
      If Len(date1) = 8 Then
        year1 = Mid$(date1, 1, 4)
        month1 = Mid$(date1, 5, 2)
        If Mid$(month1, 1, 1) = "0" Then month1 = Mid$(month1, 2, 1)
        day1 = Mid$(date1, 7, 2)
        If Mid$(day1, 1, 1) = "0" Then day1 = Mid$(day1, 2, 1)
        s2 = month1 & "/" & day1 & "/" & year1
      End If
      
      oSheetList.Cells(StartRow + RowOffset + j - 1&, 1).Value = s1
      oSheetList.Cells(StartRow + RowOffset + j - 1&, 2).Value = s2
      oSheetList.Cells(StartRow + RowOffset + j - 1&, 3).Value = open1
      oSheetList.Cells(StartRow + RowOffset + j - 1&, 4).Value = high1
      oSheetList.Cells(StartRow + RowOffset + j - 1&, 5).Value = low1
      oSheetList.Cells(StartRow + RowOffset + j - 1&, 6).Value = close1
      oSheetList.Cells(StartRow + RowOffset + j - 1&, 7).Value = volume1
      rst.MoveNext
    Next
    RowOffset = RowOffset + NumRowsPerTicker%(i)
  Next

  ' Close the connections and clean up.
  rst.Close
  cn.Close
  Set cmd = Nothing
  Set rst = Nothing
  Set cn = Nothing
  Exit Function

ErrorHandler:
  UpdateWorkSheetFromDatabase = -1
  msg = err.Source & ": Error " & err.Number & ": " & err.Description
  err.Clear
  MsgBox msg
End Function

The structure of the database tables is below.
Table dbo.market_price
COLUMN_NAME,IS_NULLABLE,DATA_TYPE,NUMERIC_PRECISION,CHARACTER_MAXIMUM_LENGTH
Ticker,NO,varchar,NULL,10
Date,NO,int,10,NULL
Open,NO,decimal,18,NULL
High,NO,decimal,18,NULL
Low,NO,decimal,18,NULL
Close,NO,decimal,18,NULL
Volume,NO,bigint,19,NULL

Table dbo.fundamentals
COLUMN_NAME,IS_NULLABLE,DATA_TYPE,NUMERIC_PRECISION,CHARACTER_MAXIMUM_LENGTH
ticker,NO,varchar,NULL,10
high52,YES,decimal,18,NULL
low52,YES,decimal,18,NULL
dividendAmount,YES,decimal,18,NULL
dividendYield,YES,decimal,18,NULL
dividendDate,YES,varchar,NULL,30
peRatio,YES,decimal,18,NULL
pegRatio,YES,decimal,18,NULL
pbRatio,YES,decimal,18,NULL
prRatio,YES,decimal,18,NULL
pcfRatio,YES,decimal,18,NULL
grossMarginTTM,YES,decimal,18,NULL
grossMarginMRQ,YES,decimal,18,NULL
netProfitMarginTTM,YES,decimal,18,NULL
netProfitMarginMRQ,YES,decimal,18,NULL
operatingMarginTTM,YES,decimal,18,NULL
operatingMarginMRQ,YES,decimal,18,NULL
returnOnEquity,YES,decimal,18,NULL
returnOnAssets,YES,decimal,18,NULL
returnOnInvestment,YES,decimal,18,NULL
quickRatio,YES,decimal,18,NULL
currentRatio,YES,decimal,18,NULL
interestCoverage,YES,decimal,18,NULL
totalDebtToCapital,YES,decimal,18,NULL
ltDebtToEquity,YES,decimal,18,NULL
totalDebtToEquity,YES,decimal,18,NULL
epsTTM,YES,decimal,18,NULL
epsChangePercentTTM,YES,decimal,18,NULL
epsChangeYear,YES,decimal,18,NULL
epsChange,YES,decimal,18,NULL
revChangeYear,YES,decimal,18,NULL
revChangeTTM,YES,decimal,18,NULL
revChangeIn,YES,decimal,18,NULL
sharesOutstanding,YES,decimal,18,NULL
marketCapFloat,YES,decimal,18,NULL
marketCap,YES,decimal,18,NULL
bookValuePerShare,YES,decimal,18,NULL
shortIntToFloat,YES,decimal,18,NULL
shortIntDayToCover,YES,decimal,18,NULL
divGrowthRate3Year,YES,decimal,18,NULL
dividendPayAmount,YES,decimal,18,NULL
dividendPayDate,YES,varchar,NULL,30
beta,YES,decimal,18,NULL
vol1DayAvg,YES,decimal,18,NULL
vol10DayAvg,YES,decimal,18,NULL
vol3MonthAvg,YES,decimal,18,NULL
cusip,YES,varchar,NULL,30
description,YES,varchar,NULL,100
exchange,YES,varchar,NULL,30
assetType,YES,varchar,NULL,30

The database views used in the program are below.

View dbo.get_last_date
SELECT        Ticker, MAX(Date) AS Last_Date, COUNT(*) AS Num_of_records
FROM            dbo.market_price AS mp
GROUP BY Ticker

View dbo.get_fundamental_field_names
SELECT        COLUMN_NAME
FROM            INFORMATION_SCHEMA.COLUMNS
WHERE        (TABLE_NAME = N'fundamentals')

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

About

Contains an example written in Visual Basic.Net of using the TD Ameritrade Web API or polygon.io API to update end-of-day stock quotes in a Microsoft SQL Server database

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published