-
Notifications
You must be signed in to change notification settings - Fork 0
stplbn249683/just-puttering
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Important update on 11/5/2023 This program is rapidly becoming obsolete because TD Ameritrade accounts are being moved to Schwab. In fact, I can no longer use it myself. Schwab is supposed to have or be developing a Schwab Developer Portal with a similar trading API. I am still trying to figure out whether it is presently available for use. I am leaving the code here for the present in case it is helpful to someone with some other application. Update on 1/4/2024 I have grown impatient about waiting for Schab to make the Trading API documentation available under the Schab Developer Portal and to approve pending requests to use it. I have programs for backtesting different strategies and using those strategies to tell me when to buy or sell stocks/ETFs that I cannot use without up-to-date stock historical data. So I have added an option to the program to use the free version of the polygon.io trading API to update the stock historical data in the database. There are significant disadvantages to using the polygon.io API compared to the TD Ameritrade API. The free version allows access to only 2 years of historical data so I cannot use it to add a full record of historical data for a new stock. However I can use it to keep a stock that is already in the database up-to-date. Also, the free version only allows 5 API calls per minute so updating the historical data for 100 stocks would take 20 minutes (if someone was using one of the paid versions then they would probably want to change the 65 second delay that I built into the program after every 5 API calls). The TD Ameritrade API allowed 120 API calls per minute. The free version does not return OHLC values for the current day until after midnight. The TD Ameritrade API returned the current day's OHLC values after the market close and even returned the OHLC values up to the current time if it was called before market close. Using the free version of the polyogn.io API is simpler than using the TD Ameritrade API since it only needs the API key. The program assumes that the API key is stored in an external text file. The filename (incuding the path) can be specified in the InitializeDefaults subroutine or in the external GetStockData.ini text file (which is assumed to be located in the application folder; the same file as the executable). I cannot check the TD Ameritrade option any more since my account was moved to Schwab so I am hoping that these changes have not inadvertently disabled it. Update on 1/15/2024 I need the OHLC values for the current day before the market closes so I will have time to run the historical data through my programs before the market closes and decide whether to buy or sell stocks. I could do this with the TD Ameritrade API by downloading the data to the database before the market closed and then downloading the data again after the market closed to update the database to the actual closing values. Since I can longer do that, I added an option to import the OHLC values for a single day from a CSV file that has been exported from a Yahoo portfolio. I made it a menu item because I did not want to clutter the screen with another button. The "Current Price" column is used as the close. The Yahoo portfolio needs to contain all the ticker symbols that are in the ticker symbol list file and some data for the ticker symbol needs to be already present in the database or an error is returned. This needs to be done carefully because it could create a gap in the historical data if the data from the previous market day is not present in the database. The data would also need to be overwritten after the market close using another Yahoo file or a download from a trading API. Also, since BRK.B for example is BRK-B, the program presently replaces the "." in the ticker symbol with "-" before searching for the Yahoo name. This is a confusing issue since Schwab uses BRK/B so I don't know how I'll handle that in the future. The original description of the program is below: This folder contains an example of using TD Ameritrade’s Web API (see developer.tdameritrade.com) to maintain and update a database of end-of-day stock prices. I am writing this in case someone finds it helpful since I have combined information that I found from various sources. I personally use the data from the database for the calculations that I do in some of my Excel workbooks. Disclaimer: This example includes code that I developed for my own personal use and I have included it as a source of information. I am not recommending that anyone else use the code as written and I am not responsible for any consequences that result from doing so. This post reflects my current understanding and may contain errors. Since the code was developed for my own use, I did not attempt to make it elegant or efficient. I think that a TD Ameritrade account is required in order to use TD Ameritrade’s Web API. I am using Visual Studio.Net (Visual Basic.Net) for the program described here that accesses the Web API and the database. The database is a Microsoft SQL Server database and I am using a Windows 10 computer. The versions of Visual Studio.Net and Microsoft SQL Server that I am using are free downloads. I also use the Newtonsoft JSON Nuget package to make it easier to decode the JSON that is returned by the Web API. Also, I set up a local Web site (redirect URL) on my local computer to receive the responses from some of the TD Ameritrade Web APIs and Web pages that allow me to manually obtain a new refresh token. This folder contains an image of the input form of the Visual Basic.Net program. An access token is required in order to obtain data from the Web API. A refresh token is required in order to obtain an access token. An authorization code is required in order to obtain the refresh token. A consumer key is required in order to obtain the authorization code. A new refresh token needs to be obtained every 3 months whereas a new access token only works for half an hour. Although there is a button on the program’s input form labelled “Get Refresh Token”, I have not created any code for getting the refresh token. Since a new refresh token is only required every 3 months and using the TD Ameritrade’s Web pages to obtain a new refresh token requires logging into a TD Ameritrade account to obtain a new authorization code, I thought it was safer from a security standpoint to just use TD Ameritrade’s Web pages. This is a little tedious since the authorization code that is returned by one Web page has to the URL decoded and then entered into another Web page (I use a small Visual Basic.Net program to do the URL decode; this is just s2 = Uri.UnescapeDataString(s1)). I saved the refresh token and consumer key in separate text files for use by the program. In the program, the button labelled “Get Access Token” calls a subroutine that obtains a new access token for the TD Ameritrade API. The subroutine reads the consumer key and refresh token from separate text input files and stores the returned access token in a different text file. The access token that is returned by the Web API is output to a text file for later use by the other subroutines. A new access token needs to be obtained first for the download subroutines to work unless it has been less than a half hour since the last access token was obtained. The buttons that are labelled “Download historical data for database” call a subroutine that downloads the end-of-day stock prices for a list of ticker symbols. The end-of-day stock price data is downloaded to a different CSV (comma separated value) file for each ticker symbol. There are 2 buttons because I use 2 different lists of ticker symbols named “ticker_list.txt” and “ticker_list1.txt”. The subroutine checks the “market_price” table in the SQL Server “market_data” database to see whether there is already end-of-day stock price data for that ticker symbol in the database. If end-of-day stock price data already exists, then the subroutine just downloads the most recent data up to the current day. The data is downloaded starting 4 days before the current day in case some recent values have changed. If there is no end-of-day stock price data for the ticker symbol in the database, then all of the data up to the current day is downloaded to a CSV file. After downloading the data for 116 ticker symbols, the subroutine waits for one minute 20 seconds because the TD Ameritrade API returns an error if more than 120 requests are made in one minute (and I allowed some extra wiggle room). The inputs to the subroutine are the filenames of the text files for the consumer key, access token and ticker list, the name of the folder where the output CSV files will be stored, and the database data source name. The database data source name is the name listed as the server name for the database engine under Microsoft SQL Server Management Studio. I’m not sure that the date conversion to the number of ticks used by the TD Ameritrade Web APIs is exactly accurate but it is close enough to use for end-of-day stock prices. Since the end-of-day data is used, the program is normally executed after the stock market closes. I use the subroutine UpdateTickerList to construct a new ticker_list.txt file by reading the ticker symbols for the setup sheets of the Excel workbooks that are listed in the indicator_files.txt. So indicator_files.txt and the subroutine UpdateTickerList are probably not needed by someone else. The buttons that are labelled “Update database” read the CSV file for each ticker symbol and add the end-of-day stock price data to the market_price table in the database. The inputs to the subroutine are the name of the folder where the output CSV files will be stored and the database data source name. There are also buttons labelled “Download fundamental data for database” that download fundamental data for each ticker symbol. There are corresponding buttons labelled “Update Database” that store the fundamental data in the “fundamentals” table in the database. I don’t currently make use of the fundamental data in my Excel workbooks and the subroutines are similar to subroutines that I described above so I’m not going to give a detailed explanation of these subroutines but I have included the code. The user needs to specify the data source name, the folder locations and the text file names either in the InitializeDefaults subroutine or else in the external text file “GetStockData.ini” that is read by the ReadDefaults subroutine if it exists. IMPORTANT: The program deletes the files in the folders so that old files won't get mixed in with new files so the folders should not contain other files that the user wants to save! Below, I have included information about the structure of the database tables. I have also included an Excel VBA function that shows how I read the end-of-day stock prices from the database data into Microsoft Excel. I normally use the last 120 market days in my Excel calculations so that errors in calculations (such as exponential moving averages) will have time to die out. The Excel VBA function that I use to read the end-of-day stock prices from the database data into Microsoft Excel is below. Function UpdateWorkSheetFromDatabase%(DataSource$, NumTickers%, tickers$(), NumRowsPerTicker%(), StartRow&, oSheet As Worksheet) UpdateWorkSheetFromDatabase = 0 Dim cn As ADODB.Connection Dim rst As ADODB.Recordset Dim cmd As ADODB.Command Dim i%, j&, RowOffset& Dim date1$, open1#, high1#, low1#, close1#, volume1& Dim year1$, month1$, day1$, s1$, s2$, msg$ ' Open the connection. Dim ConnectionString$ Set cn = New ADODB.Connection ConnectionString = "Provider='SQLOLEDB';Data Source='" & DataSource & "';Initial Catalog='market_data';Integrated Security='SSPI';" cn.Open ConnectionString RowOffset = 0 For i% = 1 To NumTickers% s1 = tickers(i) ' Set the command text. Set cmd = New ADODB.Command Set cmd.ActiveConnection = cn With cmd ' I want the BOTTOM records of the original table; fortunately I also want them in descending order so I don't need the line that I commented out '.CommandText = "SELECT * FROM (SELECT TOP " & Trim$(Str$(num%)) & " * FROM market_price t1 WHERE Ticker='" & s1 & "' ORDER BY t1.Date DESC) t2 ORDER BY t2.Date ASC" .CommandText = "Select Top " & Trim$(Str$(NumRowsPerTicker(i))) & " * from market_price where Ticker='" & s1 & "' Order By Date DESC" .CommandType = adCmdText .Execute End With ' Open the recordset. Set rst = New ADODB.Recordset Set rst.ActiveConnection = cn rst.Source = "market_price" rst.CursorType = adOpenStatic rst.LockType = adLockReadOnly rst.Open cmd Dim num_records& num_records = rst.RecordCount If num_records <> NumRowsPerTicker(i) Then rst.Close cn.Close Set cmd = Nothing Set rst = Nothing Set cn = Nothing msg = "Number of records <> " & Trim$(Str$(NumRowsPerTicker(i))) & " for ticker " & tickers(i) MsgBox msg Exit Function End If rst.MoveFirst For j = 1 To NumRowsPerTicker(i) date1 = Trim$(Str$(rst.Fields.Item("Date"))) open1 = rst.Fields.Item("Open") high1 = rst.Fields.Item("High") low1 = rst.Fields.Item("Low") close1 = rst.Fields.Item("Close") volume1 = rst.Fields.Item("Volume") s2 = "" If Len(date1) = 8 Then year1 = Mid$(date1, 1, 4) month1 = Mid$(date1, 5, 2) If Mid$(month1, 1, 1) = "0" Then month1 = Mid$(month1, 2, 1) day1 = Mid$(date1, 7, 2) If Mid$(day1, 1, 1) = "0" Then day1 = Mid$(day1, 2, 1) s2 = month1 & "/" & day1 & "/" & year1 End If oSheetList.Cells(StartRow + RowOffset + j - 1&, 1).Value = s1 oSheetList.Cells(StartRow + RowOffset + j - 1&, 2).Value = s2 oSheetList.Cells(StartRow + RowOffset + j - 1&, 3).Value = open1 oSheetList.Cells(StartRow + RowOffset + j - 1&, 4).Value = high1 oSheetList.Cells(StartRow + RowOffset + j - 1&, 5).Value = low1 oSheetList.Cells(StartRow + RowOffset + j - 1&, 6).Value = close1 oSheetList.Cells(StartRow + RowOffset + j - 1&, 7).Value = volume1 rst.MoveNext Next RowOffset = RowOffset + NumRowsPerTicker%(i) Next ' Close the connections and clean up. rst.Close cn.Close Set cmd = Nothing Set rst = Nothing Set cn = Nothing Exit Function ErrorHandler: UpdateWorkSheetFromDatabase = -1 msg = err.Source & ": Error " & err.Number & ": " & err.Description err.Clear MsgBox msg End Function The structure of the database tables is below. Table dbo.market_price COLUMN_NAME,IS_NULLABLE,DATA_TYPE,NUMERIC_PRECISION,CHARACTER_MAXIMUM_LENGTH Ticker,NO,varchar,NULL,10 Date,NO,int,10,NULL Open,NO,decimal,18,NULL High,NO,decimal,18,NULL Low,NO,decimal,18,NULL Close,NO,decimal,18,NULL Volume,NO,bigint,19,NULL Table dbo.fundamentals COLUMN_NAME,IS_NULLABLE,DATA_TYPE,NUMERIC_PRECISION,CHARACTER_MAXIMUM_LENGTH ticker,NO,varchar,NULL,10 high52,YES,decimal,18,NULL low52,YES,decimal,18,NULL dividendAmount,YES,decimal,18,NULL dividendYield,YES,decimal,18,NULL dividendDate,YES,varchar,NULL,30 peRatio,YES,decimal,18,NULL pegRatio,YES,decimal,18,NULL pbRatio,YES,decimal,18,NULL prRatio,YES,decimal,18,NULL pcfRatio,YES,decimal,18,NULL grossMarginTTM,YES,decimal,18,NULL grossMarginMRQ,YES,decimal,18,NULL netProfitMarginTTM,YES,decimal,18,NULL netProfitMarginMRQ,YES,decimal,18,NULL operatingMarginTTM,YES,decimal,18,NULL operatingMarginMRQ,YES,decimal,18,NULL returnOnEquity,YES,decimal,18,NULL returnOnAssets,YES,decimal,18,NULL returnOnInvestment,YES,decimal,18,NULL quickRatio,YES,decimal,18,NULL currentRatio,YES,decimal,18,NULL interestCoverage,YES,decimal,18,NULL totalDebtToCapital,YES,decimal,18,NULL ltDebtToEquity,YES,decimal,18,NULL totalDebtToEquity,YES,decimal,18,NULL epsTTM,YES,decimal,18,NULL epsChangePercentTTM,YES,decimal,18,NULL epsChangeYear,YES,decimal,18,NULL epsChange,YES,decimal,18,NULL revChangeYear,YES,decimal,18,NULL revChangeTTM,YES,decimal,18,NULL revChangeIn,YES,decimal,18,NULL sharesOutstanding,YES,decimal,18,NULL marketCapFloat,YES,decimal,18,NULL marketCap,YES,decimal,18,NULL bookValuePerShare,YES,decimal,18,NULL shortIntToFloat,YES,decimal,18,NULL shortIntDayToCover,YES,decimal,18,NULL divGrowthRate3Year,YES,decimal,18,NULL dividendPayAmount,YES,decimal,18,NULL dividendPayDate,YES,varchar,NULL,30 beta,YES,decimal,18,NULL vol1DayAvg,YES,decimal,18,NULL vol10DayAvg,YES,decimal,18,NULL vol3MonthAvg,YES,decimal,18,NULL cusip,YES,varchar,NULL,30 description,YES,varchar,NULL,100 exchange,YES,varchar,NULL,30 assetType,YES,varchar,NULL,30 The database views used in the program are below. View dbo.get_last_date SELECT Ticker, MAX(Date) AS Last_Date, COUNT(*) AS Num_of_records FROM dbo.market_price AS mp GROUP BY Ticker View dbo.get_fundamental_field_names SELECT COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE (TABLE_NAME = N'fundamentals') THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
About
Contains an example written in Visual Basic.Net of using the TD Ameritrade Web API or polygon.io API to update end-of-day stock quotes in a Microsoft SQL Server database
Topics
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published