Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scripts for pulling one-time data from partner data sources needed in the Admin Dashboard UI #513

Closed
burrage opened this issue Feb 5, 2021 · 2 comments

Comments

@burrage
Copy link
Contributor

burrage commented Feb 5, 2021

User Story
As a developer,
I want to write scripts that pull one-time information from various partner data sources and display them when needed in the UI,
So that I can populate the Tech Health and Deep Dive Tech sections of the Admin Dashboard with data from APIs of various services that help to run FTL.

Acceptance Criteria
Given that I have completed the UI wireframes of the Tech Health and Deep Dive Tech sections and I have identified which data can be pulled one-time on-demand vs needing to be stored over time,
When I view the Tech Health and Deep Dive Tech sections of the Admin Dashboard,
Then I should see values returned from various API services displayed in the UI.

@eric-gt
Copy link
Contributor

eric-gt commented Feb 8, 2021

Notes on Integrations:

  • Postman allows for running monitors programmatically and receiving the data
  • Stripe appears to restrict access to logs to either the dashboard or by using the Stripe CLI. It's possible to use a webhook to write data on events to a file on EC2 and then read from the file
  • Google Cloud Functions provide a logging API to read and retrieve logs, and an Error Logging API for fine grained error reporting.

@eric-gt eric-gt self-assigned this Feb 16, 2021
@eric-gt
Copy link
Contributor

eric-gt commented Feb 18, 2021

Unfortunate update on Google Cloud integration with the current project infrastructure. Google Cloud does not appear to be support being bundled through webpack: googleapis/google-cloud-node#1821
It looks like there's ways around it, such as installing Babel and using it as an external bundler specifically for google cloud libraries. But it's also breaking our unit testing library somehow

C:\Users\Eric\Documents\bin\followthelearners\node_modules\protobufjs\src\root.js:1                                      
Error: not supported                                                                                                     
    at Root.loadSync (C:\Users\Eric\Documents\bin\followthelearners\node_modules\protobufjs\src\root.js:236:15)          
    at Object.loadSync (C:\Users\Eric\Documents\bin\followthelearners\node_modules\protobufjs\src\index-light.js:69:17)  
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\node_modules\@grpc\proto-loader\build\src\index.
js:230:37)                                                                                                               
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                            
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                            
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\node_modules\@google-cloud\logging\node_modules\
google-gax\src\grpc.ts:17:1)                                                                                             
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                            
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                            
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\node_modules\@google-cloud\logging\node_modules\
google-gax\src\index.ts:18:1)                                                                                            
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                            
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                            
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\node_modules\@google-cloud\logging\src\index.ts:
23:1)                                                                                                                    
    at Generator.next (<anonymous>)                                                                                      
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                            
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\static\js\admin\cloudLogReader.ts:1)            
    at Generator.next (<anonymous>)                                                                                      
    at Object.m._compile (C:\Users\Eric\Documents\bin\followthelearners\node_modules\ts-node\src\index.ts:1056:23)       
    at Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                                   
    at Object.require.extensions.<computed> [as .ts] (C:\Users\Eric\Documents\bin\followthelearners\node_modules\ts-node\
src\index.ts:1059:12)                                                                                                    
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\static\js\admin\adminStoplightChart.ts:1)       
    at Generator.next (<anonymous>)                                                                                      
    at Object.require.extensions.<computed> [as .ts] (C:\Users\Eric\Documents\bin\followthelearners\node_modules\ts-node\
src\index.ts:1059:12)                                                                                                    
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\static\js\admin\admin.ts:1)                     
    at Generator.next (<anonymous>)                                                                                      
    at Object.require.extensions.<computed> [as .ts] (C:\Users\Eric\Documents\bin\followthelearners\node_modules\ts-node\
src\index.ts:1059:12)                                                                                                    
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\test\static\js\admin\admin.spec.ts:1)           
    at Generator.next (<anonymous>)                                                                                      
    at Object.require.extensions.<computed> [as .ts] (C:\Users\Eric\Documents\bin\followthelearners\node_modules\ts-node\
src\index.ts:1059:12)                                                                                                    
    at Object.exports.requireOrImport (C:\Users\Eric\Documents\bin\followthelearners\node_modules\mocha\lib\esm-utils.js:
42:12)                                                                                                                   
    at Object.exports.loadFilesAsync (C:\Users\Eric\Documents\bin\followthelearners\node_modules\mocha\lib\esm-utils.js:5
5:34)                                                                                                                    
    at Mocha.loadFilesAsync (C:\Users\Eric\Documents\bin\followthelearners\node_modules\mocha\lib\mocha.js:431:19)       
    at singleRun (C:\Users\Eric\Documents\bin\followthelearners\node_modules\mocha\lib\cli\run-helpers.js:125:15)        
    at exports.runMocha (C:\Users\Eric\Documents\bin\followthelearners\node_modules\mocha\lib\cli\run-helpers.js:190:10) 
    at Object.exports.handler (C:\Users\Eric\Documents\bin\followthelearners\node_modules\mocha\lib\cli\run.js:362:11)   
    at C:\Users\Eric\Documents\bin\followthelearners\node_modules\yargs\build\index.cjs:443:71    at process.runNextTicks
 [as _tickCallback] (internal/process/task_queues.js:62:5)                                                               
    at internal/main/run_main_module.js:17:47                                                                            

It should be noted that Google exposes the RESTful API underlying its client libraries, so if we can figure out how to format our GET request to pass the necessary authentication parameters we can keep the current implementation that uses the fetch API, but Google really wants you to use client libraries and I haven't found any documentation on how to pass authentication in a simple request.
I'm leaving this for now and including a link to the cloud functions builtin monitoring UI in the deep dive task. Capturing this though for when we have some cycles to actually track down what's happening

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants