Log out a detailed error msg while inserting data into big query #229

snarayanan1095 · 2021-05-13T20:24:21Z

While inserting data into big query, it sends a response back if there's an error. This can be extremely helpful while debugging

CLAassistant · 2021-05-13T20:24:26Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Shwetha Narayanan seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

packages/gcp/src/BigQueryClient.ts

* parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188453 -0500 parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188394 -0500 initial commit develop * Bump up the versions for google cloud NPM packages (#11) * Integration command exits with jest exit code (#40) * Integration command exits with jest exit code [full ci] * revert version bumps [full ci] * Fix failing Kafka integration test (#43) * Remove Deprecated new Buffer usage (#46) * #12 remove deprecated new Buffer usage * Support Changes in Metrics Tags for Prometheus (#38) * Support Changes in Metrics Tags for Prometheus * add integration test to .travis.yml [full ci] * set jestTimeout to a larger value [full ci] * testing change to integrate always returning 0 * [full ci] * increasing time to wait for Prometheus to scrape * [full ci] * increase wait to 120 sec [full ci] * adding localhost so Docker/Prom works for linux * [full ci] * look for ports starting with 300 [full ci] * show all results from netstat [full ci] * File service discovery for Prometheus targets [full ci] * moving targets.json generation to integration file * [full ci] * [full ci] * slimming down docker-compose.yml file * [full ci] * addressing comments * [full ci] * Clean up for review * Fix IMetricsTags to ILabelValues conversion * Addressing comments * kubernetes: adjust logic for creating a watch (#48) kubernetes: adjust logic for creating a watch * update kubernetes-client to latest version on master that contains various internal fixes for creation and handling of a watch * adjust logic for startWatch to wrap watch function inside of a promise and handle timeouts differently * change callback for watch to not be async azure: fix build error to take in correct BufferEncoding and update kb calculations * Revert commit of incorrect yarn.lock from #48 (#50) yarn.lock was committed with private registry information. rebuild yarn.lock in order to fix deployment issues from #48. * kubernetes: fix kubernetes-client package build issue with Node 8 (#51) kubernetes-client package requires use of node version greater than 8 but we require 8 in the travis ci build pipeline. as a temporary solution till we deprecate Node 8 completely as part of #52 update the build pipeline to use a separate package.v8.json which doesn't build it. * build: fix grep check to only match if we're using node 8 (#53) * kubernetes: update @kubernetes/client-node package to 0.11.1 (#54) * kubernetes: add in additional logging details for watch errors (#56) * improve timeout of k8s watch (#59) * Allow opt-in to creating Azure queues dynamically on write (#55) * core: update ConsoleLogger to correctly log out nested objects (#60) core: update ConsoleLogger to correctly log out nested objects The console logger renders out `[object Object]` if a field is an object so destructure it into `.` delimited nested fields (e.g. foo.bar.baz) * Update ConsoleLogger to take in options letting users select a maxDepth for log outputs * kubernetes: create k8sPollSource that periodically polls for resources instead of using a watch (#61) * Create `KubernetesBase` class that both poll and watch sources extend from * Add in tests for `KubernetesPollSource` * Add preprocessor concept for Azure Queues to support messages not following the envelope format (#65) * fix missing bind when creating new azure queues (#63) * fix lint errors (#67) * refactor queue client to avoid exporting type from azure library as p… (#68) * refactor queue client to avoid exporting type from azure library as part of public api * kubernetes: add in timeout to poll request in case request to k8s hangs (#70) * Fix bug when options are ignored in QueueClient (#69) * Decorator returning class with empty string name (#73) * Decorator returning class with empty string name [full cli] * remove export, add link to TS issue [full cli] * bump version [full ci] * core: do not overwrite pending promises for the backlog of queued items (#76) It's possible to get into a state where we've hit our limit on items that we can add to the bounded priority queue and a large backlog of items are accumulated. Once we can begin to process that backlog after a `whenNotFull` promise resolves, the first item in the backlog will create a new `whenNotFull` and proceed to await it. Subsequent items in backlog queue will do the same but overwrite the promise of the first item leading to a chain of promises for backlogged items that are unresolvable. This PR ensures that we never overwrite the `whenNotFull` promise for backlogged queue items and we can resolve them all eventually. * core: version bump to 1.2.0-beta.11 (#77) * Update Cookie Cutter Dependencies (#75) * better throughput in RPC mode while guaranteeing correctness of state (#83) * Update dependencies for Node 8 (#86) * Update dependencies for Node 8 * add white space [full ci] * remove white space [full ci] * A few more deps updates [full ci] Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Reveal Azure Blob & Queue Service URL params, to allow for pointing at a local emulator. (#89) * Way to Inspect 'Invalid' Messages (#82) * Way to Inspect 'Invalid' Messages * unit test * remove annotator from unit test * allow publishing from inside the invalid handler * Updating docs * impoving clarity of doc entry * actual change of doc * add failSpan if input validation fails for Serial * Use custom error to signal no invalid msg handler * Missed files * Do not propagate NoInvalidHandlerError * Remove custom error and add hasInvalid function * add case in unit test * more tests for ConventionBasedMessageDispatcher * refactoring to simplify code * addressing comments Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Better Log Message when a Message Fails to Process and fix to upsertSproc SeqConErr details (#91) * Better Log Message when a Message Fails to Process * upsertSproc SeqConErr details fix * use context's logger Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Update CHANGELOG.md (#98) * bump develop to 1.3 (#106) * Add Dead Letter Queue to QueueInputSource (#93) * Add Dead Letter Queue to QueueInputSource * Addressing comments * pass in a modified config to dead letter queue * addressing comment * changing API to expect values in milliseconds * Updating docs * Docs update with example dead letter queue config * update package.jsoon * doc nit Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * merge release/1.2 into develop (#112) * upgrade dependencies due to vulnerabilities (#111) * Remove support for node 8 (#113) * core: unable to close bounded priority queue (#116) core: fix issue with bounded priority queue not correctly supporting closing of queues immediately after it was drained leading to potential Cannot read property 'resolve' of undefined errors due to whenNotFull being undefined. * ConcurrentMessageProcessor suppresses error details (#124) * ConcurrentMessageProcessor suppresses error details When message handling fails outside of the message handler the ConcurrentMessageHandler currently throws a generic Error that hides the underlying root cause error. This PR changes it to re-throw the original error, similar to what the SerialMessageProcessor is doing. * Fix breaking API change in Azure Queues (#122) * Fix breaking API change in Azure Queues (#121) * Adding source unit to Config's timespanOf function * Adding Azure Queue change and other fixes Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * DeadLetterQueue fixes * bump versions * rebase and bump version Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add RedisStreamSink & RedisStreamSource (#126) * Add GCP PubSub Sink (#125) * Bump Redis version to publish new package (#127) do version bump missing in PR #126 * prometheus module should not throw an error when incrementing by 0 (#130) merge back from master * Prevent config.parse() output from being used as input (#134) * multi cosmos collections (#81) (#117) * fix MsSqlSink throws wrong error (#140) * fix MsSqlSink throws wrong error * Update package.json * Update MssqlSink.ts * Add AMQP Sink + Source (#136) * Add AMQP Sink + Source * add the actual Sink/Source files * properly close the sink's connection * Adding initialization and disposal to source * Basic producer and consumer scripts. * replace AsyncPipe with BoundedPriorityQueue * Refactor to get correct produce/consume behavior * improve connection call and add port as optional * fix yaml.lock file * Adding integration test * add new line at end of yml file * Add Copyright text and set "--passWithNoTest" * add msg release listener [full ci] * add AMQP integration to travis build * fix .travis.yml [full ci] * addressing comments * Fix integration test/setup [full ci] * Adding tracing [full ci] * minor corrections Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add metrics to AMQP (#144) * Add metrics to AMQP * bump version * switch to this.channel * trigger [full ci] * Adding periodic metrics * Add metadata [full ci] * Lint and style fix [full ci] * Addressing comments Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add docs to AMQP package (#147) * Add docs to AMQP package * correcting module name * Addressing comments * Adding example files * minor nits * rename * Simplify config Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add cookie-cutter-jaeger (#151) * Add cookie-cutter-jaeger * remove interface * add lock * update per comments Co-authored-by: Marco Garcia <marco.garcia@jet.computer> * cookie-cutter-redis: add support for multiple streams (#155) * cc-redis: add support for multiple streams * fix test * update spanLogAndSetTags * update getPendingMessagesForConsumerGroup * update RedisStreamSink * rename test * add tests * add more tests * update per feedback * update xReadGroup * full ci Co-authored-by: Marco Garcia <marco.garcia@jet.computer> * cookie-cutter-redis: add metrics (#156) * cookie-cutter-redis: add metrics * full ci * update docs * update docs * full ci * remove array tag Co-authored-by: Marco Garcia <marco.garcia@jet.computer> * Have separate queue capacity per priority level (#160) * Have separate queue capacity per priority level * Fix memory leak likely caused by promise chaining * fixing floating promises * white space change [full ci] Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> Co-authored-by: Sebastian Klose <sebastian.klose@walmart.com> * Fix lz4 error by adding resolution (#161) * Fix lz4 error by adding resolution * yarn.lock file change Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add new Jaeger package to README (#158) * fix broken metrics, reclaim PEL messages less often (#163) * fix pending list not fully drained on startup, added password config (#164) * fix messages are acked on error (#168) * check for failed acks to redis (#169) * fix xReadGroup ignores all but first message from batch (#170) * Implement IEncodedMessageEmbedder for ProtoMessageEncoder (#174) * Implement IEncodedMessageEmbedder for ProtoMessageEncoder * version change * unit tests Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * cleanup redis stream implementation (#171) * make some redis options nullable so the default value can be overwrit… (#177) * troubleshoot redis stream issue (#179) * Allow negative values in Prometheus histogram (#181) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Prevent BoundedPriorityQueue from deprioritizing waiting enqueue calls (#180) * Prevent BoundedPriorityQueue from deprioritizing waiting enqueue calls * versioon bump * handle floating promises Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Fix backwards compatibility for ProtoMessageEncoder (#182) * Version bump for Proto change (#183) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Change LogLevel from Error -> Warn when retrieving Kafka watermarks (#184) * fix 'yarn audit' issues [full ci] (#187) * fix sec vuln in node-fetch (#189) * Add auth for amqp source and sink (#192) * Add auth for amqp source and sink * trigger [full ci] * do not overwrite default creds [full ci] * do not overwrite default creds [full ci] * do not overwrite default creds [full ci] * do not overwrite default creds [full ci] * add missing new line [full ci] * lint fix * trigger ci [full ci] * trigger ci Co-authored-by: prachi.tandon@jet.computer <prachi.tandon@jet.com> * Add AMQP package to README (#193) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add vhost support for amqp (#194) * Add vhost support for amqp * Add vhost support for amqp * Add vhost support for amqp [full ci] * Add vhost support for amqp * Add vhost support for amqp * Add vhost support for amqp * version bump Co-authored-by: prachi.tandon@jet.computer <prachi.tandon@jet.com> * kubernetes: adjust logging to be less verbose (#195) * detect when kafkajs is stuck with stale broker metadata (#186) * detect when kafkajs is stuck with stale broker metadata and terminate application * lint * Ensure RedisClient's "type" metric label is always a string (#196) * Ensure metrics type label is always a string * Bump cookie-cutter-redis version to 1.3.0-beta.13 * Fix wrong string function in Kafka (#197) * Fix wrong string function in Kafka * proper conversion of object to string Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * address vulnerability in node-forge package [full ci] (#200) * bump develop to 1.4 (#201) * Don't extract data field when encoder isEmbeddable (#203) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Azure module integration tests (#204) * updates blob client to use latest azure storage library * updates blob client to use latest azure storage library * [full ci] * [full ci] * adds resolution for lz4 to avoid 0.6.4 which causes build errors * updates yarn lockfile to use correct npm repo * fixes linting errors in tests * addresses pr comments * fixes https agent compatibility * fixes linting errors * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * updates queue client to use latest azure storage library * updates yarn lock file, removing references to incorrect registry * [full ci] * [full ci] * [full ci] * [full ci] * adds integration tests for azure module * updates yarn lockfile * adds files to setup emulators for local use * removes references to incorrect registry from yarn lockfile * fixes linting errors * rerun tests * [full ci] * [full ci] * [full ci] * adds azure integration tests to travis build * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * use windows batch file to run integration tests * [full ci] * [full ci] * [full ci] * adds missing keys for integration tests * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * Copy over file with different Timeout * triger build * full ci [full ci] * checking what's available to the script * [full ci] * copy modified psm1 in travis [full ci] * moving commands to after install [full ci] * adding a folder that was missing in the path * [full ci] * Make timeout 40 minutes [full ci] * Change Timeout for Stop-CosmosDbEmulator to 20 min [full ci] * [full ci] needs to be on the first line * and end of line... [full ci] * [full ci] * [full ci] * remove timeout value [full ci] * Explicitly setting new Timeout [full ci] * Verify modified file is the one running [full ci] * Matching change in run integration [full ci] * Move copy before import [full ci] * fixing variable mixup [full ci] * Tell travis to wait 20 min instead of 10 * [full ci] * tweak travis [full ci] * tweaking travis_await [full ci] * tweaking travis again [full ci] * moving travis_wait to install command [full ci] * split script in 2 lines [full ci] * up travis_wait to 60 minutes [full ci] * Testing with 2 hour limit [full ci] * Test code that worked in test repo * [full ci] * Fix path and add missing ennv variables * [full ci] * Add integration test call * [full ci] * Add yarn command [full ci] * Add yarn build and pwd to check location * [full ci] * Add a bunch of print outs * [full ci] * remove extra cd .. [full ci] * [full ci] * fix Get-ChildItem command [full ci] * build CC-core first [full ci] * MASTER_KEY -> COSMOS_SECRET_KEY * [full ci] * Add timing to script [full ci] * Fix cd command [full ci] * Add CI var to travis [full ci] * Use same file for local and CI builds [full ci] * Refactor [full ci] * [full ci] * [full ci] * [full ci] * remove comment from .travis [full ci] * Prepare for review [full ci] * updates queue client to use latest azure storage library updates yarn lock file, removing references to incorrect registry updates blob client to use latest azure storage library updates blob client to use latest azure storage library updates yarn lockfile to use correct npm repo fixes linting errors in tests addresses pr comments fixes https agent compatibility fixes linting errors adds integration tests for azure module updates yarn lockfile adds files to setup emulators for local use removes references to incorrect registry from yarn lockfile fixes linting errors adds azure integration tests to travis build [full ci] [full ci] [full ci] use windows batch file to run integration tests [full ci] [full ci] adds missing keys for integration tests [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] Copy over file with different Timeout triger build full ci [full ci] checking what's available to the script [full ci] copy modified psm1 in travis [full ci] moving commands to after install [full ci] adding a folder that was missing in the path [full ci] Make timeout 40 minutes [full ci] Change Timeout for Stop-CosmosDbEmulator to 20 min [full ci] [full ci] needs to be on the first line and end of line... [full ci] [full ci] [full ci] remove timeout value [full ci] Explicitly setting new Timeout [full ci] Verify modified file is the one running [full ci] Matching change in run integration [full ci] Move copy before import [full ci] fixing variable mixup [full ci] Tell travis to wait 20 min instead of 10 [full ci] tweak travis [full ci] tweaking travis_await [full ci] tweaking travis again [full ci] moving travis_wait to install command [full ci] split script in 2 lines [full ci] up travis_wait to 60 minutes [full ci] Testing with 2 hour limit [full ci] Test code that worked in test repo [full ci] Fix path and add missing ennv variables [full ci] Add integration test call [full ci] Add yarn command [full ci] Add yarn build and pwd to check location [full ci] Add a bunch of print outs [full ci] remove extra cd .. [full ci] [full ci] fix Get-ChildItem command [full ci] build CC-core first [full ci] MASTER_KEY -> COSMOS_SECRET_KEY [full ci] Add timing to script [full ci] Fix cd command [full ci] Add CI var to travis [full ci] Use same file for local and CI builds [full ci] Refactor [full ci] [full ci] [full ci] [full ci] remove comment from .travis [full ci] Prepare for review [full ci] * yarn.lock [full ci] * yarn integrate does own setup [full ci] * Integrate cleans up unless arg is passed [full ci] * Fully stop VM on default integration run * Update README Co-authored-by: Emma Lynch <emmalynch@users.noreply.github.com> Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * fix redis client skipping over messages when there is an invalid message in a batch (#209) * Fix: Parse string error message to metrics (#1) (#210) * [kafka] Expose compression settings for producers via kafkaSink (#215) * force application to restart on disconnect because reconnect doesn't seem to work properly (#217) * upgrading gcp dependencies (#218) * wrong error is being logged when redis ack fails (#221) * Add Azure BlobClient to public API (#214) * Log out a detailed error msg while inserting data into big query (#229) * Log out a detailed error msg for inserting data into big query * Add a comma * Default detailedErrorMsg to e, remove NullLogger * Spelling * Remove json stringify * remove json stringify * Remove {} * Add new lines * Prettier error Co-authored-by: Shwetha Narayanan <s0n027b@homeoffice.wal-mart.com> * Switch Yarn audit level to critical (#232) * Switch Yarn audit level to critical [full ci] * disable cron check for audit * Only return non-zero on critical * Only log critical * revert back to type = cron Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Properly connect/disconnect Kafkajs producer (#234) * Properly connect/disconnect Kafkajs producer * nit * [full ci] * white space * smoke build Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Updating version of mssql to recent (#235) * updating version of mssql to recent * bump package version * adding timeouts and tests * linter * adding streaming [full ci] and full ci test * rerun tests with [full ci] * adding defaults * updating to 5.1.5 [full ci] * updating types [full ci] * travis file build [full ci] * white space * remove white space to make new build * Add TLS support to Kafka package (#230) * Add tls support to kafka package * trigger [full ci] * Kafka integration test fix [full ci] * revert travis changes * clean up * Bump version again Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Fixed issue with wrong grpcClient timing metric's unit (#220) * Dispatch context store API metadata (#238) * Update packages to reduce vulnerabilities (#236) * Update packages to reduce vulnerabilities * blank change * [full ci] * Bump versions * Remove temp change to audit * Generate yarn.lock from scratch [full ci] * Add longer jest timeout [full ci] * Fix new vulnerability * Bump versions after rebase * Clean up and [full ci] * Trigger only AMQP and Kafka [full ci] * Add docker login [full ci] * Add all components of .travis.yml back [full ci] * Test running only for integrate [full ci] * try if stage = integration [full ci] * Try if stage [full ci] * Check OS [full ci] * Run as script: [full ci] * Bash if os [full ci] * remove script: [full ci] * echo attributes [full ci] * Docker login to each integration test [full ci] * bump core version Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Include `ttl` setting when storing to Azure cosmosDB (#240) * Handle url separately from connectionString (#233) * Handle url separetely from connectionString * [full ci] * Run just Azure integration [full ci] * Add optional connectionString * [full ci] * revert connection string value [full ci] * use proper connectionString value [full ci] * Make url optional and add default url building * [full ci] * Revert all changes [full ci] * tweak travis [full ci] * log config in constructor [full ci] * Add connectionString back to QueueClient [full ci] * Finalize [full ci] * cleanup * version bump * Add comments adding clarity * Make url backwards compatible [full ci] * bump version Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> Co-authored-by: Sebastian Klose <sebastian.klose@walmart.com> * Missing `metadata` param from `IDispatchContext` interface (#243) * Initialize github action workflow for develop (#246) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Run cron more frequently (#251) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Add full workflow to develop (#242) * Test github action workflows * Add more components to github workflow * Fix missing $ * Use specific yarn version * Add yarn set version * Just use yarn set * Ubuntu Node 12 only * Only Test Windows * Test with macOS only * Test all three * Split test in 3 separate jobs * Modify macOS to run set and install in one run * Add needs smoke * with github token * Move with: before run * Missed a with after a run * Move with: to a step with uses: * Switch repo-token to token * Try repo-token in node setup step * Add repo-token to yarn step in macOS test * speed up smoke job * env for macOS only * remove repo-token from with * Try with all 3 test jobs * Add Integration test step * Fix typo in array * Add all ubuntu integration tests * Add Azure test on Windows * remove test yml * Echo default env variables * Try commit message * Evaluate before assigning to env var * Wrong step env * Try with event * echo github_context * Evaluate github cintext and assign to env var * try to extract commits * Extract [0] commit * Add .href * Rename yml file * Add more CMT vars * Just one cmt var * INPUT_COMMITS * git log command * another version of git log * missing quotes * Just git log -1 * specify --git-dir * --first-parent * -2 on git log * git log only * git reflog * git reflog show * Event Before and After * Add quotes * Just after * checkout with head.sha * online * missing quote * before not working, switching to after * Move call to earlier part * Move another command * Set env var from commit and display many contexts * try cat * If condition * if in a step * Set env var * Add correct script * remove extra : * Missing } * Can the next job access the env var * Try with brackets * Add cron for 17:15 and start cleanup * Match GMT * Try the env and artifact for next job * Fix missing run * Artifact file is empty, try to fix * Still empty * Resolving the env first * missing run * Remove $() * this has full ci * Initia set up with full if * Let's try a full ci * Do not use contractions in commit message full ci * Switch away from using artifacts * Clean up and run all without the trigger * Add [] to full ci and minor corrections [full ci] * Debug why windows condition fails [full ci] * Typo * Windows specific env set * Try with set instead of setx [full ci] * Try another version * MIssing quote * Test env var for windows * TRy another thing * Try it with the git command * Don't specify shell as bash * echo COMMIT_MESSAGE in same run * Add quotes * More trying * Try different env var name * Test var 1 * Try var 2 * Try var 3 * Try var 4 * Try without full ci * And now with [full ci] * Make change to second windows job [full ci] * Fix needs * Return other jobs [full-ci] * Return all parts of all jobs * Add placeholders for publish steps and names * Clean up publish jobs and change cron * Minimize travis and run [full ci] * Run just azure integration [full ci] * Test better exit code [full ci] * Print exit code for integrate.sh [full ci] * Run decrypt/encrypt * Add on: workflow_dispatch * Rename Workflow * Add details to workflow_dispatch * Rename workflow as Develop * Rename workflow file * Add full workflow to develop * Revert commented travis * white space * [full-ci] * Try a custom action in same repo * Try with regular checkout first * Revert back to regular conditional checkout * remove duplicate yarn install from macOs test * [full ci] * Uncomment jobs * remove commented out line * Add if on jobs * Trying job outputs * Try with a simple assignment * fix job name * Try env.COMMIT_MESSAGE again * Missing $ * Echo just env var * Try with ENV: * With correct sha * proper sha on checkout * Try if on previous job output [small ci] * Missing ' * Run without full ci * white space * [full ci] * Clean up and run a full [full ci] * Try exit_code without quotes [full ci] * Actual change [full ci] * rebase [full ci] * uncomment * Single checkout and split build/test [full ci] * rebase * [full ci] * rebase * cron and [full ci] * white space * [full ci] * Add exit_code to PowerShell script [full ci] * Clean up non gh action changes and [full ci] * Fix needs and [full ci] * Line [full ci] * Don't get commit message on push * white space * Trigger build with very long commit message that takes more than one line * Convert into oneline commit message. This is also really long * Remove conditions for commit message * Add yarn install steps to npm publish and pages * Check github.ref on publish pages * white space * one last [full ci] Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Azure Integration not failing on failed tests (#252) * Azure Integration not failing on failed tests * Set $ErrorActionPreference = "Stop" [full ci] * Throw error from ps scrippt [full ci] * Simplify [full ci] * Last Exit Code [full ci] * Try [full ci] * Simplify [full ci] * Integration test fix [full ci] * Exit code from PS is boolean [full ci] * Put the rest of the workflow back in * Missing empty line * Another missing empty line Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Merge master into develop (#267) * Update .travis.yml * Decorator returning class with empty string name (hotfix 1.1) (#74) * Decorator returning class with empty string name [full cli] * bump version [full ci] * build change to address issue with yarn 1.22.0 * [full ci] * core: support multiple concurrent writers for BoundedPriorityQueue (#78) It's possible to get into a state where we've hit our limit on items that we can add to the bounded priority queue and a large backlog of items are accumulated. Once we can begin to process that backlog after a `whenNotFull` promise resolves, the first item in the backlog will create a new `whenNotFull` and proceed to await it. Subsequent items in backlog queue will do the same but overwrite the promise of the first item leading to a chain of promises for backlogged items that are unresolvable. This PR ensures that we never overwrite the `whenNotFull` promise for backlogged queue items and we can resolve them all eventually. * core: unable to close bounded priority queue (#114) core: fix issue with bounded priority queue not correctly supporting closing of queues immediately after it was drained leading to potential `Cannot read property 'resolve' of undefined` errors due to whenNotFull being undefined. * Cookie Cutter 1.2 (#119) * parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188453 -0500 parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188394 -0500 initial commit develop * Bump up the versions for google cloud NPM packages (#11) * Integration command exits with jest exit code (#40) * Integration command exits with jest exit code [full ci] * revert version bumps [full ci] * Fix failing Kafka integration test (#43) * Remove Deprecated new Buffer usage (#46) * #12 remove deprecated new Buffer usage * Support Changes in Metrics Tags for Prometheus (#38) * Support Changes in Metrics Tags for Prometheus * add integration test to .travis.yml [full ci] * set jestTimeout to a larger value [full ci] * testing change to integrate always returning 0 * [full ci] * increasing time to wait for Prometheus to scrape * [full ci] * increase wait to 120 sec [full ci] * adding localhost so Docker/Prom works for linux * [full ci] * look for ports starting with 300 [full ci] * show all results from netstat [full ci] * File service discovery for Prometheus targets [full ci] * moving targets.json generation to integration file * [full ci] * [full ci] * slimming down docker-compose.yml file * [full ci] * addressing comments * [full ci] * Clean up for review * Fix IMetricsTags to ILabelValues conversion * Addressing comments * kubernetes: adjust logic for creating a watch (#48) kubernetes: adjust logic for creating a watch * update kubernetes-client to latest version on master that contains various internal fixes for creation and handling of a watch * adjust logic for startWatch to wrap watch function inside of a promise and handle timeouts differently * change callback for watch to not be async azure: fix build error to take in correct BufferEncoding and update kb calculations * Revert commit of incorrect yarn.lock from #48 (#50) yarn.lock was committed with private registry information. rebuild yarn.lock in order to fix deployment issues from #48. * kubernetes: fix kubernetes-client package build issue with Node 8 (#51) kubernetes-client package requires use of node version greater than 8 but we require 8 in the travis ci build pipeline. as a temporary solution till we deprecate Node 8 completely as part of #52 update the build pipeline to use a separate package.v8.json which doesn't build it. * build: fix grep check to only match if we're using node 8 (#53) * kubernetes: update @kubernetes/client-node package to 0.11.1 (#54) * kubernetes: add in additional logging details for watch errors (#56) * improve timeout of k8s watch (#59) * Allow opt-in to creating Azure queues dynamically on write (#55) * core: update ConsoleLogger to correctly log out nested objects (#60) core: update ConsoleLogger to correctly log out nested objects The console logger renders out `[object Object]` if a field is an object so destructure it into `.` delimited nested fields (e.g. foo.bar.baz) * Update ConsoleLogger to take in options letting users select a maxDepth for log outputs * kubernetes: create k8sPollSource that periodically polls for resources instead of using a watch (#61) * Create `KubernetesBase` class that both poll and watch sources extend from * Add in tests for `KubernetesPollSource` * Add preprocessor concept for Azure Queues to support messages not following the envelope format (#65) * fix missing bind when creating new azure queues (#63) * fix lint errors (#67) * refactor queue client to avoid exporting type from azure library as p… (#68) * refactor queue client to avoid exporting type from azure library as part of public api * kubernetes: add in timeout to poll request in case request to k8s hangs (#70) * Fix bug when options are ignored in QueueClient (#69) * Decorator returning class with empty string name (#73) * Decorator returning class with empty string name [full cli] * remove export, add link to TS issue [full cli] * bump version [full ci] * core: do not overwrite pending promises for the backlog of queued items (#76) It's possible to get into a state where we've hit our limit on items that we can add to the bounded priority queue and a large backlog of items are accumulated. Once we can begin to process that backlog after a `whenNotFull` promise resolves, the first item in the backlog will create a new `whenNotFull` and proceed to await it. Subsequent items in backlog queue will do the same but overwrite the promise of the first item leading to a chain of promises for backlogged items that are unresolvable. This PR ensures that we never overwrite the `whenNotFull` promise for backlogged queue items and we can resolve them all eventually. * core: version bump to 1.2.0-beta.11 (#77) * Update Cookie Cutter Dependencies (#75) * better throughput in RPC mode while guaranteeing correctness of state (#83) * Update dependencies for Node 8 (#86) * Update dependencies for Node 8 * add white space [full ci] * remove white space [full ci] * A few more deps updates [full ci] Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Reveal Azure Blob & Queue Service URL params, to allow for pointing at a local emulator. (#89) * Way to Inspect 'Invalid' Messages (#82) * Way to Inspect 'Invalid' Messages * unit test * remove annotator from unit test * allow publishing from inside the invalid handler * Updating docs * impoving clarity of doc entry * actual change of doc * add failSpan if input validation fails for Serial * Use custom error to signal no invalid msg handler * Missed files * Do not propagate NoInvalidHandlerError * Remove custom error and add hasInvalid function * add case in unit test * more tests for ConventionBasedMessageDispatcher * refactoring to simplify code * addressing comments Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Better Log Message when a Message Fails to Process and fix to upsertSproc SeqConErr details (#91) * Better Log Message when a Message Fails to Process * upsertSproc SeqConErr details fix * use context's logger Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Update CHANGELOG.md (#98) * prepare 1.2-rc.1 (#105) * upgrade dependencies due to vulnerabilities (#111) * bump version * update changelog / fix link Co-authored-by: Kshitiz Gupta <kgupta40@gatech.edu> Co-authored-by: plameniv <47831130+plameniv@users.noreply.github.com> Co-authored-by: Connor Ross <cross311@users.noreply.github.com> Co-authored-by: Tanvir Alam <talam@users.noreply.github.com> Co-authored-by: Chris Pinola <trogdor3222@gmail.com> Co-authored-by: Ilya Butorine <i@ilya.com> Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> Co-authored-by: Sean Halpin <softdev87@users.noreply.github.com> * Fix breaking API change in Azure Queues (#121) * Adding source unit to Config's timespanOf function * Adding Azure Queue change and other fixes Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * prometheus module should not throw an error when incrementing by 0 (#128) * Have separate queue capacity per priority level (#162) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Stabilize 1.3 (#202) * parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188453 -0500 parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188394 -0500 initial commit develop * Bump up the versions for google cloud NPM packages (#11) * Integration command exits with jest exit code (#40) * Integration command exits with jest exit code [full ci] * revert version bumps [full ci] * Fix failing Kafka integration test (#43) * Remove Deprecated new Buffer usage (#46) * #12 remove deprecated new Buffer usage * Support Changes in Metrics Tags for Prometheus (#38) * Support Changes in Metrics Tags for Prometheus * add integration test to .travis.yml [full ci] * set jestTimeout to a larger value [full ci] * testing change to integrate always returning 0 * [full ci] * increasing time to wait for Prometheus to scrape * [full ci] * increase wait to 120 sec [full ci] * adding localhost so Docker/Prom works for linux * [full ci] * look for ports starting with 300 [full ci] * show all results from netstat [full ci] * File service discovery for Prometheus targets [full ci] * moving targets.json generation to integration file * [full ci] * [full ci] * slimming down docker-compose.yml file * [full ci] * addressing comments * [full ci] * Clean up for review * Fix IMetricsTags to ILabelValues conversion * Addressing comments * kubernetes: adjust logic for creating a watch (#48) kubernetes: adjust logic for creating a watch * update kubernetes-client to latest version on master that contains various internal fixes for creation and handling of a watch * adjust logic for startWatch to wrap watch function inside of a promise and handle timeouts differently * change callback for watch to not be async azure: fix build error to take in correct BufferEncoding and update kb calculations * Revert commit of incorrect yarn.lock from #48 (#50) yarn.lock was committed with private registry information. rebuild yarn.lock in order to fix deployment issues from #48. * kubernetes: fix kubernetes-client package build issue with Node 8 (#51) kubernetes-client package requires use of node version greater than 8 but we require 8 in the travis ci build pipeline. as a temporary solution till we deprecate Node 8 completely as part of #52 update the build pipeline to use a separate package.v8.json which doesn't build it. * build: fix grep check to only match if we're using node 8 (#53) * kubernetes: update @kubernetes/client-node package to 0.11.1 (#54) * kubernetes: add in additional logging details for watch errors (#56) * improve timeout of k8s watch (#59) * Allow opt-in to creating Azure queues dynamically on write (#55) * core: update ConsoleLogger to correctly log out nested objects (#60) core: update ConsoleLogger to correctly log out nested objects The console logger renders out `[object Object]` if a field is an object so destructure it into `.` delimited nested fields (e.g. foo.bar.baz) * Update ConsoleLogger to take in options letting users select a maxDepth for log outputs * kubernetes: create k8sPollSource that periodically polls for resources instead of using a watch (#61) * Create `KubernetesBase` class that both poll and watch sources extend from * Add in tests for `KubernetesPollSource` * Add preprocessor concept for Azure Queues to support messages not following the envelope format (#65) * fix missing bind when creating new azure queues (#63) * fix lint errors (#67) * refactor queue client to avoid exporting type from azure library as p… (#68) * refactor queue client to avoid exporting type from azure library as part of public api * kubernetes: add in timeout to poll request in case request to k8s hangs (#70) * Fix bug when options are ignored in QueueClient (#69) * Decorator returning class with empty string name (#73) * Decorator returning class with empty string name [full cli] * remove export, add link to TS issue [full cli] * bump version [full ci] * core: do not overwrite pending promises for the backlog of queued items (#76) It's possible to get into a state where we've hit our limit on items that we can add to the bounded priority queue and a large backlog of items are accumulated. Once we can begin to process that backlog after a `whenNotFull` promise resolves, the first item in the backlog will create a new `whenNotFull` and proceed to await it. Subsequent items in backlog queue will do the same but overwrite the promise of the first item leading to a chain of promises for backlogged items that are unresolvable. This PR ensures that we never overwrite the `whenNotFull` promise for backlogged queue items and we can resolve them all eventually. * core: version bump to 1.2.0-beta.11 (#77) * Update Cookie Cutter Dependencies (#75) * better throughput in RPC mode while guaranteeing correctness of state (#83) * Update dependencies for Node 8 (#86) * Update dependencies for Node 8 * add white space [full ci] * remove white space [full ci] * A few more deps updates [full ci] Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Reveal Azure Blob & Queue Service URL params, to allow for pointing at a local emulator. (#89) * Way to Inspect 'Invalid' Messages (#82) * Way to Inspect 'Invalid' Messages * unit test * remove annotator from unit test * allow publishing from inside the invalid handler * Updating docs * impoving clarity of doc entry * actual change of doc * add failSpan if input validation fails for Serial * Use custom error to signal no invalid msg handler * Missed files * Do not propagate NoInvalidHandlerError * Remove custom error and add hasInvalid function * add case in unit test * more tests for ConventionBasedMessageDispatcher * refactoring to simplify code * addressing comments Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Better Log Message when a Message Fails to Process and fix to upsertSproc SeqConErr details (#91) * Better Log Message when a Message Fails to Process * upsertSproc SeqConErr details fix * use context's logger Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Update CHANGELOG.md (#98) * bump develop to 1.3 (#106) * Add Dead Letter Queue to QueueInputSource (#93) * Add Dead Letter Queue to QueueInputSource * Addressing comments * pass in a modified config to dead letter queue * addressing comment * changing API to expect values in milliseconds * Updating docs * Docs update with example dead letter queue config * update package.jsoon * doc nit Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * merge release/1.2 into develop (#112) * upgrade dependencies due to vulnerabilities (#111) * Remove support for node 8 (#113) * core: unable to close bounded priority queue (#116) core: fix issue with bounded priority queue not correctly supporting closing of queues immediately after it was drained leading to potential Cannot read property 'resolve' of undefined errors due to whenNotFull being undefined. * ConcurrentMessageProcessor suppresses error details (#124) * ConcurrentMessageProcessor suppresses error details When message handling fails outside of the message handler the ConcurrentMessageHandler currently throws a generic Error that hides the underlying root cause error. This PR changes it to re-throw the original error, similar to what the SerialMessageProcessor is doing. * Fix breaking API change in Azure Queues (#122) * Fix breaking API change in Azure Queues (#121) * Adding source unit to Config's timespanOf function * Adding Azure Queue change and other fixes Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * DeadLetterQueue fixes * bump versions * rebase and bump version Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add RedisStreamSink & RedisStreamSource (#126) * Add GCP PubSub Sink (#125) * Bump Redis version to publish new package (#127) do version bump missing in PR #126 * prometheus module should not throw an error when incrementing by 0 (#130) merge back from master * Prevent config.parse() output from being used as input (#134) * multi cosmos collections (#81) (#117) * fix MsSqlSink throws wrong error (#140) * fix MsSqlSink throws wrong error * Update package.json * Update MssqlSink.ts * Add AMQP Sink + Source (#136) * Add AMQP Sink + Source * add the actual Sink/Source files * properly close the sink's connection * Adding initialization and disposal to source * Basic producer and consumer scripts. * replace AsyncPipe with BoundedPriorityQueue * Refactor to get correct produce/consume behavior * improve connection call and add port as optional * fix yaml.lock file * Adding integration test * add new line at end of yml file * Add Copyright text and set "--passWithNoTest" * add msg release listener [full ci] * add AMQP integration to travis build * fix .travis.yml [full ci] * addressing comments * Fix integration test/setup [full ci] * Adding tracing [full ci] * minor corrections Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add metrics to AMQP (#144) * Add metrics to AMQP * bump version * switch to this.channel * trigger [full ci] * Adding periodic metrics * Add metadata [full ci] * Lint and style fix [full ci] * Addressing comments Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add docs to AMQP package (#147) * Add docs to AMQP package * correcting module name * Addressing comments * Adding example files * minor nits * rename * Simplify config Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add cookie-cutter-jaeger (#151) * Add cookie-cutter-jaeger * remove interface * add lock * update per comments Co-authored-by: Marco Garcia <marco.garcia@jet.computer> * cookie-cutter-redis: add support for multiple streams (#155) * cc-redis: add support for multiple streams * fix test * update spanLogAndSetTags * update getPendingMessagesForConsumerGroup * update RedisStreamSink * rename test * add tests * add more tests * update per feedback * update xReadGroup * full ci Co-authored-by: Marco Garcia <marco.garcia@jet.computer> * cookie-cutter-redis: add metrics (#156) * cookie-cutter-redis: add metrics * full ci * update docs * update docs * full ci * remove array tag Co-authored-by: Marco Garcia <marco.garcia@jet.computer> * Have separate queue capacity per priority level (#160) * Have separate queue capacity per priority level * Fix memory leak likely caused by promise chaining * fixing floating promises * white space change [full ci] Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> Co-authored-by: Sebastian Klose <sebastian.klose@walmart.com> * Fix lz4 error by adding resolution (#161) * Fix lz4 error by adding resolution * yarn.lock file change Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add new Jaeger package to README (#158) * fix broken metrics, reclaim PEL messages less often (#163) * fix pending list not fully drained on startup, added password config (#164) * fix messages are acked on error (#168) * check for failed acks to redis (#169) * fix xReadGroup ignores all but first message from batch (#170) * Implement IEncodedMessageEmbedder for ProtoMessageEncoder (#174) * Implement IEncodedMessageEmbedder for ProtoMessageEncoder * version change * unit tests Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * cleanup redis stream implementation (#171) * make some redis options nullable so the default value can be overwrit… (#177) * troubleshoot redis stream issue (#179) * Allow negative values in Prometheus histogram (#181) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Prevent BoundedPriorityQueue from deprioritizing waiting enqueue calls (#180) * Prevent BoundedPriorityQueue from deprioritizing waiting enqueue calls * versioon bump * handle floating promises Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Fix backwards compatibility for ProtoMessageEncoder (#182) * Version bump for Proto change (#183) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Change LogLevel from Error -> Warn when retrieving Kafka watermarks (#184) * fix 'yarn audit' issues [full ci] (#187) * fix sec vuln in node-fetch (#189) * Add auth for amqp source and sink (#192) * Add auth for amqp source and sink * trigger [full ci] * do not overwrite default creds [full ci] * do not overwrite default creds [full ci] * do not overwrite default creds [full ci] * do not overwrite default creds [full ci] * add missing new line [full ci] * lint fix * trigger ci [full ci] * trigger ci Co-authored-by: prachi.tandon@jet.computer <prachi.tandon@jet.com> * Add AMQP package to README (#193) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add vhost support for amqp (#194) * Add vhost support for amqp * Add vhost support for amqp * Add vhost support for amqp [full ci] * Add vhost support for amqp * Add vhost support for amqp * Add vhost support for amqp * version bump Co-authored-by: prachi.tandon@jet.computer <prachi.tandon@jet.com> * kubernetes: adjust logging to be less verbose (#195) * detect when kafkajs is stuck with stale broker metadata (#186) * detect when kafkajs is stuck with stale broker metadata and terminate application * lint * Ensure RedisClient's "type" metric label is always a string (#196) * Ensure metrics type label is always a string * Bump cookie-cutter-redis version to 1.3.0-beta.13 * Fix wrong string function in Kafka (#197) * Fix wrong string function in Kafka * proper conversion of object to string Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * address vulnerability in node-forge package [full ci] (#200) * create 1.3-rc [full ci] * bump version, add missing license headers * fix code dupe * update changelog Co-authored-by: Kshitiz Gupta <kgupta40@gatech.edu> Co-authored-by: plameniv <47831130+plameniv@users.noreply.github.com> Co-authored-by: Connor Ross <cross311@users.noreply.github.com> Co-authored-by: Tanvir Alam <talam@users.noreply.github.com> Co-authored-by: Chris Pinola <trogdor3222@gmail.com> Co-authored-by: Ilya Butorine <i@ilya.com> Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> Co-authored-by: Sean Halpin <softdev87@users.noreply.github.com> Co-authored-by: Emma Lynch <emmalynch@users.noreply.github.com> Co-authored-by: Dillon Mulroy <dillon.mulroy@gmail.com> Co-authored-by: Kshitiz Gupta <kshitiz.gupta@walmart.com> Co-authored-by: Chris Pinola <chris@pinola.co> Co-authored-by: Marco Garcia <mag415@gmail.com> Co-authored-by: Marco Garcia <marco.garcia@jet.computer> Co-authored-by: prachi30 <51481988+prachi30@users.noreply.github.com> Co-authored-by: prachi.tandon@jet.computer <prachi.tandon@jet.com> * Don't extract data field in QueueInputSource when encoder isEmbeddable (#205) * Don't extract data field in QueueInputSource when encoder isEmbeddable * remove yarn.lock changes * trigger [full ci] * clean up Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Initialize github action workflow for master (#244) * Initialize github action workflow for master * Rename workflow * Cron at minute 40 of each hour * Clean up * Add develop branch as well Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Run with different branch on schedule (#248) * Run with different branch on schedule * More cron times * Add inputs for workflow_dispatch Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Make checkout jobs mutually exclusive, add labels (#249) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Add full workflow to master (#250) * Add full workflow to master * Run all jobs on master without full ci keyword * Add longer timeout in core test * Use only 1 checkout step [ful ci] * Shorted build for integrate jobs [full ci] * Cleanup and rename workflow file [full ci] * Run [full ci] without needing macOs test job * Return macOS to integration needs Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Fix issue with commit message (#255) * Fix issue with commit message * Try without commit message * Add : after if * Add missing $ * Remove setting of commit message * Add if conditions on commit message set * Revert temp changes * Use oneline git log format instead of adding an if check on the commit message steps * Add missing space in git log command. Make this multiline again. Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Core application test increase timeout (#253) * Core application test increase timeout * Don't bump version * Add yarn install steps to npm publish and pages Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Check github.ref on push for publishing pages (#256) * Check github.ref on push for publishing pages * Switch to contains check and echo contents * remove echo of ref * clean up Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Properly connect/disconnect Kafkajs producer mstr (#259) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Remove extra if check from build step in master (#262) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Convert Redis err to str then pass to Prometheus master (#258) * Convert Redis err to str then pass to Prometheus master * Switch to only using the error name Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Remove .travis.yml and update badges (#265) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Clean up extra files * Revert to develop yarn.lock * Missing return * Run [full ci] Co-authored-by: Sebastian Klose <sebastian.klose@walmart.com> Co-authored-by: Tanvir Alam <talam@users.noreply.github.com> Co-authored-by: Kshitiz Gupta <kgupta40@gatech.edu> Co-authored-by: Connor Ross <cross311@users.noreply.github.com> Co-authored-by: Chris Pinola <trogdor3222@gmail.com> Co-authored-by: Ilya Butorine <i@ilya.com> Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> Co-authored-by: Sean Halpin <softdev87@users.noreply.github.com> Co-authored-by: Emma Lynch <emmalynch@users.noreply.github.com> Co-authored-by: Dillon Mulroy <dillon.mulroy@gmail.com> Co-authored-by: Kshitiz Gupta <kshitiz.gupta@walmart.com> Co-authored-by: Chris Pinola <chris@pinola.co> Co-authored-by: Marco Garcia <mag415@gmail.com> Co-authored-by: Marco Garcia <marco.garcia@jet.computer> Co-authored-by: prachi30 <51481988+prachi30@users.noreply.github.com> Co-authored-by: prachi.tandon@jet.computer <prachi.tandon@jet.com> Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Disable error on disconnect on proper shutdown of Redis Proxy (#270) * Disable error on disconnect on proper shutdown * bump version [full ci] Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Convert Redis Error to String then pass to Prometheus Develop Continued (#274) * Update .travis.yml * Decorator returning class with empty string name (hotfix 1.1) (#74) * Decorator returning class with empty string name [full cli] * bump version [full ci] * build change to address issue with yarn 1.22.0 * [full ci] * core: support multiple concurrent writers for BoundedPriorityQueue (#78) It's possible to get into a state where we've hit our limit on items that we can add to the bounded priority queue and a large backlog of items are accumulated. Once we can begin to process that backlog after a `whenNotFull` promise resolves, the first item in the backlog will create a new `whenNotFull` and proceed to await it. Subsequent items in backlog queue will do the same but overwrite the promise of the first item leading to a chain of promises for backlogged items that are unresolvable. This PR ensures that we never overwrite the `whenNotFull` promise for backlogged queue items and we can resolve them all eventually. * core: unable to close bounded priority queue (#114) core: fix issue with bounded priority queue not correctly supporting closing of queues immediately after it was drained leading to potential `Cannot read property 'resolve' of undefined` errors due to whenNotFull being undefined. * Cookie Cutter 1.2 (#119) * parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188453 -0500 parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188394 -0500 initial commit develop * Bump up the versions for google cloud NPM packages (#11) * Integration command exits with jest exit code (#40) * Integration command exits with jest exit code [full ci] * revert version bumps [full ci] * Fix failing Kafka integration test (#43) * Remove Deprecated new Buffer usage (#46) * #12 remove deprecated new Buffer usage * Support Changes in Metrics Tags for Prometheus (#38) * Support Changes in Metrics Tags for Prometheus * add integration test to .travis.yml [full ci] * set jestTimeout to a larger value [full ci] * testing change to integrate always returning 0 * [full ci] * increasing time to wait for Prometheus to scrape * [full ci] * increase wait to 120 sec [full ci] * adding localhost so Docker/Prom works for linux * [full ci] * look for ports starting with 300 [full ci] * show all results from netstat [full ci] * File service discovery for Prometheus targets [full ci] * moving targets.json generation to integration file * [full ci] * [full ci] * slimming down docker-compose.yml file * [full ci] * addressing comments * [full ci] * Clean up for review * Fix IMetricsTags to ILabelValues conversion * Addressing comments * kubernetes: adjust logic for creating a watch (#48) kubernetes: adjust logic for creating a watch * update kubernetes-client to latest version on master that contains various internal fixes for creation and handling of a watch * adjust logic for startWatch to wrap watch function inside of a promise and handle timeouts differently * change callback for watch to not be async azure: fix build error to take in correct BufferEncoding and update kb calculations * Revert commit of incorrect yarn.lock from #48 (#50) yarn.lock was committed with private registry information. rebuild yarn.lock in order to fix deployment issues from #48. * kubernetes: fix kubernetes-client package build issue with Node 8 (#51) kubernetes-client package requires use of node version greater than 8 but we require 8 in the travis ci build pipeline. as a temporary solution till we deprecate Node 8 completely as part of #52 update the build pipeline to use a separate package.v8.json which doesn't build it. * build: fix grep check to only match if we're using node 8 (#53) * kubernetes: update @kubernetes/client-node package to 0.11.1 (#54) * kubernetes: add in additional logging details for watch errors (#56) * improve timeout of k8s watch (#59) * Allow opt-in to creating Azure queues dynamically on write (#55) * core: update ConsoleLogger to correctly log out nested objects (#60) core: update ConsoleLogger to correctly log out nested objects The console lo…

* parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188453 -0500 parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188394 -0500 initial commit develop * Bump up the versions for google cloud NPM packages (#11) * Integration command exits with jest exit code (#40) * Integration command exits with jest exit code [full ci] * revert version bumps [full ci] * Fix failing Kafka integration test (#43) * Remove Deprecated new Buffer usage (#46) * #12 remove deprecated new Buffer usage * Support Changes in Metrics Tags for Prometheus (#38) * Support Changes in Metrics Tags for Prometheus * add integration test to .travis.yml [full ci] * set jestTimeout to a larger value [full ci] * testing change to integrate always returning 0 * [full ci] * increasing time to wait for Prometheus to scrape * [full ci] * increase wait to 120 sec [full ci] * adding localhost so Docker/Prom works for linux * [full ci] * look for ports starting with 300 [full ci] * show all results from netstat [full ci] * File service discovery for Prometheus targets [full ci] * moving targets.json generation to integration file * [full ci] * [full ci] * slimming down docker-compose.yml file * [full ci] * addressing comments * [full ci] * Clean up for review * Fix IMetricsTags to ILabelValues conversion * Addressing comments * kubernetes: adjust logic for creating a watch (#48) kubernetes: adjust logic for creating a watch * update kubernetes-client to latest version on master that contains various internal fixes for creation and handling of a watch * adjust logic for startWatch to wrap watch function inside of a promise and handle timeouts differently * change callback for watch to not be async azure: fix build error to take in correct BufferEncoding and update kb calculations * Revert commit of incorrect yarn.lock from #48 (#50) yarn.lock was committed with private registry information. rebuild yarn.lock in order to fix deployment issues from #48. * kubernetes: fix kubernetes-client package build issue with Node 8 (#51) kubernetes-client package requires use of node version greater than 8 but we require 8 in the travis ci build pipeline. as a temporary solution till we deprecate Node 8 completely as part of #52 update the build pipeline to use a separate package.v8.json which doesn't build it. * build: fix grep check to only match if we're using node 8 (#53) * kubernetes: update @kubernetes/client-node package to 0.11.1 (#54) * kubernetes: add in additional logging details for watch errors (#56) * improve timeout of k8s watch (#59) * Allow opt-in to creating Azure queues dynamically on write (#55) * core: update ConsoleLogger to correctly log out nested objects (#60) core: update ConsoleLogger to correctly log out nested objects The console logger renders out `[object Object]` if a field is an object so destructure it into `.` delimited nested fields (e.g. foo.bar.baz) * Update ConsoleLogger to take in options letting users select a maxDepth for log outputs * kubernetes: create k8sPollSource that periodically polls for resources instead of using a watch (#61) * Create `KubernetesBase` class that both poll and watch sources extend from * Add in tests for `KubernetesPollSource` * Add preprocessor concept for Azure Queues to support messages not following the envelope format (#65) * fix missing bind when creating new azure queues (#63) * fix lint errors (#67) * refactor queue client to avoid exporting type from azure library as p… (#68) * refactor queue client to avoid exporting type from azure library as part of public api * kubernetes: add in timeout to poll request in case request to k8s hangs (#70) * Fix bug when options are ignored in QueueClient (#69) * Decorator returning class with empty string name (#73) * Decorator returning class with empty string name [full cli] * remove export, add link to TS issue [full cli] * bump version [full ci] * core: do not overwrite pending promises for the backlog of queued items (#76) It's possible to get into a state where we've hit our limit on items that we can add to the bounded priority queue and a large backlog of items are accumulated. Once we can begin to process that backlog after a `whenNotFull` promise resolves, the first item in the backlog will create a new `whenNotFull` and proceed to await it. Subsequent items in backlog queue will do the same but overwrite the promise of the first item leading to a chain of promises for backlogged items that are unresolvable. This PR ensures that we never overwrite the `whenNotFull` promise for backlogged queue items and we can resolve them all eventually. * core: version bump to 1.2.0-beta.11 (#77) * Update Cookie Cutter Dependencies (#75) * better throughput in RPC mode while guaranteeing correctness of state (#83) * Update dependencies for Node 8 (#86) * Update dependencies for Node 8 * add white space [full ci] * remove white space [full ci] * A few more deps updates [full ci] Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Reveal Azure Blob & Queue Service URL params, to allow for pointing at a local emulator. (#89) * Way to Inspect 'Invalid' Messages (#82) * Way to Inspect 'Invalid' Messages * unit test * remove annotator from unit test * allow publishing from inside the invalid handler * Updating docs * impoving clarity of doc entry * actual change of doc * add failSpan if input validation fails for Serial * Use custom error to signal no invalid msg handler * Missed files * Do not propagate NoInvalidHandlerError * Remove custom error and add hasInvalid function * add case in unit test * more tests for ConventionBasedMessageDispatcher * refactoring to simplify code * addressing comments Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Better Log Message when a Message Fails to Process and fix to upsertSproc SeqConErr details (#91) * Better Log Message when a Message Fails to Process * upsertSproc SeqConErr details fix * use context's logger Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Update CHANGELOG.md (#98) * bump develop to 1.3 (#106) * Add Dead Letter Queue to QueueInputSource (#93) * Add Dead Letter Queue to QueueInputSource * Addressing comments * pass in a modified config to dead letter queue * addressing comment * changing API to expect values in milliseconds * Updating docs * Docs update with example dead letter queue config * update package.jsoon * doc nit Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * merge release/1.2 into develop (#112) * upgrade dependencies due to vulnerabilities (#111) * Remove support for node 8 (#113) * core: unable to close bounded priority queue (#116) core: fix issue with bounded priority queue not correctly supporting closing of queues immediately after it was drained leading to potential Cannot read property 'resolve' of undefined errors due to whenNotFull being undefined. * ConcurrentMessageProcessor suppresses error details (#124) * ConcurrentMessageProcessor suppresses error details When message handling fails outside of the message handler the ConcurrentMessageHandler currently throws a generic Error that hides the underlying root cause error. This PR changes it to re-throw the original error, similar to what the SerialMessageProcessor is doing. * Fix breaking API change in Azure Queues (#122) * Fix breaking API change in Azure Queues (#121) * Adding source unit to Config's timespanOf function * Adding Azure Queue change and other fixes Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * DeadLetterQueue fixes * bump versions * rebase and bump version Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add RedisStreamSink & RedisStreamSource (#126) * Add GCP PubSub Sink (#125) * Bump Redis version to publish new package (#127) do version bump missing in PR #126 * prometheus module should not throw an error when incrementing by 0 (#130) merge back from master * Prevent config.parse() output from being used as input (#134) * multi cosmos collections (#81) (#117) * fix MsSqlSink throws wrong error (#140) * fix MsSqlSink throws wrong error * Update package.json * Update MssqlSink.ts * Add AMQP Sink + Source (#136) * Add AMQP Sink + Source * add the actual Sink/Source files * properly close the sink's connection * Adding initialization and disposal to source * Basic producer and consumer scripts. * replace AsyncPipe with BoundedPriorityQueue * Refactor to get correct produce/consume behavior * improve connection call and add port as optional * fix yaml.lock file * Adding integration test * add new line at end of yml file * Add Copyright text and set "--passWithNoTest" * add msg release listener [full ci] * add AMQP integration to travis build * fix .travis.yml [full ci] * addressing comments * Fix integration test/setup [full ci] * Adding tracing [full ci] * minor corrections Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add metrics to AMQP (#144) * Add metrics to AMQP * bump version * switch to this.channel * trigger [full ci] * Adding periodic metrics * Add metadata [full ci] * Lint and style fix [full ci] * Addressing comments Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add docs to AMQP package (#147) * Add docs to AMQP package * correcting module name * Addressing comments * Adding example files * minor nits * rename * Simplify config Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add cookie-cutter-jaeger (#151) * Add cookie-cutter-jaeger * remove interface * add lock * update per comments Co-authored-by: Marco Garcia <marco.garcia@jet.computer> * cookie-cutter-redis: add support for multiple streams (#155) * cc-redis: add support for multiple streams * fix test * update spanLogAndSetTags * update getPendingMessagesForConsumerGroup * update RedisStreamSink * rename test * add tests * add more tests * update per feedback * update xReadGroup * full ci Co-authored-by: Marco Garcia <marco.garcia@jet.computer> * cookie-cutter-redis: add metrics (#156) * cookie-cutter-redis: add metrics * full ci * update docs * update docs * full ci * remove array tag Co-authored-by: Marco Garcia <marco.garcia@jet.computer> * Have separate queue capacity per priority level (#160) * Have separate queue capacity per priority level * Fix memory leak likely caused by promise chaining * fixing floating promises * white space change [full ci] Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> Co-authored-by: Sebastian Klose <sebastian.klose@walmart.com> * Fix lz4 error by adding resolution (#161) * Fix lz4 error by adding resolution * yarn.lock file change Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add new Jaeger package to README (#158) * fix broken metrics, reclaim PEL messages less often (#163) * fix pending list not fully drained on startup, added password config (#164) * fix messages are acked on error (#168) * check for failed acks to redis (#169) * fix xReadGroup ignores all but first message from batch (#170) * Implement IEncodedMessageEmbedder for ProtoMessageEncoder (#174) * Implement IEncodedMessageEmbedder for ProtoMessageEncoder * version change * unit tests Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * cleanup redis stream implementation (#171) * make some redis options nullable so the default value can be overwrit… (#177) * troubleshoot redis stream issue (#179) * Allow negative values in Prometheus histogram (#181) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Prevent BoundedPriorityQueue from deprioritizing waiting enqueue calls (#180) * Prevent BoundedPriorityQueue from deprioritizing waiting enqueue calls * versioon bump * handle floating promises Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Fix backwards compatibility for ProtoMessageEncoder (#182) * Version bump for Proto change (#183) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Change LogLevel from Error -> Warn when retrieving Kafka watermarks (#184) * fix 'yarn audit' issues [full ci] (#187) * fix sec vuln in node-fetch (#189) * Add auth for amqp source and sink (#192) * Add auth for amqp source and sink * trigger [full ci] * do not overwrite default creds [full ci] * do not overwrite default creds [full ci] * do not overwrite default creds [full ci] * do not overwrite default creds [full ci] * add missing new line [full ci] * lint fix * trigger ci [full ci] * trigger ci Co-authored-by: prachi.tandon@jet.computer <prachi.tandon@jet.com> * Add AMQP package to README (#193) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add vhost support for amqp (#194) * Add vhost support for amqp * Add vhost support for amqp * Add vhost support for amqp [full ci] * Add vhost support for amqp * Add vhost support for amqp * Add vhost support for amqp * version bump Co-authored-by: prachi.tandon@jet.computer <prachi.tandon@jet.com> * kubernetes: adjust logging to be less verbose (#195) * detect when kafkajs is stuck with stale broker metadata (#186) * detect when kafkajs is stuck with stale broker metadata and terminate application * lint * Ensure RedisClient's "type" metric label is always a string (#196) * Ensure metrics type label is always a string * Bump cookie-cutter-redis version to 1.3.0-beta.13 * Fix wrong string function in Kafka (#197) * Fix wrong string function in Kafka * proper conversion of object to string Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * address vulnerability in node-forge package [full ci] (#200) * bump develop to 1.4 (#201) * Don't extract data field when encoder isEmbeddable (#203) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Azure module integration tests (#204) * updates blob client to use latest azure storage library * updates blob client to use latest azure storage library * [full ci] * [full ci] * adds resolution for lz4 to avoid 0.6.4 which causes build errors * updates yarn lockfile to use correct npm repo * fixes linting errors in tests * addresses pr comments * fixes https agent compatibility * fixes linting errors * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * updates queue client to use latest azure storage library * updates yarn lock file, removing references to incorrect registry * [full ci] * [full ci] * [full ci] * [full ci] * adds integration tests for azure module * updates yarn lockfile * adds files to setup emulators for local use * removes references to incorrect registry from yarn lockfile * fixes linting errors * rerun tests * [full ci] * [full ci] * [full ci] * adds azure integration tests to travis build * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * use windows batch file to run integration tests * [full ci] * [full ci] * [full ci] * adds missing keys for integration tests * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * [full ci] * Copy over file with different Timeout * triger build * full ci [full ci] * checking what's available to the script * [full ci] * copy modified psm1 in travis [full ci] * moving commands to after install [full ci] * adding a folder that was missing in the path * [full ci] * Make timeout 40 minutes [full ci] * Change Timeout for Stop-CosmosDbEmulator to 20 min [full ci] * [full ci] needs to be on the first line * and end of line... [full ci] * [full ci] * [full ci] * remove timeout value [full ci] * Explicitly setting new Timeout [full ci] * Verify modified file is the one running [full ci] * Matching change in run integration [full ci] * Move copy before import [full ci] * fixing variable mixup [full ci] * Tell travis to wait 20 min instead of 10 * [full ci] * tweak travis [full ci] * tweaking travis_await [full ci] * tweaking travis again [full ci] * moving travis_wait to install command [full ci] * split script in 2 lines [full ci] * up travis_wait to 60 minutes [full ci] * Testing with 2 hour limit [full ci] * Test code that worked in test repo * [full ci] * Fix path and add missing ennv variables * [full ci] * Add integration test call * [full ci] * Add yarn command [full ci] * Add yarn build and pwd to check location * [full ci] * Add a bunch of print outs * [full ci] * remove extra cd .. [full ci] * [full ci] * fix Get-ChildItem command [full ci] * build CC-core first [full ci] * MASTER_KEY -> COSMOS_SECRET_KEY * [full ci] * Add timing to script [full ci] * Fix cd command [full ci] * Add CI var to travis [full ci] * Use same file for local and CI builds [full ci] * Refactor [full ci] * [full ci] * [full ci] * [full ci] * remove comment from .travis [full ci] * Prepare for review [full ci] * updates queue client to use latest azure storage library updates yarn lock file, removing references to incorrect registry updates blob client to use latest azure storage library updates blob client to use latest azure storage library updates yarn lockfile to use correct npm repo fixes linting errors in tests addresses pr comments fixes https agent compatibility fixes linting errors adds integration tests for azure module updates yarn lockfile adds files to setup emulators for local use removes references to incorrect registry from yarn lockfile fixes linting errors adds azure integration tests to travis build [full ci] [full ci] [full ci] use windows batch file to run integration tests [full ci] [full ci] adds missing keys for integration tests [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] [full ci] Copy over file with different Timeout triger build full ci [full ci] checking what's available to the script [full ci] copy modified psm1 in travis [full ci] moving commands to after install [full ci] adding a folder that was missing in the path [full ci] Make timeout 40 minutes [full ci] Change Timeout for Stop-CosmosDbEmulator to 20 min [full ci] [full ci] needs to be on the first line and end of line... [full ci] [full ci] [full ci] remove timeout value [full ci] Explicitly setting new Timeout [full ci] Verify modified file is the one running [full ci] Matching change in run integration [full ci] Move copy before import [full ci] fixing variable mixup [full ci] Tell travis to wait 20 min instead of 10 [full ci] tweak travis [full ci] tweaking travis_await [full ci] tweaking travis again [full ci] moving travis_wait to install command [full ci] split script in 2 lines [full ci] up travis_wait to 60 minutes [full ci] Testing with 2 hour limit [full ci] Test code that worked in test repo [full ci] Fix path and add missing ennv variables [full ci] Add integration test call [full ci] Add yarn command [full ci] Add yarn build and pwd to check location [full ci] Add a bunch of print outs [full ci] remove extra cd .. [full ci] [full ci] fix Get-ChildItem command [full ci] build CC-core first [full ci] MASTER_KEY -> COSMOS_SECRET_KEY [full ci] Add timing to script [full ci] Fix cd command [full ci] Add CI var to travis [full ci] Use same file for local and CI builds [full ci] Refactor [full ci] [full ci] [full ci] [full ci] remove comment from .travis [full ci] Prepare for review [full ci] * yarn.lock [full ci] * yarn integrate does own setup [full ci] * Integrate cleans up unless arg is passed [full ci] * Fully stop VM on default integration run * Update README Co-authored-by: Emma Lynch <emmalynch@users.noreply.github.com> Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * fix redis client skipping over messages when there is an invalid message in a batch (#209) * Fix: Parse string error message to metrics (#1) (#210) * [kafka] Expose compression settings for producers via kafkaSink (#215) * force application to restart on disconnect because reconnect doesn't seem to work properly (#217) * upgrading gcp dependencies (#218) * wrong error is being logged when redis ack fails (#221) * Add Azure BlobClient to public API (#214) * Log out a detailed error msg while inserting data into big query (#229) * Log out a detailed error msg for inserting data into big query * Add a comma * Default detailedErrorMsg to e, remove NullLogger * Spelling * Remove json stringify * remove json stringify * Remove {} * Add new lines * Prettier error Co-authored-by: Shwetha Narayanan <s0n027b@homeoffice.wal-mart.com> * Switch Yarn audit level to critical (#232) * Switch Yarn audit level to critical [full ci] * disable cron check for audit * Only return non-zero on critical * Only log critical * revert back to type = cron Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Properly connect/disconnect Kafkajs producer (#234) * Properly connect/disconnect Kafkajs producer * nit * [full ci] * white space * smoke build Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Updating version of mssql to recent (#235) * updating version of mssql to recent * bump package version * adding timeouts and tests * linter * adding streaming [full ci] and full ci test * rerun tests with [full ci] * adding defaults * updating to 5.1.5 [full ci] * updating types [full ci] * travis file build [full ci] * white space * remove white space to make new build * Add TLS support to Kafka package (#230) * Add tls support to kafka package * trigger [full ci] * Kafka integration test fix [full ci] * revert travis changes * clean up * Bump version again Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Fixed issue with wrong grpcClient timing metric's unit (#220) * Dispatch context store API metadata (#238) * Update packages to reduce vulnerabilities (#236) * Update packages to reduce vulnerabilities * blank change * [full ci] * Bump versions * Remove temp change to audit * Generate yarn.lock from scratch [full ci] * Add longer jest timeout [full ci] * Fix new vulnerability * Bump versions after rebase * Clean up and [full ci] * Trigger only AMQP and Kafka [full ci] * Add docker login [full ci] * Add all components of .travis.yml back [full ci] * Test running only for integrate [full ci] * try if stage = integration [full ci] * Try if stage [full ci] * Check OS [full ci] * Run as script: [full ci] * Bash if os [full ci] * remove script: [full ci] * echo attributes [full ci] * Docker login to each integration test [full ci] * bump core version Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Include `ttl` setting when storing to Azure cosmosDB (#240) * Handle url separately from connectionString (#233) * Handle url separetely from connectionString * [full ci] * Run just Azure integration [full ci] * Add optional connectionString * [full ci] * revert connection string value [full ci] * use proper connectionString value [full ci] * Make url optional and add default url building * [full ci] * Revert all changes [full ci] * tweak travis [full ci] * log config in constructor [full ci] * Add connectionString back to QueueClient [full ci] * Finalize [full ci] * cleanup * version bump * Add comments adding clarity * Make url backwards compatible [full ci] * bump version Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> Co-authored-by: Sebastian Klose <sebastian.klose@walmart.com> * Missing `metadata` param from `IDispatchContext` interface (#243) * Initialize github action workflow for develop (#246) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Run cron more frequently (#251) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Add full workflow to develop (#242) * Test github action workflows * Add more components to github workflow * Fix missing $ * Use specific yarn version * Add yarn set version * Just use yarn set * Ubuntu Node 12 only * Only Test Windows * Test with macOS only * Test all three * Split test in 3 separate jobs * Modify macOS to run set and install in one run * Add needs smoke * with github token * Move with: before run * Missed a with after a run * Move with: to a step with uses: * Switch repo-token to token * Try repo-token in node setup step * Add repo-token to yarn step in macOS test * speed up smoke job * env for macOS only * remove repo-token from with * Try with all 3 test jobs * Add Integration test step * Fix typo in array * Add all ubuntu integration tests * Add Azure test on Windows * remove test yml * Echo default env variables * Try commit message * Evaluate before assigning to env var * Wrong step env * Try with event * echo github_context * Evaluate github cintext and assign to env var * try to extract commits * Extract [0] commit * Add .href * Rename yml file * Add more CMT vars * Just one cmt var * INPUT_COMMITS * git log command * another version of git log * missing quotes * Just git log -1 * specify --git-dir * --first-parent * -2 on git log * git log only * git reflog * git reflog show * Event Before and After * Add quotes * Just after * checkout with head.sha * online * missing quote * before not working, switching to after * Move call to earlier part * Move another command * Set env var from commit and display many contexts * try cat * If condition * if in a step * Set env var * Add correct script * remove extra : * Missing } * Can the next job access the env var * Try with brackets * Add cron for 17:15 and start cleanup * Match GMT * Try the env and artifact for next job * Fix missing run * Artifact file is empty, try to fix * Still empty * Resolving the env first * missing run * Remove $() * this has full ci * Initia set up with full if * Let's try a full ci * Do not use contractions in commit message full ci * Switch away from using artifacts * Clean up and run all without the trigger * Add [] to full ci and minor corrections [full ci] * Debug why windows condition fails [full ci] * Typo * Windows specific env set * Try with set instead of setx [full ci] * Try another version * MIssing quote * Test env var for windows * TRy another thing * Try it with the git command * Don't specify shell as bash * echo COMMIT_MESSAGE in same run * Add quotes * More trying * Try different env var name * Test var 1 * Try var 2 * Try var 3 * Try var 4 * Try without full ci * And now with [full ci] * Make change to second windows job [full ci] * Fix needs * Return other jobs [full-ci] * Return all parts of all jobs * Add placeholders for publish steps and names * Clean up publish jobs and change cron * Minimize travis and run [full ci] * Run just azure integration [full ci] * Test better exit code [full ci] * Print exit code for integrate.sh [full ci] * Run decrypt/encrypt * Add on: workflow_dispatch * Rename Workflow * Add details to workflow_dispatch * Rename workflow as Develop * Rename workflow file * Add full workflow to develop * Revert commented travis * white space * [full-ci] * Try a custom action in same repo * Try with regular checkout first * Revert back to regular conditional checkout * remove duplicate yarn install from macOs test * [full ci] * Uncomment jobs * remove commented out line * Add if on jobs * Trying job outputs * Try with a simple assignment * fix job name * Try env.COMMIT_MESSAGE again * Missing $ * Echo just env var * Try with ENV: * With correct sha * proper sha on checkout * Try if on previous job output [small ci] * Missing ' * Run without full ci * white space * [full ci] * Clean up and run a full [full ci] * Try exit_code without quotes [full ci] * Actual change [full ci] * rebase [full ci] * uncomment * Single checkout and split build/test [full ci] * rebase * [full ci] * rebase * cron and [full ci] * white space * [full ci] * Add exit_code to PowerShell script [full ci] * Clean up non gh action changes and [full ci] * Fix needs and [full ci] * Line [full ci] * Don't get commit message on push * white space * Trigger build with very long commit message that takes more than one line * Convert into oneline commit message. This is also really long * Remove conditions for commit message * Add yarn install steps to npm publish and pages * Check github.ref on publish pages * white space * one last [full ci] Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Azure Integration not failing on failed tests (#252) * Azure Integration not failing on failed tests * Set $ErrorActionPreference = "Stop" [full ci] * Throw error from ps scrippt [full ci] * Simplify [full ci] * Last Exit Code [full ci] * Try [full ci] * Simplify [full ci] * Integration test fix [full ci] * Exit code from PS is boolean [full ci] * Put the rest of the workflow back in * Missing empty line * Another missing empty line Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Merge master into develop (#267) * Update .travis.yml * Decorator returning class with empty string name (hotfix 1.1) (#74) * Decorator returning class with empty string name [full cli] * bump version [full ci] * build change to address issue with yarn 1.22.0 * [full ci] * core: support multiple concurrent writers for BoundedPriorityQueue (#78) It's possible to get into a state where we've hit our limit on items that we can add to the bounded priority queue and a large backlog of items are accumulated. Once we can begin to process that backlog after a `whenNotFull` promise resolves, the first item in the backlog will create a new `whenNotFull` and proceed to await it. Subsequent items in backlog queue will do the same but overwrite the promise of the first item leading to a chain of promises for backlogged items that are unresolvable. This PR ensures that we never overwrite the `whenNotFull` promise for backlogged queue items and we can resolve them all eventually. * core: unable to close bounded priority queue (#114) core: fix issue with bounded priority queue not correctly supporting closing of queues immediately after it was drained leading to potential `Cannot read property 'resolve' of undefined` errors due to whenNotFull being undefined. * Cookie Cutter 1.2 (#119) * parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188453 -0500 parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188394 -0500 initial commit develop * Bump up the versions for google cloud NPM packages (#11) * Integration command exits with jest exit code (#40) * Integration command exits with jest exit code [full ci] * revert version bumps [full ci] * Fix failing Kafka integration test (#43) * Remove Deprecated new Buffer usage (#46) * #12 remove deprecated new Buffer usage * Support Changes in Metrics Tags for Prometheus (#38) * Support Changes in Metrics Tags for Prometheus * add integration test to .travis.yml [full ci] * set jestTimeout to a larger value [full ci] * testing change to integrate always returning 0 * [full ci] * increasing time to wait for Prometheus to scrape * [full ci] * increase wait to 120 sec [full ci] * adding localhost so Docker/Prom works for linux * [full ci] * look for ports starting with 300 [full ci] * show all results from netstat [full ci] * File service discovery for Prometheus targets [full ci] * moving targets.json generation to integration file * [full ci] * [full ci] * slimming down docker-compose.yml file * [full ci] * addressing comments * [full ci] * Clean up for review * Fix IMetricsTags to ILabelValues conversion * Addressing comments * kubernetes: adjust logic for creating a watch (#48) kubernetes: adjust logic for creating a watch * update kubernetes-client to latest version on master that contains various internal fixes for creation and handling of a watch * adjust logic for startWatch to wrap watch function inside of a promise and handle timeouts differently * change callback for watch to not be async azure: fix build error to take in correct BufferEncoding and update kb calculations * Revert commit of incorrect yarn.lock from #48 (#50) yarn.lock was committed with private registry information. rebuild yarn.lock in order to fix deployment issues from #48. * kubernetes: fix kubernetes-client package build issue with Node 8 (#51) kubernetes-client package requires use of node version greater than 8 but we require 8 in the travis ci build pipeline. as a temporary solution till we deprecate Node 8 completely as part of #52 update the build pipeline to use a separate package.v8.json which doesn't build it. * build: fix grep check to only match if we're using node 8 (#53) * kubernetes: update @kubernetes/client-node package to 0.11.1 (#54) * kubernetes: add in additional logging details for watch errors (#56) * improve timeout of k8s watch (#59) * Allow opt-in to creating Azure queues dynamically on write (#55) * core: update ConsoleLogger to correctly log out nested objects (#60) core: update ConsoleLogger to correctly log out nested objects The console logger renders out `[object Object]` if a field is an object so destructure it into `.` delimited nested fields (e.g. foo.bar.baz) * Update ConsoleLogger to take in options letting users select a maxDepth for log outputs * kubernetes: create k8sPollSource that periodically polls for resources instead of using a watch (#61) * Create `KubernetesBase` class that both poll and watch sources extend from * Add in tests for `KubernetesPollSource` * Add preprocessor concept for Azure Queues to support messages not following the envelope format (#65) * fix missing bind when creating new azure queues (#63) * fix lint errors (#67) * refactor queue client to avoid exporting type from azure library as p… (#68) * refactor queue client to avoid exporting type from azure library as part of public api * kubernetes: add in timeout to poll request in case request to k8s hangs (#70) * Fix bug when options are ignored in QueueClient (#69) * Decorator returning class with empty string name (#73) * Decorator returning class with empty string name [full cli] * remove export, add link to TS issue [full cli] * bump version [full ci] * core: do not overwrite pending promises for the backlog of queued items (#76) It's possible to get into a state where we've hit our limit on items that we can add to the bounded priority queue and a large backlog of items are accumulated. Once we can begin to process that backlog after a `whenNotFull` promise resolves, the first item in the backlog will create a new `whenNotFull` and proceed to await it. Subsequent items in backlog queue will do the same but overwrite the promise of the first item leading to a chain of promises for backlogged items that are unresolvable. This PR ensures that we never overwrite the `whenNotFull` promise for backlogged queue items and we can resolve them all eventually. * core: version bump to 1.2.0-beta.11 (#77) * Update Cookie Cutter Dependencies (#75) * better throughput in RPC mode while guaranteeing correctness of state (#83) * Update dependencies for Node 8 (#86) * Update dependencies for Node 8 * add white space [full ci] * remove white space [full ci] * A few more deps updates [full ci] Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Reveal Azure Blob & Queue Service URL params, to allow for pointing at a local emulator. (#89) * Way to Inspect 'Invalid' Messages (#82) * Way to Inspect 'Invalid' Messages * unit test * remove annotator from unit test * allow publishing from inside the invalid handler * Updating docs * impoving clarity of doc entry * actual change of doc * add failSpan if input validation fails for Serial * Use custom error to signal no invalid msg handler * Missed files * Do not propagate NoInvalidHandlerError * Remove custom error and add hasInvalid function * add case in unit test * more tests for ConventionBasedMessageDispatcher * refactoring to simplify code * addressing comments Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Better Log Message when a Message Fails to Process and fix to upsertSproc SeqConErr details (#91) * Better Log Message when a Message Fails to Process * upsertSproc SeqConErr details fix * use context's logger Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Update CHANGELOG.md (#98) * prepare 1.2-rc.1 (#105) * upgrade dependencies due to vulnerabilities (#111) * bump version * update changelog / fix link Co-authored-by: Kshitiz Gupta <kgupta40@gatech.edu> Co-authored-by: plameniv <47831130+plameniv@users.noreply.github.com> Co-authored-by: Connor Ross <cross311@users.noreply.github.com> Co-authored-by: Tanvir Alam <talam@users.noreply.github.com> Co-authored-by: Chris Pinola <trogdor3222@gmail.com> Co-authored-by: Ilya Butorine <i@ilya.com> Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> Co-authored-by: Sean Halpin <softdev87@users.noreply.github.com> * Fix breaking API change in Azure Queues (#121) * Adding source unit to Config's timespanOf function * Adding Azure Queue change and other fixes Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * prometheus module should not throw an error when incrementing by 0 (#128) * Have separate queue capacity per priority level (#162) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Stabilize 1.3 (#202) * parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188453 -0500 parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188394 -0500 initial commit develop * Bump up the versions for google cloud NPM packages (#11) * Integration command exits with jest exit code (#40) * Integration command exits with jest exit code [full ci] * revert version bumps [full ci] * Fix failing Kafka integration test (#43) * Remove Deprecated new Buffer usage (#46) * #12 remove deprecated new Buffer usage * Support Changes in Metrics Tags for Prometheus (#38) * Support Changes in Metrics Tags for Prometheus * add integration test to .travis.yml [full ci] * set jestTimeout to a larger value [full ci] * testing change to integrate always returning 0 * [full ci] * increasing time to wait for Prometheus to scrape * [full ci] * increase wait to 120 sec [full ci] * adding localhost so Docker/Prom works for linux * [full ci] * look for ports starting with 300 [full ci] * show all results from netstat [full ci] * File service discovery for Prometheus targets [full ci] * moving targets.json generation to integration file * [full ci] * [full ci] * slimming down docker-compose.yml file * [full ci] * addressing comments * [full ci] * Clean up for review * Fix IMetricsTags to ILabelValues conversion * Addressing comments * kubernetes: adjust logic for creating a watch (#48) kubernetes: adjust logic for creating a watch * update kubernetes-client to latest version on master that contains various internal fixes for creation and handling of a watch * adjust logic for startWatch to wrap watch function inside of a promise and handle timeouts differently * change callback for watch to not be async azure: fix build error to take in correct BufferEncoding and update kb calculations * Revert commit of incorrect yarn.lock from #48 (#50) yarn.lock was committed with private registry information. rebuild yarn.lock in order to fix deployment issues from #48. * kubernetes: fix kubernetes-client package build issue with Node 8 (#51) kubernetes-client package requires use of node version greater than 8 but we require 8 in the travis ci build pipeline. as a temporary solution till we deprecate Node 8 completely as part of #52 update the build pipeline to use a separate package.v8.json which doesn't build it. * build: fix grep check to only match if we're using node 8 (#53) * kubernetes: update @kubernetes/client-node package to 0.11.1 (#54) * kubernetes: add in additional logging details for watch errors (#56) * improve timeout of k8s watch (#59) * Allow opt-in to creating Azure queues dynamically on write (#55) * core: update ConsoleLogger to correctly log out nested objects (#60) core: update ConsoleLogger to correctly log out nested objects The console logger renders out `[object Object]` if a field is an object so destructure it into `.` delimited nested fields (e.g. foo.bar.baz) * Update ConsoleLogger to take in options letting users select a maxDepth for log outputs * kubernetes: create k8sPollSource that periodically polls for resources instead of using a watch (#61) * Create `KubernetesBase` class that both poll and watch sources extend from * Add in tests for `KubernetesPollSource` * Add preprocessor concept for Azure Queues to support messages not following the envelope format (#65) * fix missing bind when creating new azure queues (#63) * fix lint errors (#67) * refactor queue client to avoid exporting type from azure library as p… (#68) * refactor queue client to avoid exporting type from azure library as part of public api * kubernetes: add in timeout to poll request in case request to k8s hangs (#70) * Fix bug when options are ignored in QueueClient (#69) * Decorator returning class with empty string name (#73) * Decorator returning class with empty string name [full cli] * remove export, add link to TS issue [full cli] * bump version [full ci] * core: do not overwrite pending promises for the backlog of queued items (#76) It's possible to get into a state where we've hit our limit on items that we can add to the bounded priority queue and a large backlog of items are accumulated. Once we can begin to process that backlog after a `whenNotFull` promise resolves, the first item in the backlog will create a new `whenNotFull` and proceed to await it. Subsequent items in backlog queue will do the same but overwrite the promise of the first item leading to a chain of promises for backlogged items that are unresolvable. This PR ensures that we never overwrite the `whenNotFull` promise for backlogged queue items and we can resolve them all eventually. * core: version bump to 1.2.0-beta.11 (#77) * Update Cookie Cutter Dependencies (#75) * better throughput in RPC mode while guaranteeing correctness of state (#83) * Update dependencies for Node 8 (#86) * Update dependencies for Node 8 * add white space [full ci] * remove white space [full ci] * A few more deps updates [full ci] Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Reveal Azure Blob & Queue Service URL params, to allow for pointing at a local emulator. (#89) * Way to Inspect 'Invalid' Messages (#82) * Way to Inspect 'Invalid' Messages * unit test * remove annotator from unit test * allow publishing from inside the invalid handler * Updating docs * impoving clarity of doc entry * actual change of doc * add failSpan if input validation fails for Serial * Use custom error to signal no invalid msg handler * Missed files * Do not propagate NoInvalidHandlerError * Remove custom error and add hasInvalid function * add case in unit test * more tests for ConventionBasedMessageDispatcher * refactoring to simplify code * addressing comments Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Better Log Message when a Message Fails to Process and fix to upsertSproc SeqConErr details (#91) * Better Log Message when a Message Fails to Process * upsertSproc SeqConErr details fix * use context's logger Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Update CHANGELOG.md (#98) * bump develop to 1.3 (#106) * Add Dead Letter Queue to QueueInputSource (#93) * Add Dead Letter Queue to QueueInputSource * Addressing comments * pass in a modified config to dead letter queue * addressing comment * changing API to expect values in milliseconds * Updating docs * Docs update with example dead letter queue config * update package.jsoon * doc nit Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * merge release/1.2 into develop (#112) * upgrade dependencies due to vulnerabilities (#111) * Remove support for node 8 (#113) * core: unable to close bounded priority queue (#116) core: fix issue with bounded priority queue not correctly supporting closing of queues immediately after it was drained leading to potential Cannot read property 'resolve' of undefined errors due to whenNotFull being undefined. * ConcurrentMessageProcessor suppresses error details (#124) * ConcurrentMessageProcessor suppresses error details When message handling fails outside of the message handler the ConcurrentMessageHandler currently throws a generic Error that hides the underlying root cause error. This PR changes it to re-throw the original error, similar to what the SerialMessageProcessor is doing. * Fix breaking API change in Azure Queues (#122) * Fix breaking API change in Azure Queues (#121) * Adding source unit to Config's timespanOf function * Adding Azure Queue change and other fixes Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * DeadLetterQueue fixes * bump versions * rebase and bump version Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add RedisStreamSink & RedisStreamSource (#126) * Add GCP PubSub Sink (#125) * Bump Redis version to publish new package (#127) do version bump missing in PR #126 * prometheus module should not throw an error when incrementing by 0 (#130) merge back from master * Prevent config.parse() output from being used as input (#134) * multi cosmos collections (#81) (#117) * fix MsSqlSink throws wrong error (#140) * fix MsSqlSink throws wrong error * Update package.json * Update MssqlSink.ts * Add AMQP Sink + Source (#136) * Add AMQP Sink + Source * add the actual Sink/Source files * properly close the sink's connection * Adding initialization and disposal to source * Basic producer and consumer scripts. * replace AsyncPipe with BoundedPriorityQueue * Refactor to get correct produce/consume behavior * improve connection call and add port as optional * fix yaml.lock file * Adding integration test * add new line at end of yml file * Add Copyright text and set "--passWithNoTest" * add msg release listener [full ci] * add AMQP integration to travis build * fix .travis.yml [full ci] * addressing comments * Fix integration test/setup [full ci] * Adding tracing [full ci] * minor corrections Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add metrics to AMQP (#144) * Add metrics to AMQP * bump version * switch to this.channel * trigger [full ci] * Adding periodic metrics * Add metadata [full ci] * Lint and style fix [full ci] * Addressing comments Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add docs to AMQP package (#147) * Add docs to AMQP package * correcting module name * Addressing comments * Adding example files * minor nits * rename * Simplify config Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add cookie-cutter-jaeger (#151) * Add cookie-cutter-jaeger * remove interface * add lock * update per comments Co-authored-by: Marco Garcia <marco.garcia@jet.computer> * cookie-cutter-redis: add support for multiple streams (#155) * cc-redis: add support for multiple streams * fix test * update spanLogAndSetTags * update getPendingMessagesForConsumerGroup * update RedisStreamSink * rename test * add tests * add more tests * update per feedback * update xReadGroup * full ci Co-authored-by: Marco Garcia <marco.garcia@jet.computer> * cookie-cutter-redis: add metrics (#156) * cookie-cutter-redis: add metrics * full ci * update docs * update docs * full ci * remove array tag Co-authored-by: Marco Garcia <marco.garcia@jet.computer> * Have separate queue capacity per priority level (#160) * Have separate queue capacity per priority level * Fix memory leak likely caused by promise chaining * fixing floating promises * white space change [full ci] Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> Co-authored-by: Sebastian Klose <sebastian.klose@walmart.com> * Fix lz4 error by adding resolution (#161) * Fix lz4 error by adding resolution * yarn.lock file change Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add new Jaeger package to README (#158) * fix broken metrics, reclaim PEL messages less often (#163) * fix pending list not fully drained on startup, added password config (#164) * fix messages are acked on error (#168) * check for failed acks to redis (#169) * fix xReadGroup ignores all but first message from batch (#170) * Implement IEncodedMessageEmbedder for ProtoMessageEncoder (#174) * Implement IEncodedMessageEmbedder for ProtoMessageEncoder * version change * unit tests Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * cleanup redis stream implementation (#171) * make some redis options nullable so the default value can be overwrit… (#177) * troubleshoot redis stream issue (#179) * Allow negative values in Prometheus histogram (#181) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Prevent BoundedPriorityQueue from deprioritizing waiting enqueue calls (#180) * Prevent BoundedPriorityQueue from deprioritizing waiting enqueue calls * versioon bump * handle floating promises Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Fix backwards compatibility for ProtoMessageEncoder (#182) * Version bump for Proto change (#183) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Change LogLevel from Error -> Warn when retrieving Kafka watermarks (#184) * fix 'yarn audit' issues [full ci] (#187) * fix sec vuln in node-fetch (#189) * Add auth for amqp source and sink (#192) * Add auth for amqp source and sink * trigger [full ci] * do not overwrite default creds [full ci] * do not overwrite default creds [full ci] * do not overwrite default creds [full ci] * do not overwrite default creds [full ci] * add missing new line [full ci] * lint fix * trigger ci [full ci] * trigger ci Co-authored-by: prachi.tandon@jet.computer <prachi.tandon@jet.com> * Add AMQP package to README (#193) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Add vhost support for amqp (#194) * Add vhost support for amqp * Add vhost support for amqp * Add vhost support for amqp [full ci] * Add vhost support for amqp * Add vhost support for amqp * Add vhost support for amqp * version bump Co-authored-by: prachi.tandon@jet.computer <prachi.tandon@jet.com> * kubernetes: adjust logging to be less verbose (#195) * detect when kafkajs is stuck with stale broker metadata (#186) * detect when kafkajs is stuck with stale broker metadata and terminate application * lint * Ensure RedisClient's "type" metric label is always a string (#196) * Ensure metrics type label is always a string * Bump cookie-cutter-redis version to 1.3.0-beta.13 * Fix wrong string function in Kafka (#197) * Fix wrong string function in Kafka * proper conversion of object to string Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * address vulnerability in node-forge package [full ci] (#200) * create 1.3-rc [full ci] * bump version, add missing license headers * fix code dupe * update changelog Co-authored-by: Kshitiz Gupta <kgupta40@gatech.edu> Co-authored-by: plameniv <47831130+plameniv@users.noreply.github.com> Co-authored-by: Connor Ross <cross311@users.noreply.github.com> Co-authored-by: Tanvir Alam <talam@users.noreply.github.com> Co-authored-by: Chris Pinola <trogdor3222@gmail.com> Co-authored-by: Ilya Butorine <i@ilya.com> Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> Co-authored-by: Sean Halpin <softdev87@users.noreply.github.com> Co-authored-by: Emma Lynch <emmalynch@users.noreply.github.com> Co-authored-by: Dillon Mulroy <dillon.mulroy@gmail.com> Co-authored-by: Kshitiz Gupta <kshitiz.gupta@walmart.com> Co-authored-by: Chris Pinola <chris@pinola.co> Co-authored-by: Marco Garcia <mag415@gmail.com> Co-authored-by: Marco Garcia <marco.garcia@jet.computer> Co-authored-by: prachi30 <51481988+prachi30@users.noreply.github.com> Co-authored-by: prachi.tandon@jet.computer <prachi.tandon@jet.com> * Don't extract data field in QueueInputSource when encoder isEmbeddable (#205) * Don't extract data field in QueueInputSource when encoder isEmbeddable * remove yarn.lock changes * trigger [full ci] * clean up Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> * Initialize github action workflow for master (#244) * Initialize github action workflow for master * Rename workflow * Cron at minute 40 of each hour * Clean up * Add develop branch as well Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Run with different branch on schedule (#248) * Run with different branch on schedule * More cron times * Add inputs for workflow_dispatch Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Make checkout jobs mutually exclusive, add labels (#249) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Add full workflow to master (#250) * Add full workflow to master * Run all jobs on master without full ci keyword * Add longer timeout in core test * Use only 1 checkout step [ful ci] * Shorted build for integrate jobs [full ci] * Cleanup and rename workflow file [full ci] * Run [full ci] without needing macOs test job * Return macOS to integration needs Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Fix issue with commit message (#255) * Fix issue with commit message * Try without commit message * Add : after if * Add missing $ * Remove setting of commit message * Add if conditions on commit message set * Revert temp changes * Use oneline git log format instead of adding an if check on the commit message steps * Add missing space in git log command. Make this multiline again. Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Core application test increase timeout (#253) * Core application test increase timeout * Don't bump version * Add yarn install steps to npm publish and pages Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Check github.ref on push for publishing pages (#256) * Check github.ref on push for publishing pages * Switch to contains check and echo contents * remove echo of ref * clean up Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Properly connect/disconnect Kafkajs producer mstr (#259) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Remove extra if check from build step in master (#262) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Convert Redis err to str then pass to Prometheus master (#258) * Convert Redis err to str then pass to Prometheus master * Switch to only using the error name Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Remove .travis.yml and update badges (#265) Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Clean up extra files * Revert to develop yarn.lock * Missing return * Run [full ci] Co-authored-by: Sebastian Klose <sebastian.klose@walmart.com> Co-authored-by: Tanvir Alam <talam@users.noreply.github.com> Co-authored-by: Kshitiz Gupta <kgupta40@gatech.edu> Co-authored-by: Connor Ross <cross311@users.noreply.github.com> Co-authored-by: Chris Pinola <trogdor3222@gmail.com> Co-authored-by: Ilya Butorine <i@ilya.com> Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.com> Co-authored-by: Sean Halpin <softdev87@users.noreply.github.com> Co-authored-by: Emma Lynch <emmalynch@users.noreply.github.com> Co-authored-by: Dillon Mulroy <dillon.mulroy@gmail.com> Co-authored-by: Kshitiz Gupta <kshitiz.gupta@walmart.com> Co-authored-by: Chris Pinola <chris@pinola.co> Co-authored-by: Marco Garcia <mag415@gmail.com> Co-authored-by: Marco Garcia <marco.garcia@jet.computer> Co-authored-by: prachi30 <51481988+prachi30@users.noreply.github.com> Co-authored-by: prachi.tandon@jet.computer <prachi.tandon@jet.com> Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Disable error on disconnect on proper shutdown of Redis Proxy (#270) * Disable error on disconnect on proper shutdown * bump version [full ci] Co-authored-by: Plamen Ivanov <plamen.ivanov@jet.computer> * Convert Redis Error to String then pass to Prometheus Develop Continued (#274) * Update .travis.yml * Decorator returning class with empty string name (hotfix 1.1) (#74) * Decorator returning class with empty string name [full cli] * bump version [full ci] * build change to address issue with yarn 1.22.0 * [full ci] * core: support multiple concurrent writers for BoundedPriorityQueue (#78) It's possible to get into a state where we've hit our limit on items that we can add to the bounded priority queue and a large backlog of items are accumulated. Once we can begin to process that backlog after a `whenNotFull` promise resolves, the first item in the backlog will create a new `whenNotFull` and proceed to await it. Subsequent items in backlog queue will do the same but overwrite the promise of the first item leading to a chain of promises for backlogged items that are unresolvable. This PR ensures that we never overwrite the `whenNotFull` promise for backlogged queue items and we can resolve them all eventually. * core: unable to close bounded priority queue (#114) core: fix issue with bounded priority queue not correctly supporting closing of queues immediately after it was drained leading to potential `Cannot read property 'resolve' of undefined` errors due to whenNotFull being undefined. * Cookie Cutter 1.2 (#119) * parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188453 -0500 parent 598250b089f6ad64cf3655c110b6756dc0ef3302 author sklose <sebastian.klose@walmart.com> 1574703790 -0500 committer sklose <sebastian.klose@walmart.com> 1579188394 -0500 initial commit develop * Bump up the versions for google cloud NPM packages (#11) * Integration command exits with jest exit code (#40) * Integration command exits with jest exit code [full ci] * revert version bumps [full ci] * Fix failing Kafka integration test (#43) * Remove Deprecated new Buffer usage (#46) * #12 remove deprecated new Buffer usage * Support Changes in Metrics Tags for Prometheus (#38) * Support Changes in Metrics Tags for Prometheus * add integration test to .travis.yml [full ci] * set jestTimeout to a larger value [full ci] * testing change to integrate always returning 0 * [full ci] * increasing time to wait for Prometheus to scrape * [full ci] * increase wait to 120 sec [full ci] * adding localhost so Docker/Prom works for linux * [full ci] * look for ports starting with 300 [full ci] * show all results from netstat [full ci] * File service discovery for Prometheus targets [full ci] * moving targets.json generation to integration file * [full ci] * [full ci] * slimming down docker-compose.yml file * [full ci] * addressing comments * [full ci] * Clean up for review * Fix IMetricsTags to ILabelValues conversion * Addressing comments * kubernetes: adjust logic for creating a watch (#48) kubernetes: adjust logic for creating a watch * update kubernetes-client to latest version on master that contains various internal fixes for creation and handling of a watch * adjust logic for startWatch to wrap watch function inside of a promise and handle timeouts differently * change callback for watch to not be async azure: fix build error to take in correct BufferEncoding and update kb calculations * Revert commit of incorrect yarn.lock from #48 (#50) yarn.lock was committed with private registry information. rebuild yarn.lock in order to fix deployment issues from #48. * kubernetes: fix kubernetes-client package build issue with Node 8 (#51) kubernetes-client package requires use of node version greater than 8 but we require 8 in the travis ci build pipeline. as a temporary solution till we deprecate Node 8 completely as part of #52 update the build pipeline to use a separate package.v8.json which doesn't build it. * build: fix grep check to only match if we're using node 8 (#53) * kubernetes: update @kubernetes/client-node package to 0.11.1 (#54) * kubernetes: add in additional logging details for watch errors (#56) * improve timeout of k8s watch (#59) * Allow opt-in to creating Azure queues dynamically on write (#55) * core: update ConsoleLogger to correctly log out nested objects (#60) core: update ConsoleLogger to correctly log out nested objects The co…

Log out a detailed error msg for inserting data into big query

3ceb1fe

plameniv reviewed May 13, 2021

View reviewed changes

packages/gcp/src/BigQueryClient.ts Outdated Show resolved Hide resolved

Add a comma

b244cfa

plameniv reviewed May 14, 2021

View reviewed changes

packages/gcp/src/BigQueryClient.ts Show resolved Hide resolved

plameniv reviewed May 14, 2021

View reviewed changes

packages/gcp/src/BigQueryClient.ts Outdated Show resolved Hide resolved

plameniv added enhancement New feature or request M-gcp This issue is related to the GCP module labels May 14, 2021

plameniv added this to the 1.4 milestone May 14, 2021

Shwetha Narayanan added 5 commits May 14, 2021 11:48

Default detailedErrorMsg to e, remove NullLogger

20a74c6

Spelling

683467d

Remove json stringify

690a341

remove json stringify

b8535cf

Remove {}

a1e7f62

plameniv approved these changes May 14, 2021

View reviewed changes

Shwetha Narayanan added 2 commits May 14, 2021 15:40

Add new lines

999df16

Prettier error

14a32ef

plameniv merged commit 66859a1 into walmartlabs:develop May 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Log out a detailed error msg while inserting data into big query #229

Log out a detailed error msg while inserting data into big query #229

snarayanan1095 commented May 13, 2021

CLAassistant commented May 13, 2021

Log out a detailed error msg while inserting data into big query #229

Log out a detailed error msg while inserting data into big query #229

Conversation

snarayanan1095 commented May 13, 2021

CLAassistant commented May 13, 2021