Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: transaction_ignore_urls wildcard matching #1870

Closed
wants to merge 11 commits into from

Conversation

astorm
Copy link
Contributor

@astorm astorm commented Nov 11, 2020

Will Fix: #1689

Implements ELASTIC_TRANSACTION_IGNORE_URL/transactionIgnoreUrls configuration for wildcard URL ignoring. The intent of this new configuration is to gives users the ability to skip generating transactions for a request to a URL that matches an exact string, or a URL that matches simple wildcard pattern. This wildcard pattern will behave consistently across language agents.

This PR leaves the existing ignoreUrls configuration in place to allow users access to more powerful regular expression matching and to preserve existing functionality for users using this older configuration setting.

This PR also introduces a new dependency, escape-string-regexp. In order to implement the suggested wildcard matching algorithm we needed the ability to escape a string for use in a regular expression. Since Node.js doesn't have this functionality natively, I pulled in escape-string-regexp rather than attempt to write (easy) and maintain (harder) this functionality ourselves. Happy to revisit this decision if we'd rather implement this functionality ourselves.

Open items for consideration/completion

  • is escape-string-regexp an acceptable dependency for ignore regular expressions?
  • Docs
  • Fixture downloading
  • central config handling? How does this work in Node Agent, do we need more here?
  • should leading / be assumed? (we'll prepend this, meaning a configured value of 'foo*' and '/foo*' will match the same)
  • need clarity on (?-i) and (?+i) prefixes across agents
  • Does this apply only to transactions? (yes, isRequestBlacklisted is only called in instrumentRequest, and instrumentRequest is only used when wrapping http(s).Server's emit method (i.e. it's used in transaction starting)

Checklist

  • Implement code
  • Add tests
  • Update TypeScript typings
  • Update documentation
  • Commit message follows commit guidelines

@astorm astorm marked this pull request as draft November 11, 2020 23:39
@apmmachine
Copy link
Contributor

apmmachine commented Nov 11, 2020

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #1870 updated]

  • Start Time: 2020-11-17T00:32:33.855+0000

  • Duration: 13 min 44 sec

Test stats 🧪

Test Results
Failed 0
Passed 16398
Skipped 0
Total 16398

@astorm astorm changed the title feat: transaction_ignore_url wildcard matching feat: transaction_ignore_urls wildcard matching Nov 12, 2020
@astorm astorm force-pushed the astorm/transaction-ignore-url-wildcard branch from f248eff to 1f14816 Compare November 13, 2020 05:24
@trentm trentm added the agent-nodejs Make available for APM Agents project planning. label Nov 16, 2020
Alan Storm and others added 10 commits November 16, 2020 12:26
This prefers package.json#files (explicitly listing files to include) over
.npmignore (trying to list all files to not include) for controlling files
that are included in npm publish. The latter can cause a surprise if
.gitignore is updated and .npmignore accidentally isn't.
In elastic/apm-server#4393 apm-server updated to
new and generate JSON schema. This commit updates the tests to get the
schema files from the new location and switches from the
"is-my-json-valid" JSON Schema validator to the more capable "ajv"
validator. This also changes the schema file cache dir so developers
need not manually delete the old cache.

Fixes #1867
Co-authored-by: apmmachine <infra-root-apmmachine@elastic.co>
Co-authored-by: Alan Storm <astorm@users.noreply.github.com>
@astorm astorm changed the base branch from master to 3.x November 17, 2020 00:43
@astorm astorm changed the base branch from 3.x to master November 17, 2020 00:43
@astorm astorm closed this Nov 17, 2020
@apmmachine
Copy link
Contributor

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Pull request #1870 event]

  • Reason: Aborted from #16

  • Start Time: 2020-11-17T00:43:47.072+0000

  • Duration: 3 min 10 sec

  • Commit: 8706e62

Log output

Expand to view the last 100 lines of log output

[2020-11-17T00:46:32.373Z]  ---> 032af6c8e162
[2020-11-17T00:46:32.373Z] Successfully built 032af6c8e162
[2020-11-17T00:46:32.373Z] Successfully tagged docker_node_tests:latest
[2020-11-17T00:46:32.373Z] Creating docker_mongodb_1 ... 
[2020-11-17T00:46:32.373Z] Creating docker_elasticsearch_1 ... 
[2020-11-17T00:46:32.374Z] Creating docker_postgres_1      ... 
[2020-11-17T00:46:32.374Z] Creating docker_mysql_1         ... 
[2020-11-17T00:46:32.374Z] Creating docker_mssql_1         ... 
[2020-11-17T00:46:32.374Z] Creating docker_redis_1         ... 
[2020-11-17T00:46:32.374Z] Creating docker_cassandra_1     ... 
[2020-11-17T00:46:32.374Z] Creating docker_memcached_1     ... 
[2020-11-17T00:46:34.929Z] Creating docker_mongodb_1       ... done
[2020-11-17T00:46:34.929Z] Creating docker_memcached_1     ... done
[2020-11-17T00:46:34.929Z] Creating docker_redis_1         ... done
[2020-11-17T00:46:35.190Z] Creating docker_elasticsearch_1 ... done
[2020-11-17T00:46:35.782Z] Creating docker_mssql_1         ... done
[2020-11-17T00:46:35.782Z] Creating docker_cassandra_1     ... done
[2020-11-17T00:46:35.782Z] Creating docker_mysql_1         ... done
[2020-11-17T00:46:36.741Z] Creating docker_postgres_1      ... done
[2020-11-17T00:46:43.632Z] Still waiting to schedule task
[2020-11-17T00:46:43.632Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.634Z] Still waiting to schedule task
[2020-11-17T00:46:43.634Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.635Z] Still waiting to schedule task
[2020-11-17T00:46:43.636Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.637Z] Still waiting to schedule task
[2020-11-17T00:46:43.637Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.639Z] Still waiting to schedule task
[2020-11-17T00:46:43.639Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.641Z] Still waiting to schedule task
[2020-11-17T00:46:43.641Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.643Z] Still waiting to schedule task
[2020-11-17T00:46:43.643Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.645Z] Still waiting to schedule task
[2020-11-17T00:46:43.645Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.647Z] Still waiting to schedule task
[2020-11-17T00:46:43.647Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.649Z] Still waiting to schedule task
[2020-11-17T00:46:43.649Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.651Z] Still waiting to schedule task
[2020-11-17T00:46:43.651Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.653Z] Still waiting to schedule task
[2020-11-17T00:46:43.653Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.654Z] Still waiting to schedule task
[2020-11-17T00:46:43.654Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.656Z] Still waiting to schedule task
[2020-11-17T00:46:43.656Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.658Z] Still waiting to schedule task
[2020-11-17T00:46:43.658Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.660Z] Still waiting to schedule task
[2020-11-17T00:46:43.660Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.662Z] Still waiting to schedule task
[2020-11-17T00:46:43.662Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:43.664Z] Still waiting to schedule task
[2020-11-17T00:46:43.664Z] All nodes of label ‘linux&&immutable’ are offline
[2020-11-17T00:46:51.203Z] Sending interrupt signal to process
[2020-11-17T00:46:52.464Z] Failed in branch Node.js-14
[2020-11-17T00:46:52.465Z] Failed in branch Node.js-14-async-hooks-false
[2020-11-17T00:46:52.466Z] Failed in branch Node.js-14.0
[2020-11-17T00:46:52.466Z] Failed in branch Node.js-14.0-async-hooks-false
[2020-11-17T00:46:52.467Z] Failed in branch Node.js-13
[2020-11-17T00:46:52.467Z] Failed in branch Node.js-13-async-hooks-false
[2020-11-17T00:46:52.468Z] Failed in branch Node.js-12
[2020-11-17T00:46:52.469Z] Failed in branch Node.js-12-async-hooks-false
[2020-11-17T00:46:52.469Z] Failed in branch Node.js-12.0
[2020-11-17T00:46:52.470Z] Failed in branch Node.js-12.0-async-hooks-false
[2020-11-17T00:46:52.470Z] Failed in branch Node.js-10
[2020-11-17T00:46:52.471Z] Failed in branch Node.js-10-async-hooks-false
[2020-11-17T00:46:52.471Z] Failed in branch Node.js-10.0
[2020-11-17T00:46:52.472Z] Failed in branch Node.js-10.0-async-hooks-false
[2020-11-17T00:46:52.472Z] Failed in branch Node.js-8
[2020-11-17T00:46:52.473Z] Failed in branch Node.js-8-async-hooks-false
[2020-11-17T00:46:52.474Z] Failed in branch Node.js-8.6
[2020-11-17T00:46:52.474Z] Failed in branch Node.js-8.6-async-hooks-false
[2020-11-17T00:46:54.940Z] .ci/scripts/windows/prepare-test.sh: line 16:  2816 Terminated              NODE_VERSION=${NODE_VERSION} USER_ID="$(id -u):$(id -g)" docker-compose --no-ansi --log-level ERROR -f .ci/docker/docker-compose-all.yml up --build --remove-orphans --quiet-pull --detach
[2020-11-17T00:46:54.948Z] script returned exit code 143
[2020-11-17T00:46:54.950Z] Failed in branch Windows-Node.js-12
[2020-11-17T00:46:56.002Z] Stage "TAV Test" skipped due to earlier failure(s)
[2020-11-17T00:46:56.033Z] Stage "Edge Test" skipped due to earlier failure(s)
[2020-11-17T00:46:56.066Z] Stage "Nightly Test" skipped due to earlier failure(s)
[2020-11-17T00:46:56.067Z] Stage "Nightly Test - No async hooks" skipped due to earlier failure(s)
[2020-11-17T00:46:56.068Z] Stage "RC Test" skipped due to earlier failure(s)
[2020-11-17T00:46:56.068Z] Stage "RC Test - No async hooks" skipped due to earlier failure(s)
[2020-11-17T00:46:56.109Z] Failed in branch Nightly Test
[2020-11-17T00:46:56.110Z] Failed in branch Nightly Test - No async hooks
[2020-11-17T00:46:56.110Z] Failed in branch RC Test
[2020-11-17T00:46:56.111Z] Failed in branch RC Test - No async hooks
[2020-11-17T00:46:56.156Z] Stage "Integration Tests" skipped due to earlier failure(s)
[2020-11-17T00:46:56.183Z] Stage "Release" skipped due to earlier failure(s)
[2020-11-17T00:46:56.197Z] Stage "Release" skipped due to earlier failure(s)
[2020-11-17T00:46:56.241Z] Stage "Benchmarks" skipped due to earlier failure(s)
[2020-11-17T00:46:56.430Z] Running on worker-1095690 in /var/lib/jenkins/workspace/ejs_apm-agent-nodejs-mbp_PR-1870
[2020-11-17T00:46:56.509Z] [INFO] getVaultSecret: Getting secrets
[2020-11-17T00:46:56.645Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-11-17T00:46:58.744Z] + chmod 755 generate-build-data.sh
[2020-11-17T00:46:58.744Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-nodejs/apm-agent-nodejs-mbp/PR-1870/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-nodejs/apm-agent-nodejs-mbp/PR-1870/runs/15 ABORTED 190288
[2020-11-17T00:46:58.744Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-nodejs/apm-agent-nodejs-mbp/PR-1870/runs/15/steps/?limit=10000 -o steps-info.json
[2020-11-17T00:46:59.442Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-nodejs/apm-agent-nodejs-mbp/PR-1870/runs/15/tests/?status=FAILED -o tests-errors.json
[2020-11-17T00:47:00.140Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2020-11-17T00:47:01.587Z] Retry 2/3 exited 22, retrying in 2 seconds...

@zube zube bot removed the [zube]: Done label Feb 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
agent-nodejs Make available for APM Agents project planning.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement transaction_ignore_urls
3 participants