Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move Kerberos FAST config flag to shared kerberos config #26141

Merged
merged 18 commits into from
Jun 16, 2021

Conversation

faec
Copy link
Contributor

@faec faec commented Jun 4, 2021

What does this PR do?

This PR moves the enable_krb5_fast flag, which controls Kerberos FAST authentication, from the Kafka output configuration to the shared Kerberos configuration used by both the Kafka input and output, and applies the flag in the Kafka input. FAST authentication doesn't work with Active Directory, so this flag is disabled by default. We expect that this will address some authentication problems we've seen in the Kafka input.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@faec faec added bug enhancement Team:Elastic-Agent Label for the Agent team labels Jun 4, 2021
@faec faec requested a review from kvch June 4, 2021 01:28
@faec faec requested review from a team as code owners June 4, 2021 01:28
@elasticmachine
Copy link
Collaborator

Pinging @elastic/agent (Team:Agent)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 4, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 4, 2021

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #26141 updated

  • Start Time: 2021-06-15T20:56:19.689+0000

  • Duration: 91 min 46 sec

  • Commit: a56f87a

Test stats 🧪

Test Results
Failed 1
Passed 47520
Skipped 5237
Total 52758

Trends 🧪

Image of Build Times

Image of Tests

Test errors 1

Expand to view the tests failures

Extended / heartbeat-windows-2008-windows-2008-r2 / TestQueueRunsInOrder – github.com/elastic/beats/v7/heartbeat/scheduler/timerqueue
    Expand to view the error details

     Failed 
    

    Expand to view the stacktrace

     === RUN   TestQueueRunsInOrder
    coverage: 81.4% of statements
    panic: test timed out after 10m0s
    
    goroutine 50 [running]:
    testing.(*M).startAlarm.func1()
    	C:/Users/jenkins/workspace/PR-26141-13-2dbee845-6923-482a-8ef9-484a65617a9f/.gvm/versions/go1.16.5.windows.amd64/src/testing/testing.go:1700 +0xe6
    created by time.goFunc
    	C:/Users/jenkins/workspace/PR-26141-13-2dbee845-6923-482a-8ef9-484a65617a9f/.gvm/versions/go1.16.5.windows.amd64/src/time/sleep.go:180 +0x4b
    
    goroutine 1 [chan receive]:
    testing.(*T).Run(0xc00003b080, 0x5fe96d, 0x14, 0x60dba0, 0x49592d)
    	C:/Users/jenkins/workspace/PR-26141-13-2dbee845-6923-482a-8ef9-484a65617a9f/.gvm/versions/go1.16.5.windows.amd64/src/testing/testing.go:1239 +0x2da
    testing.runTests.func1(0xc00003af00)
    	C:/Users/jenkins/workspace/PR-26141-13-2dbee845-6923-482a-8ef9-484a65617a9f/.gvm/versions/go1.16.5.windows.amd64/src/testing/testing.go:1511 +0x7f
    testing.tRunner(0xc00003af00, 0xc00010fda8)
    	C:/Users/jenkins/workspace/PR-26141-13-2dbee845-6923-482a-8ef9-484a65617a9f/.gvm/versions/go1.16.5.windows.amd64/src/testing/testing.go:1193 +0xef
    testing.runTests(0xc000004150, 0x750ec0, 0x2, 0x2, 0xc02a67cb6b0d22f0, 0x8bb2f62425, 0x75aa20, 0x5fc381)
    	C:/Users/jenkins/workspace/PR-26141-13-2dbee845-6923-482a-8ef9-484a65617a9f/.gvm/versions/go1.16.5.windows.amd64/src/testing/testing.go:1509 +0x310
    testing.(*M).Run(0xc00007e100, 0x0)
    	C:/Users/jenkins/workspace/PR-26141-13-2dbee845-6923-482a-8ef9-484a65617a9f/.gvm/versions/go1.16.5.windows.amd64/src/testing/testing.go:1417 +0x1f5
    main.main()
    	_testmain.go:97 +0x1c8
    
    goroutine 6 [chan receive]:
    github.com/elastic/beats/v7/heartbeat/scheduler/timerqueue.testQueueRunsInOrderOnce(0xc00003b080)
    	C:/Users/jenkins/workspace/PR-26141-13-2dbee845-6923-482a-8ef9-484a65617a9f/src/github.com/elastic/beats/heartbeat/scheduler/timerqueue/queue_test.go:79 +0x238
    github.com/elastic/beats/v7/heartbeat/scheduler/timerqueue.TestQueueRunsInOrder(0xc00003b080)
    	C:/Users/jenkins/workspace/PR-26141-13-2dbee845-6923-482a-8ef9-484a65617a9f/src/github.com/elastic/beats/heartbeat/scheduler/timerqueue/queue_test.go:33 +0x3b
    testing.tRunner(0xc00003b080, 0x60dba0)
    	C:/Users/jenkins/workspace/PR-26141-13-2dbee845-6923-482a-8ef9-484a65617a9f/.gvm/versions/go1.16.5.windows.amd64/src/testing/testing.go:1193 +0xef
    created by testing.(*T).Run
    	C:/Users/jenkins/workspace/PR-26141-13-2dbee845-6923-482a-8ef9-484a65617a9f/.gvm/versions/go1.16.5.windows.amd64/src/testing/testing.go:1238 +0x2b3
    
    goroutine 53 [runnable]:
    github.com/elastic/beats/v7/heartbeat/scheduler/timerqueue.(*TimerQueue).Start.func1(0xc0000a6000)
    	C:/Users/jenkins/workspace/PR-26141-13-2dbee845-6923-482a-8ef9-484a65617a9f/src/github.com/elastic/beats/heartbeat/scheduler/timerqueue/queue.go:72
    created by github.com/elastic/beats/v7/heartbeat/scheduler/timerqueue.(*TimerQueue).Start
    	C:/Users/jenkins/workspace/PR-26141-13-2dbee845-6923-482a-8ef9-484a65617a9f/src/github.com/elastic/beats/heartbeat/scheduler/timerqueue/queue.go:72 +0x56
     
    

Steps errors 3

Expand to view the steps failures

heartbeat-windows-2008-windows-2008-r2 - mage build unitTest
  • Took 11 min 15 sec . View more details on here
  • Description: mage build unitTest
gsutil -m -q cp -a public-read test-build-artifacts-heartbeat-windows-2008-windows-2008-r2-tgz gs://
  • Took 0 min 2 sec . View more details on here
  • Description: @echo off gsutil -m -q cp -a public-read test-build-artifacts-heartbeat-windows-2008-windows-2008-r2-tgz gs://beats-ci-temp/Beats/beats/PR-26141-13
Error signal
  • Took 0 min 0 sec . View more details on here
  • Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-06-15T22:24:55.046Z] module\zookeeper\test_zookeeper.py ss                                    [100%]
[2021-06-15T22:24:55.046Z] 
[2021-06-15T22:24:55.046Z] ============================== warnings summary ===============================
[2021-06-15T22:24:55.046Z] tests\system\test_lightmodules.py:57
[2021-06-15T22:24:55.046Z]   C:\Users\jenkins\workspace\PR-26141-13-22a5d72f-67cc-448e-b148-a872b1bf9695\src\github.com\elastic\beats\metricbeat\tests\system\test_lightmodules.py:57: PytestCollectionWarning: cannot collect test class 'TestHTTPHandler' because it has a __init__ constructor (from: metricbeat/tests/system/test_lightmodules.py)
[2021-06-15T22:24:55.046Z]     class TestHTTPHandler(http.server.BaseHTTPRequestHandler):
[2021-06-15T22:24:55.046Z] 
[2021-06-15T22:24:55.046Z] -- Docs: https://docs.pytest.org/en/stable/warnings.html
[2021-06-15T22:24:55.046Z] - generated xml file: C:\Users\jenkins\workspace\PR-26141-13-22a5d72f-67cc-448e-b148-a872b1bf9695\src\github.com\elastic\beats\metricbeat\build\TEST-python-unit.xml -
[2021-06-15T22:24:55.046Z] ============================ slowest 20 durations =============================
[2021-06-15T22:24:55.046Z] 5.40s call     metricbeat/tests/system/test_reload.py::Test::test_start_stop
[2021-06-15T22:24:55.046Z] 4.70s call     metricbeat/tests/system/test_config.py::ConfigTest::test_service_name
[2021-06-15T22:24:55.047Z] 3.75s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_test_error
[2021-06-15T22:24:55.047Z] 3.60s call     metricbeat/tests/system/test_processors.py::Test::test_dropevent_with_condition
[2021-06-15T22:24:55.047Z] 2.76s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_disable
[2021-06-15T22:24:55.047Z] 2.74s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_enable
[2021-06-15T22:24:55.047Z] 2.73s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_list
[2021-06-15T22:24:55.047Z] 2.52s call     metricbeat/module/http/test_http.py::Test::test_server
[2021-06-15T22:24:55.047Z] 2.39s call     metricbeat/tests/system/test_reload.py::Test::test_reload
[2021-06-15T22:24:55.047Z] 2.19s call     metricbeat/module/system/test_system.py::Test::test_process
[2021-06-15T22:24:55.047Z] 2.08s call     metricbeat/tests/system/test_lightmodules.py::Test::test_processors
[2021-06-15T22:24:55.047Z] 1.90s call     metricbeat/tests/system/test_timeseries.py::TestTimeseries::test_enable_timeseries
[2021-06-15T22:24:55.047Z] 1.83s call     metricbeat/tests/system/test_base.py::Test::test_export_index_pattern_migration
[2021-06-15T22:24:55.047Z] 1.81s call     metricbeat/tests/system/test_base.py::Test::test_export_index_pattern
[2021-06-15T22:24:55.047Z] 1.70s call     metricbeat/tests/system/test_base.py::Test::test_export_template
[2021-06-15T22:24:55.047Z] 1.64s call     metricbeat/module/system/test_system.py::Test::test_network
[2021-06-15T22:24:55.047Z] 1.62s call     metricbeat/module/system/test_system.py::Test::test_core
[2021-06-15T22:24:55.047Z] 1.62s call     metricbeat/module/golang/test_golang.py::Test::test_stats
[2021-06-15T22:24:55.047Z] 1.62s call     metricbeat/module/system/test_system.py::Test::test_socket_summary
[2021-06-15T22:24:55.047Z] 1.61s call     metricbeat/module/system/test_system.py::Test::test_core_with_cpu_ticks
[2021-06-15T22:24:55.047Z] ============ 42 passed, 204 skipped, 1 warning in 86.60s (0:01:26) ============
[2021-06-15T22:24:55.047Z] >> python test: Unit Testing Complete
[2021-06-15T22:24:55.391Z] 
[2021-06-15T22:24:55.391Z] C:\Users\jenkins\workspace\PR-26141-13-22a5d72f-67cc-448e-b148-a872b1bf9695\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-06-15T22:24:56.261Z] 
[2021-06-15T22:24:56.261Z] C:\Users\jenkins\workspace\PR-26141-13-22a5d72f-67cc-448e-b148-a872b1bf9695\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-06-15T22:24:56.830Z] Copy .\metricbeat\build into build\metricbeat\build
[2021-06-15T22:24:56.831Z] Copy .\metricbeat\null\build into build\metricbeat\null\build
[2021-06-15T22:24:56.848Z] Running in C:\Users\jenkins\workspace\PR-26141-13-22a5d72f-67cc-448e-b148-a872b1bf9695\src\github.com\elastic\beats\build
[2021-06-15T22:24:56.870Z] Recording test results
[2021-06-15T22:25:00.036Z] [Checks API] No suitable checks publisher found.
[2021-06-15T22:25:00.404Z] 
[2021-06-15T22:25:00.404Z] C:\Users\jenkins\workspace\PR-26141-13-22a5d72f-67cc-448e-b148-a872b1bf9695\src\github.com\elastic\beats>go clean -modcache 
[2021-06-15T22:25:44.481Z] warn: failed to upgrade pip (ignoring): running "null\build\ve\windows\Scripts\pip install -U pip" failed with exit code 1============================= test session starts =============================
[2021-06-15T22:25:44.481Z] platform win32 -- Python 3.8.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
[2021-06-15T22:25:44.481Z] rootdir: C:\Users\jenkins\workspace\PR-26141-13-3cc528ab-0eae-40e3-96a6-3f5524c969e3\src\github.com\elastic\beats, configfile: pytest.ini
[2021-06-15T22:25:44.481Z] plugins: rerunfailures-9.1.1, timeout-1.4.2
[2021-06-15T22:25:44.481Z] timeout: 90.0s
[2021-06-15T22:25:44.481Z] timeout method: thread
[2021-06-15T22:25:44.481Z] timeout func_only: True
[2021-06-15T22:25:44.481Z] collected 13 items
[2021-06-15T22:25:44.481Z] 
[2021-06-15T22:25:44.481Z] tests\system\test_base.py s......s                                       [ 61%]
[2021-06-15T22:25:44.481Z] tests\system\test_file_integrity.py .s                                   [ 76%]
[2021-06-15T22:25:44.481Z] tests\system\test_show_command.py sss                                    [100%]
[2021-06-15T22:25:44.481Z] 
[2021-06-15T22:25:44.481Z] - generated xml file: C:\Users\jenkins\workspace\PR-26141-13-3cc528ab-0eae-40e3-96a6-3f5524c969e3\src\github.com\elastic\beats\auditbeat\build\TEST-python-unit.xml -
[2021-06-15T22:25:44.481Z] ============================ slowest 20 durations =============================
[2021-06-15T22:25:44.481Z] 1.43s call     auditbeat/tests/system/test_file_integrity.py::Test::test_non_recursive
[2021-06-15T22:25:44.481Z] 0.39s call     auditbeat/tests/system/test_base.py::Test::test_export_index_pattern
[2021-06-15T22:25:44.481Z] 0.38s call     auditbeat/tests/system/test_base.py::Test::test_export_index_pattern_migration
[2021-06-15T22:25:44.481Z] 0.36s call     auditbeat/tests/system/test_base.py::Test::test_export_config
[2021-06-15T22:25:44.481Z] 0.33s call     auditbeat/tests/system/test_base.py::Test::test_export_template
[2021-06-15T22:25:44.481Z] 0.29s call     auditbeat/tests/system/test_base.py::Test::test_start_stop
[2021-06-15T22:25:44.481Z] 0.24s call     auditbeat/tests/system/test_base.py::Test::test_export_ilm_policy
[2021-06-15T22:25:44.481Z] 
[2021-06-15T22:25:44.481Z] (13 durations < 0.005s hidden.  Use -vv to show these durations.)
[2021-06-15T22:25:44.481Z] ======================== 7 passed, 6 skipped in 4.00s =========================
[2021-06-15T22:25:44.481Z] >> python test: Unit Testing Complete
[2021-06-15T22:25:44.829Z] 
[2021-06-15T22:25:44.830Z] C:\Users\jenkins\workspace\PR-26141-13-3cc528ab-0eae-40e3-96a6-3f5524c969e3\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-06-15T22:25:47.040Z] 
[2021-06-15T22:25:47.040Z] C:\Users\jenkins\workspace\PR-26141-13-3cc528ab-0eae-40e3-96a6-3f5524c969e3\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-06-15T22:25:47.604Z] Copy .\auditbeat\build into build\auditbeat\build
[2021-06-15T22:25:47.604Z] Copy .\auditbeat\null\build into build\auditbeat\null\build
[2021-06-15T22:25:47.618Z] Running in C:\Users\jenkins\workspace\PR-26141-13-3cc528ab-0eae-40e3-96a6-3f5524c969e3\src\github.com\elastic\beats\build
[2021-06-15T22:25:47.636Z] Recording test results
[2021-06-15T22:25:50.424Z] [Checks API] No suitable checks publisher found.
[2021-06-15T22:25:50.840Z] 
[2021-06-15T22:25:50.840Z] C:\Users\jenkins\workspace\PR-26141-13-3cc528ab-0eae-40e3-96a6-3f5524c969e3\src\github.com\elastic\beats>go clean -modcache 
[2021-06-15T22:25:50.913Z] + gsutil --version
[2021-06-15T22:25:52.337Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-15T22:25:52.745Z] + gcloud auth activate-service-account --key-file ****
[2021-06-15T22:25:53.346Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-06-15T22:25:53.671Z] + gsutil -m -q cp -a public-read bWV0cmljYmVhdC13aW5kb3dzLTIwMTItd2luZG93cy0yMDEyLXIyYTU2Zjg3YTljOTAwMzU5YzU1ZjZmY2U3NzJkNzRiMzgyYjlmZTYyMA gs://beats-ci-temp/ci/cache/
[2021-06-15T22:27:00.073Z] + gsutil --version
[2021-06-15T22:27:01.490Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-15T22:27:01.806Z] + gcloud auth activate-service-account --key-file ****
[2021-06-15T22:27:02.375Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-06-15T22:27:02.947Z] + gsutil -m -q cp -a public-read YXVkaXRiZWF0LXdpbmRvd3MtMTAtd2luZG93cy0xMGE1NmY4N2E5YzkwMDM1OWM1NWY2ZmNlNzcyZDc0YjM4MmI5ZmU2MjA gs://beats-ci-temp/ci/cache/
[2021-06-15T22:27:04.459Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-06-15T22:27:04.500Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-06-15T22:27:04.568Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26141/src/github.com/elastic/beats
[2021-06-15T22:27:04.911Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-26141
[2021-06-15T22:27:04.963Z] [INFO] getVaultSecret: Getting secrets
[2021-06-15T22:27:05.007Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-15T22:27:05.776Z] + chmod 755 generate-build-data.sh
[2021-06-15T22:27:05.776Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26141/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26141/runs/13 FAILURE 5445826
[2021-06-15T22:27:05.776Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26141/runs/13/steps/?limit=10000 -o steps-info.json
[2021-06-15T22:27:17.874Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26141/runs/13/tests/?status=FAILED -o tests-errors.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test Results
Failed 1
Passed 47520
Skipped 5237
Total 52758

Genuine test errors 1

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

  • Name: Extended / heartbeat-windows-2008-windows-2008-r2 / TestQueueRunsInOrder – github.com/elastic/beats/v7/heartbeat/scheduler/timerqueue

@faec faec added the backport-v7.14.0 Automated backport with mergify label Jun 7, 2021
@ph ph assigned faec Jun 14, 2021
@faec faec merged commit 124a2c3 into elastic:master Jun 16, 2021
mergify bot pushed a commit that referenced this pull request Jun 16, 2021
@faec faec deleted the kafka-kerberos branch June 17, 2021 14:10
faec added a commit that referenced this pull request Jun 17, 2021
…6349)

(cherry picked from commit 124a2c3)

Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Jun 21, 2021
* master: (25 commits)
  Fix UBI source URL (elastic#26384)
  Skip test_rotating_file in osx and windows (elastic#26379)
  Remove outdated k8s manifests for managed elastic-agent (elastic#26368)
  Enable agent to send custom headers to kibana/ES (elastic#26275)
  [Automation] Update elastic stack version to 8.0.0-943ef2c0 for testing (elastic#26354)
  Make the Syslog input GA (elastic#26293)
  Move Kerberos FAST config flag to shared kerberos config (elastic#26141)
  Add k8s cluster identifiers (elastic#26056)
  Store message from MongoDB json logs in message field (elastic#26338)
  update threatintel ECS version (elastic#26274)
  update envoyproxy ECS version (elastic#26277)
  [Filebeat] [MongoDB] Support MongoDB 4.4 json logs (elastic#24774)
  Update go-structform to 0.0.9 (elastic#26251)
  Forward port 7.13.2 changelog to master (elastic#26323)
  Updated filter expression for filtering 86 artifacts (elastic#26313)
  Osquerybeat: Align with the rest of the beats, set the ECS version (elastic#26324)
  [Packetbeat] Add `url.extension` to Packetbeat HTTP events (elastic#25999)
  Change link to snapshots in README (elastic#26317)
  Don't include full ES index template in errors (elastic#25743)
  First refactor of the system module - system/cpu and system/core (elastic#25771)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v7.14.0 Automated backport with mergify bug enhancement Team:Elastic-Agent Label for the Agent team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants