Skip to content
This repository has been archived by the owner on Dec 17, 2021. It is now read-only.

fix: new execution of enricher #163

Merged
merged 28 commits into from
Oct 22, 2021

Conversation

omrozowicz-splunk
Copy link
Contributor

@omrozowicz-splunk omrozowicz-splunk commented Oct 21, 2021

Description

This fix change the way enricher is being executed. There are two scenarios:

  1. Any change was issued in IF-MIB oid family -> in this case walk for IF-MIBs is being executed, and the data is enriched the same way as it was working before
  2. There is a change only outside IF-MIB -> in this case only additional data was changed, so no need to trigger walk

Fixes # ADDON-43522

Type of change

Please delete options that are not relevant.

  • Bug fix

How Has This Been Tested?

I was testing many scenarios of deleting, adding, modifying and all of those at once on my local instance and I was checking mongodb contents every time. Also I installed this version on NOVA and modified enricher for a few times.

Checklist

  • My commit message is conventional
  • I have run pre-commit on all files before creating the PR
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have checked my code and corrected any misspellings

@codecov-commenter
Copy link

codecov-commenter commented Oct 21, 2021

Codecov Report

Merging #163 (67cf113) into develop (d69a9c7) will increase coverage by 3.43%.
The diff coverage is 54.16%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #163      +/-   ##
===========================================
+ Coverage    46.93%   50.36%   +3.43%     
===========================================
  Files           22       22              
  Lines         1287     1358      +71     
===========================================
+ Hits           604      684      +80     
+ Misses         683      674       -9     
Impacted Files Coverage Δ
splunk_connect_for_snmp_poller/mongo.py 32.94% <25.00%> (-1.31%) ⬇️
splunk_connect_for_snmp_poller/manager/poller.py 29.94% <45.00%> (+29.94%) ⬆️
...onnect_for_snmp_poller/manager/poller_utilities.py 46.37% <65.78%> (+7.37%) ⬆️
...t_for_snmp_poller/manager/realtime/oid_constant.py 100.00% <100.00%> (ø)
...plunk_connect_for_snmp_poller/manager/variables.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d69a9c7...67cf113. Read the comment docs.

@github-actions
Copy link
Contributor

Expected release notes (by @omrozowicz-splunk)

features:
issue a warning when assigned profile is missing (#162) (96b7f8d)
force inventory refresh (#149) (8026248)
add additionalvarbinds to enricher (#126) (98aeaf3)
enrich data with linked tables (#101) (75bdb13)
extract enricher data from config (#97) (123864f)
extract IF-MIB data from an SNMPWALK (#96) (7699591)
support polling the same IP address using multiple profiles (#92) (a6320ca)

fixes:
change way of requesting walk for IF-MIB (67cf113)
add saving InventoryRecord for every host in inventory (8f48ea4)
remove unnecessary boolean cast (8706317)
adjusting code to PR comments (22796f6)
refactor variable name for delete_ifmib decorator (2c9abc4)
delete unnecessary loggers (0c87e27)
refactor code (67f5f00)
fix unnecessary to_json() (703c2d3)
change sequence of enricher check (1fb7142)
change sequence of enricher check (a807341)
add logs and redesign function (24370e8)
add logs and redesign function (19fbc04)
change .items() to .keys() (48ac58d)
add loggers (8bd1c58)
add loggers (87494e3)
update enricher to trigger only in case of exisitng IF-MIB (9efe54c)
onetime walk retry (#158) (d69a9c7)
snmp_pooling function (e7055e3)
update OID (a8c674a)
add loggers (4c5ef6c)
refactor method to update enricher structure (f09c031)
add loggers (f26839a)
change way of adding additional varbinds (f87282b)
add loggers (fb299c6)
bring back extract_network_interface_data_from_additional_config to walk with enricher (c5c9868)
change variable of extract_network_interface_data_from_additional_config (9a415bd)
add delay tp snmp_polling (a841d03)
removal of unsafe eval call (#157) (843c26a)
bring back previous changes (#161) (0e87f17)
implement new way for enricher to work (8e24841)
onetime walk with enricher (#154) (11a42ca)
matching on both dysDescr and sysObjectId (#156) (15158e8)
default event formatting (#155) (ed96f44)
fetch frequency (#152) (4602304)
allow * in profile (#151) (880dbce)
changed order of validation (#147) (8dd65e4)
add some restrictions for getting profiles from mib server (#145) (3c690a5)
add missing "break" (#144) (9b728d6)
error with no snmp received before timeout (#143) (b50480f)
'get_snmp_data() missing 1 required positional argument: 'additional_metric_fields'' and change error logger to debug (494ecbb)
change github.repository_owner to github.actor (#129) (89a3fb5)
add toJson() method to onetime_task (#127) (04b16f7)
update splunk indexes with meta_index (#116) (6d3f1bc)
changed indexes (#114) (9644663)
change way of checking if profile is an OID to use regex (#112) (277a050)
mongo real time data check (#107) (4526b0c)
change the way of creating host name which is an id in the database (#104) (bbf2733)
forced commit for creating new release (f3081c1)

others (will not be included in Semantic-Release notes):
92e091e Fix/default freq dynamic (#160)
456f6db [ADDON-43460] retry and fail if mib unavailable
048ba1d [ADDON-43435] don't remove host if other record uses it
861a40e [ADDON-43362] special builder for error message
bf2293d Feature/dynamic profile (#138)
adding pull request template (7314352)
better exception handling for mib request (a41fab9)
1.5.0-develop.9 (51d7678)
5a8a418 added 2 missing variables
0c2faf1 encapsulated hec sending request + test + correct pycov coverage report
803ff1a added sending error messages to hec isntead of ignoring them
remove local actionlint (3cc058e)
8ea42f0 added missing error messages
c23399b fixing logging level
1.5.0-develop.8 (8e7cc3d)
change image path (7167c4c)
edf4fc9 adding frequency to all metric data and profile if it is set in conf
4dd2c28 Feat/async rework (#122)
disable shellcheck SC1090 (734d523)
add actionlint as pre-commit step (52ac6f8)
allow dependabot[bot] and remove custom allow policy (437745f)
pre-commit autoupdate [ci skip] (e0544a4)
7ef1a09 refactor
remove black from dev dependency (1b07dd8)
add pre-commit step and remove lint step (9ef6611)
update severity threshold for snyk to "high" (#113) (7540a5b)
pre-commit autoupdate [ci skip] (6ff7b00)
pre-commit and debug statements [ci skip] (bb5e0ff)
set threshold for jscdp linter (#105) (c01fb4d)
1.5.0-develop.3 (7306e5d)
fix of duplicated code linter problem (c855088)
fix shellcheck (2fa415a)
apply mypy and flake8 (ec32122)
apply pre-commit isort (9c906f9)
exclude json files from copyright check (f3d6b2d)
update README to include pre-commit (cd6a9eb)
apply pre-commit black (c53b8b6)
pre-commit (9a24536)
33acbbf feat: Replace multiple SNMP walks with one SNMP Bulk (#100)
ebe3ebf Feature/redo walk (#95)
d5879c5 Stubs for lint
9cff184 Stubs for lint
adddbea Stubs for lint
5dc1549 Stubs for lint
2c5a5c3 Feat/multiple profiles2 (#93)

  • I hereby acknowledge these release notes are 🥙 AWESOME 🥙

@omrozowicz-splunk omrozowicz-splunk merged commit b646e19 into develop Oct 22, 2021
@omrozowicz-splunk omrozowicz-splunk deleted the fix/new-execution-of-enricher branch October 22, 2021 06:49
@github-actions github-actions bot locked and limited conversation to collaborators Oct 22, 2021
@ryanfaircloth
Copy link
Contributor

🎉 This PR is included in version 1.5.0-develop.26 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants