Skip to content
This repository has been archived by the owner on Dec 17, 2021. It is now read-only.

feat: force inventory refresh #149

Merged
merged 7 commits into from
Oct 15, 2021
Merged

feat: force inventory refresh #149

merged 7 commits into from
Oct 15, 2021

Conversation

weliasz
Copy link
Contributor

@weliasz weliasz commented Oct 13, 2021

Description

Previously we used UNIX touch command to force refresh of inventory but as we have readonly filesystem on Kubernetes I redesigned this feature

Type of change

Please delete options that are not relevant.

  • Refactor/improvement

How Has This Been Tested?

I placed proper log entry and by updating record in Mongo I tricked our system to issue WALK command and inventory refresh. One need to put bigger value than existing in mongo for sysUpTime to trigger rewalk.

Checklist

  • My commit message is conventional
  • I have run pre-commit on all files before creating the PR
  • I have commented my code, particularly in hard-to-understand areas
  • New and existing unit tests pass locally with my changes
  • I have checked my code and corrected any misspellings

@@ -88,7 +92,9 @@ def __check_inventory(self):
)

# update job when either inventory changes or config changes
if server_config_modified or inventory_config_modified:
if server_config_modified or inventory_config_modified or self._force_refresh:
logger.info("Refreshing inventory and config")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe add force_refresh state to that log to have more information

@@ -56,6 +56,10 @@ def __init__(self, args, server_config):
self._local_snmp_engine = SnmpEngine()
self._unmatched_devices = {}
self._lock = threading.Lock()
self._force_refresh = False

def force_inventory_refresh(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we change it and request in a few different scheduled functions, can't we run into any race condition?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Boolean assignment is considered threadsafe in Python.
self._force_refresh = False - is the first statement in refreshing fragment of code so if other thread sets the flag to true it will be picked up during next iteration

refresh_inventory,
inventory_file_path,
)
if not initial_walk:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we schedule automatic_realtime_task with False in __start_realtime_scheduler_task
wouldn't we schedule new jobs every time it will run?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The feature of forcing inventory refreshing is supposed to be called when new walk for device was conducted and in case sysDescr has changed we would like run the matching again. When we start the application automatic_realtime_task is called to schedule first walk for device and there is no need to force inventory reloading.

@codecov-commenter
Copy link

codecov-commenter commented Oct 13, 2021

Codecov Report

Merging #149 (3cc0021) into develop (8dd65e4) will decrease coverage by 0.27%.
The diff coverage is 18.18%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #149      +/-   ##
===========================================
- Coverage    46.89%   46.61%   -0.28%     
===========================================
  Files           22       22              
  Lines         1190     1195       +5     
===========================================
- Hits           558      557       -1     
- Misses         632      638       +6     
Impacted Files Coverage Δ
splunk_connect_for_snmp_poller/manager/poller.py 0.00% <0.00%> (ø)
...onnect_for_snmp_poller/manager/poller_utilities.py 40.90% <40.00%> (-1.14%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8dd65e4...3cc0021. Read the comment docs.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 14, 2021

Expected release notes (by @weliasz)

features:
force inventory refresh (d09e65b)
add additionalvarbinds to enricher (#126) (98aeaf3)
enrich data with linked tables (#101) (75bdb13)
extract enricher data from config (#97) (123864f)
extract IF-MIB data from an SNMPWALK (#96) (7699591)
support polling the same IP address using multiple profiles (#92) (a6320ca)

fixes:
changed order of validation (#147) (8dd65e4)
build fix (8726135)
fetch frequency only when required (764ce07)
PR comments (0b7a0fa)
PR comments (bed9a60)
build fix (db1a90a)
add some restrictions for getting profiles from mib server (#145) (3c690a5)
add missing "break" (#144) (9b728d6)
error with no snmp received before timeout (#143) (b50480f)
'get_snmp_data() missing 1 required positional argument: 'additional_metric_fields'' and change error logger to debug (494ecbb)
change github.repository_owner to github.actor (#129) (89a3fb5)
add toJson() method to onetime_task (#127) (04b16f7)
update splunk indexes with meta_index (#116) (6d3f1bc)
changed indexes (#114) (9644663)
change way of checking if profile is an OID to use regex (#112) (277a050)
mongo real time data check (#107) (4526b0c)
change the way of creating host name which is an id in the database (#104) (bbf2733)
forced commit for creating new release (f3081c1)

others (will not be included in Semantic-Release notes):
bf2293d Feature/dynamic profile (#138)
adding pull request template (7314352)
better exception handling for mib request (a41fab9)
1.5.0-develop.9 (51d7678)
5a8a418 added 2 missing variables
0c2faf1 encapsulated hec sending request + test + correct pycov coverage report
803ff1a added sending error messages to hec isntead of ignoring them
remove local actionlint (3cc058e)
8ea42f0 added missing error messages
c23399b fixing logging level
1.5.0-develop.8 (8e7cc3d)
change image path (7167c4c)
edf4fc9 adding frequency to all metric data and profile if it is set in conf
4dd2c28 Feat/async rework (#122)
disable shellcheck SC1090 (734d523)
add actionlint as pre-commit step (52ac6f8)
allow dependabot[bot] and remove custom allow policy (437745f)
pre-commit autoupdate [ci skip] (e0544a4)
7ef1a09 refactor
remove black from dev dependency (1b07dd8)
add pre-commit step and remove lint step (9ef6611)
update severity threshold for snyk to "high" (#113) (7540a5b)
pre-commit autoupdate [ci skip] (6ff7b00)
pre-commit and debug statements [ci skip] (bb5e0ff)
set threshold for jscdp linter (#105) (c01fb4d)
1.5.0-develop.3 (7306e5d)
fix of duplicated code linter problem (c855088)
fix shellcheck (2fa415a)
apply mypy and flake8 (ec32122)
apply pre-commit isort (9c906f9)
exclude json files from copyright check (f3d6b2d)
update README to include pre-commit (cd6a9eb)
apply pre-commit black (c53b8b6)
pre-commit (9a24536)
33acbbf feat: Replace multiple SNMP walks with one SNMP Bulk (#100)
ebe3ebf Feature/redo walk (#95)
d5879c5 Stubs for lint
9cff184 Stubs for lint
adddbea Stubs for lint
5dc1549 Stubs for lint
2c5a5c3 Feat/multiple profiles2 (#93)

  • I hereby acknowledge these release notes are 🥙 AWESOME 🥙

@weliasz weliasz requested a review from okuzhel October 14, 2021 10:42
@weliasz weliasz merged commit 8026248 into develop Oct 15, 2021
@weliasz weliasz deleted the feat/force_refresh branch October 15, 2021 07:53
@github-actions github-actions bot locked and limited conversation to collaborators Oct 15, 2021
@ryanfaircloth
Copy link
Contributor

🎉 This PR is included in version 1.5.0-develop.16 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants