-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
use ks2 as MC default #13131
use ks2 as MC default #13131
Conversation
@ktsaou i think the default algo should be still |
another random example video (6min) with what i mean |
@andrewm4894 we did some tests yesterday and
Because of these 2 points, we decided that |
yep it defo is not random and volume will for sure find useful stuff but ks2 is more general and gives us a better chance to find a wider range of changes because it works at the overall distribution of the data as opposed to just trying to boil it all down to one number which is what volume does. There is some more discussion of this in this video |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tested this PR locally and the algorithm was able to identify a anomaly while I was compiling a software during more than 10 minutes.
I asked @andrewm4894 about the Volume
method, because I only knew the KS2
method, considering what I heard from Andrew, I agree 100% that this PR has potential to improve our anomaly detection, because the basis of Volume
can suffer a strong bias when averages are calculated, on the other hand, the KS
compares deviation from a distribution (sample) . I also have the understand that KS
is used more with Normal distribution that is not our case unless we deal with a real large sample and this can affect the performance of the algorithm, but comparing the statistical basis that KS
has it is stronger than the information I received from Volume
. LGTM!
Thank you @andrewm4894 to bring one of the tools I used during my master degree! 😆
Watched both videos. I understand what you are saying @andrewm4894 . The volume calculation is polluted in an unpredictable way with the dimensions that were just zero in the baseline and non-zero in the highlighted window. The pollution is unpredictable because the score they get is just the average of the highlighted window. I tried setting it to a huge number (1 billion), but then all these dimensions were automatically selected (they were the only ones at the far left of the slider). So, I preferred to set it to the average of the highlighted window, that "blends" them into the other results. Keep in mind that we have the ability to apply any algorithm to calculate a single value for baseline and a single value for highlight. This means we could come up with an algorithm that combines average with min, max, standard deviation, etc. About the tests you do to verify they work, I would love to see some more focus examples. I did 2 tests myself:
Then I tried to find out which algorithm detected more relevant metrics. What I tried to do, is move the slider to the far left and examine the results. Then move the slider a little to the right and examine them again. The best algorithm is the one that can provide the best relative weights for the dimensions. In this test, I liked the fact that "volume" was producing more granular results (I could fine tune it to a great deal, compared to ks2). We could do more synthetic tests, like: a database stressing test, or a web server stressing test, which will influence CPU, memory, network and disk and see which algorithm identifies better the more important metrics. But the weight of the metrics and its granularity is important here. Anyway, I am ok to merge. Let's switch to ks2 and see the results in action. |
yeah was going to try think about if anything that can do this, its what ks2 does (indirectly by working on the CDF) but wonder if a way to just take the multiple numbers that describe the distribution avg, median, stdev, min, max, skew, kurtosis, iqr and compare them in some way that's useful and also even cheaper than ks2. Am going to have a look into that separately. Another option could be to combine the two in some way such that stuff they tend to agree on gets even more weight - would be easy enough in future to add a "combined" option perhaps that does something like this. Anyway - if i had to pick one algo to do next it would be PCA. PCA is in dlib so we probably just need to make the recipe (blog of how it would work) as opposed to the ingredients like we had to for ks2. This would probably also be fairly cheap (PCA should be well optimized in dlib based on fft tricks etc) and a useful addition since it operates in a completely different approach to both volume and ks2 by looking at the variances of the data and so should be good at finding other different "types" of changes - eg where something transitions from fairly stable to move flatlining - this would be a huge change in the variances of the data and so a PCA built on the baseline would have a high reconstruction error on the highlight window. Here is a PCA example using dlib from this book for reference. |
@andrewm4894 what if when the baseline is just zero, to score the highlighted window based on the amount of time it was non-zero? The result will always be 0-100%, but since the baseline window is zero, we ignore the value. We will be only interested for the amount of time the highlighted window was non-zero. To blend the results in a more meaningful way, we could also say that this 0-100% of the zero-baseline metrics will be proportionally mapped to the max ratio of the non-zero-baseline metrics. So, if the max for the non-zero-baseline is 500%, then 500% will also the max of the zero-baseline metrics. I think this will blend the zero-baselines metrics in order of importance among all the other volume rated metrics. |
I modified my comment above. Read it again if you have already read it. |
I think something like that could make sense for the edge cases in I think at moment a single little spike in either window when other is 0 gets almost treated the same maybe (since cant get a proper percentage difference) as a proper persistent "turning on" or "turning off" and so something like you suggest would handle that better. |
* Expose anomaly-bit option to health. (netdata#12835) * Consider ZFS ARC shrinkable as cache on FreeBSD (netdata#12879) * Update dashboard to version v2.25.2. (netdata#12884) Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> * [ci skip] Update changelog and version for nightly build: v1.34.0-148-nightly. * feat(proc/cgroups.plugin): add PSI stall time charts (netdata#12869) * Fix compilation warnings (netdata#12886) * chore(deps): bump docker/setup-buildx-action from 1 to 2 (netdata#12848) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 1 to 2. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](docker/setup-buildx-action@v1...v2) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump docker/setup-qemu-action from 1 to 2 (netdata#12849) Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 1 to 2. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](docker/setup-qemu-action@v1...v2) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump docker/login-action from 1 to 2 (netdata#12850) Bumps [docker/login-action](https://github.com/docker/login-action) from 1 to 2. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](docker/login-action@v1...v2) --- updated-dependencies: - dependency-name: docker/login-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump docker/build-push-action from 2 to 3 (netdata#12851) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 2 to 3. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](docker/build-push-action@v2...v3) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Pause alert pushes to the cloud (netdata#12852) * pause and unpause alert pushes to the cloud * move the check to when creating opcode * check for worker * remove previous checks for dbsync_workers. queue and clean aclk_alert tables even if no workers are up. Get wc then check before setting pause * remove sync_syncronize * remove sync_synchronize_2 * Take into account the in queue wait time when executing a data query (netdata#12885) Take into account the in queue wait time when executing a query with a timeout * [ci skip] Update changelog and version for nightly build: v1.34.0-157-nightly. * Return stable or nightly based on version if the file check fails (netdata#12894) * Implements new capability fields in aclk_schemas (netdata#12602) use new capability fields * Remove CPU-specific info from cpuidle dimensions (netdata#12898) * Update reconnect node with kickstart info (netdata#12891) * Update reconnect node with kickstart info * Split stop from start Second change done because restart may very easily result in the agent failing to connect because of problematic handling of reconnections. * fix(cgroups.plugin): use correct identifier when registering the main thread "chart" worker job (netdata#12902) use main thread chart job registration * chore(worker_utilization): log an error when re-registering an already registered job (netdata#12903) * [ci skip] Update changelog and version for nightly build: v1.34.0-164-nightly. * Fix release channel in the node info message (netdata#12905) Fix release channel in the node info message (was hardcoded) * Fix compilation warnings in FreeBSD (netdata#12887) * [ci skip] Update changelog and version for nightly build: v1.34.0-167-nightly. * added worker jobs for cgroup-rename, cgroup-network and cgroup-first-time (netdata#12910) * chore(streaming): bump default "buffer size bytes" to 10MB (netdata#12913) * [ci skip] Update changelog and version for nightly build: v1.34.0-170-nightly. * fix(cgroups.plugin): improve check for uninitialized containers in k8s (netdata#12912) * fix `[global statistics]` section in netdata.conf (netdata#12916) * fix virtualization detection when `systemd-detect-virt` is not available (netdata#12911) * Fix the log entry for incoming cloud start streaming commands (netdata#12908) Add the correct requested chart sequence id from the cloud and also record the local one we have * user configurable sqlite PRAGMAs (netdata#12917) * user configurable sqlite PRAGMAs * added cache size * chore(cgroups.plugin): remove "enable new cgroups detected at run time" config option (netdata#12906) * Restore a broken symbolic link (netdata#12923) * chore(fping.plugin): bump default fping version to 5.1 (netdata#12930) * chore: add links to SQLite init options in the src code (netdata#12920) * collectors: apps.plugin: apps_groups: update net, aws, ha groups (netdata#12921) * [ci skip] Update changelog and version for nightly build: v1.34.0-181-nightly. * Adjust alarms count (netdata#12896) * check for is_available when counting * remove empty line * Remove Alpine 3.12 from CI. (netdata#12919) It went EOL upstream on 2022-05-01. * Adjust the dimension liveness status check (netdata#12933) * Mark a chart to be exposed only if dimension is created or metadata changes * Add a calculate liveness for the dimension for collected to non collected (live -> stale) and vice versa * queue_dimension_to_aclk will have the rrdset and either 0 or last collected time If 0 then it will be marked as live else it will be marked as stale and last collected time will be sent to the cloud * Add an extra parameter to indicate if the payload check should be done in the database or it has been done already * Queue dimension sets dimension liveness and queues the exact payload to store in the database * Fix compilation error when --disable-cloud is specified * Reduce timeout to 1 second for getting cloud instance info (netdata#12941) * feat: move dirs, logs, and env vars config options to separate sections (netdata#12935) * include proper package dependency (netdata#12518) * Add support for installing static builds on systems without usable internet connections. (netdata#12809) * Add code to prepare an offline installation source directory. * Add support to `download` function for `file://` URLs. This will be used to reference local files instead of having to set up a web server or complicate the code for fetching files. * Modify static archive URL handling to use offline install source if present. * Make check for offline install source support more robust. * Fix fetching of files for offline installs. * Fixed installer stub script. * Mark release channel in offline install source directory. To ensure that users are getting what they expect. * Fix handling of option ordering. * Assorted fixes. * Don’t enable auto-updates by defualt for offline installs. * Update documentation. * Add dry run mode support. * Fix checking of SHA256 checksums. * Fix handling of custom install version selection. * Fix CPU architecture selection. * Fix typo. * Fix typos in user-facing messages * Fix broken merge. * Fix incorrect variable name. * Move kickstart argument parsing code to a function. (netdata#12805) Move kickstart argument arsing code to a function. This will make it slightly more maintainable in the long term, and slightly easier for newcomers to understand. * Fix and improve netdata-updater.sh script (netdata#12757) * minor improvement in compatible DISTRO detection * revert distro detection code; added almalinux and rocky distros as centos compatibility distro * [ci skip] Update changelog and version for nightly build: v1.34.0-191-nightly. * Autodetect channel for specific version (netdata#12856) * autodetect channel * fix kickstart.sh * Prevent command_to_be_logged from overflowing (netdata#12947) * prevent command_to_be_logged from overflowing * dont access with size * Update libbpf version (netdata#12945) * detailed dbengine stats (netdata#12948) * Optimize the dimensions option store to the metadata database (netdata#12952) * Add a flag to "cache" the latest hidden status written in the database * rrddim hide and unhide will check "cached" state, update the database if needed and set the cache flag accordingly * Check the dimension option and only do the database update if the cached state is different * Defer the dimension payload check to the ACLK sync thread (netdata#12951) Defer payload check to the aclk sync thread * [ci skip] Update changelog and version for nightly build: v1.34.0-198-nightly. * Suppress warning when freeing a NULL pointer in onewayalloc_freez (netdata#12955) * Cleanup chart hash and map tables on startup (netdata#12956) * statsd sets should count unique values (netdata#12963) statsd set should count unique values * [ci skip] Update changelog and version for nightly build: v1.34.0-202-nightly. * Apply some logic to possible streaming destinations (netdata#12866) * replace connect_to_one_of with connect_to_one_of_destinations * move functions from socket.c * use sizeof * move current destination pointer to host * formatting * use snprintfz * get entries in same order * handle single destination as before (or when it is the last of the list), instead of skiping it every other loop * try other destinations on ssl problem * fix: keep virtualization unknown if all used commands are not available (netdata#12964) * cleanup and optimize rrdeng_load_metric_next() (netdata#12966) * cleanup and optimize rrdeng_load_metric_next() * fixed typo * Don't permanetly disable a destination because of denied access (netdata#12971) dont make denied destination permanent * feat(charts.d/apcupds): add load usage chart (Watts) (netdata#12965) * chore: check link local address before querying cloud instance metadata (netdata#12973) check link local address before querying cloud providers data * [ci skip] Update changelog and version for nightly build: v1.34.0-209-nightly. * optimize poll_events() to spread the work over the threads more evenly (netdata#12975) * optimize poll_events() to spread the work over the threads more evenly * fixed typos, code cleanup * better error handling * prevent crash in case callbacks manipulate the sockets arrays - added warnings * query engine optimizations and cleanup (netdata#12978) * move number unpacking close to next_metric * dont miss group value flags * [ci skip] Update changelog and version for nightly build: v1.34.0-212-nightly. * chore(apps.plugin): change cpu_guest chart context (netdata#12983) * Add automatic retries fo static builds during nightly and release builds. (netdata#12961) * fix(kickstart.sh): handle the case when `tput colors` doesn't return a number (netdata#12979) * fix: don't kill Netdata PIDs if successfully stopped Netdata (netdata#12982) * modify code to resolve compile warning issue (netdata#12969) * Make heartbeat a static chart (netdata#12986) * Improve reconnect node instructions (netdata#12987) * [ci skip] Update changelog and version for nightly build: v1.34.0-220-nightly. * Faster queries (netdata#12988) * faster rrdeng_load_metric_next() * no need to check validity for number - already done at the query side * solve discrepancy between query create and free * inline unpack_storage_number * Stream and advertise metric correlations to the cloud (netdata#12940) * stream and advertise mc to the cloud * better reporting * remove log * remove aclk debug * Return rc->last_update from alarms_values api (netdata#12968) * add rc last_update as when in alarms values api * rename when to last_updated * update swagger * Fix compilation warnings (netdata#12993) * Fix nanosleep on platforms other than Linux (netdata#12991) * Run the /net/dev module of the proc plugin in a separate thread (netdata#12996) * Update dashboard to version v2.25.4. (netdata#13000) Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> * Don't expose the chart definition to streaming if there is no metadata change (netdata#12990) * Only clear the RRDSET_FLAG_UPSTREAM_EXPOSED chart flag if metadata has changed * Handle modification of units as well * Initialize old_units in the chart state * add dictionary support to statsd (netdata#12980) * add dictionary support to statsd * faster statsd sets and dictionaries; disabled events dimensions by default * properly detect tags, even without a sampling rate * statsd assumes a pipe between fields * missing param * allow names without values and support more unknown fields * more parser fixes * support multiple tags; remove the sum from the dimensions of histograms and timers, but keep it for synthetic charts * Parse statsd tags and support changing units of private charts * remove debug code * added support for naming dimensions too * updated docs * added support for family tags * updated docs * [ci skip] Update changelog and version for nightly build: v1.34.0-230-nightly. * Delay children chart obsoletion check (netdata#12992) * wait untill after 2 minutes of last chart received to run obsoletion check * turn write to read locks * Use printf instead of echo for printing collected warnings in kickstart.sh. (netdata#13002) Use printf instead of echo for printing collected warnings. This ensures that the formatting actually happens correctly. * Autodetect coverity install path to increase robustness (netdata#12995) * [ci skip] Update changelog and version for nightly build: v1.34.0-234-nightly. * Alarms py collector add filtering (netdata#12972) add ability to filter alarms in `alarms.conf` * Add Alpine 3.16 to CI and platform support document. (netdata#12997) * fix COVERITY_PATH added with INSTALL_DIR into PATH (netdata#13014) * Cleanup Challenge Response Code (netdata#11730) - Use OpenSSL for Base64 encode/decode - general cleanup of code * Pass the child machine's guid to the goto_url link (netdata#12609) * pass child machine guid * pass parameter * Only try to update repo metadata in updater script if needed. (netdata#13009) * Cache invariant components in static builds to reduce build times. (netdata#12877) * Add basic build caching support to static builds. Cache is store din `artifacts/cache/${BUILDARCH}`. Each third-party component utilizes a separate build cache. Invalidation is only done for version changes (more rigorous invalidation is expected to be handled externally). * Integrate static build caching with CI. * Fix fping cache handling. * Test caching in CI. * Properly skip rebuilds on cache hits. * Remove static build container when done with it. * Reuse existing image automatically if it’s for the correct platform. * Test CI build caching. * Fix static build job names. * Fix coverity issue 378598 (netdata#13022) move appconfig_get before aclk lock * Initialize a pointer and add a check for it (netdata#13023) * prevent gap filling on dbengine gaps (netdata#13027) * Update dashboard to version v2.25.6. (netdata#13028) Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> * [ci skip] Update changelog and version for nightly build: v1.34.0-246-nightly. * add hostname to mirrored hosts (netdata#13030) * [ci skip] Update changelog and version for nightly build: v1.34.0-248-nightly. * Update apps_groups.conf (netdata#13033) * Trigger queue removed alerts on health log exchange with cloud (netdata#12954) trigger queue removed on health log exchange with cloud * Check return value and log an error on failure (netdata#13037) * [ci skip] Update changelog and version for nightly build: v1.34.0-252-nightly. * Fix coverity issue 378617,378615 (netdata#13021) * Fix CID 378617 * Fix CID 378615 * Make sure the ST rrdr lock indicator is set/reset while holding a lock * Switch to int * Skip collecting network interface speed and duplex if carrier is down (netdata#13019) * When sending a dimension for the first time, make sure there is a non zero created_at timestamp (netdata#13035) * Treat dimensions as normal when we don't have enough/valid data. (netdata#13005) Ideally, we'd log such cases but this is not currently feasible because we have to process thousands of dimensions per second. * replace `history` with relevant `dbengine` params (netdata#13041) * Fix the retry count and netdata_exit check when running an sqlite3_step command (netdata#13040) * Move retry count to the header file * Add SQL_MAX_RETRY count and fix the netdata_exit check * Write pid file (needed by logrotate) (netdata#9377) * Add additional metadata to the data response (netdata#13036) * Consolidate query params * Add new option to show full dimensions in the json header (this will include dimensions, charts and chart labels) * Group and pass parameters with query_params * [ci skip] Update changelog and version for nightly build: v1.34.0-261-nightly. * Fix disabled apps (ebpf.plugin) (netdata#13044) * add note about anomaly advisor (netdata#13042) * Schedule retention message calculation to a worker thread (netdata#13039) * Move aclk_update_retention to the proper header file * Do a scan but avoid going through all the dimensions if we have too much to delete -- do not generate a retention message in that case * Schedule the retention calculation to a worker * Adjust messages in the access log * Fix compilation errors with --disable-cloud * Dictionary with JudyHS and double linked list (netdata#13032) * dictionary internals isolation * more dictionary cleanups * added unit test * we should use DICT internally * disable cups in cmake * implement DICTIONARY with Judy arrays * operational JUDY implementation * JUDY cleanup * JUDY summary added * JudyHS implementation with double linked list * test negative searches too * optimize destruction * optimize set to insert first without lookup * updated stats * code cleanup; better organization; updated info * more code cleanup and commenting * more cleanup, renames and comments * fix rename * more cleanups * use Judy.h from system paths * added foreach traversal; added flag to add item in front; isolated locks to their own functions; destruction returns the number of bytes freed * more comments; flags are now 16-bit * completed unittesting * addressed comments and added reference counters maintainance * added unittest in main; tested removal of items in front, back and middle * added read/write walkthrough and foreach; allowed walkthrough and foreach in write mode to delete the current element (used by cups.plugin); referenced counters removed from the API * DICTFE.name should be const too * added API calls for exposing all statistics * dictionary flags as enum and reference counters as atomic operations * more comments; improved error handling at unit tests * added functions to allow unsafe access while traversing the dictionary with locks in place * check for libcups in cmake * added delete callback; implemented statsd with this dictionary * added missing dfe_done() * added alternative implementation with AVL * added documentation * added comments and warning about AVL * dictionary walktrhough on new code * simplified foreach; updated docs * updated docs * AVL is much faster without hashes * AVL should follow DBENGINE * coverity fixes about statsd; removal of strsame (netdata#13049) * Fix dictionary crash walkthrough empty (netdata#13051) fix dictionary walkthrouhg crash when the dictionary is empty * [ci skip] Update changelog and version for nightly build: v1.34.0-268-nightly. * dictionary improvements (netdata#13052) * fix typo in foreach write; added unit tests to traverse empty dictionaries * rename variable dfe in macro to be uniform with name variable * Check for host labels when linking alerts for children (netdata#13053) check for host labels when linking alerts * Fix coverity 378625 (netdata#13055) free jsonb buffer * add the ability to merge dictionary items (netdata#13054) * add the ability to merge old value and new value * docs * merge to conflict * fix(cgroup.plugin): read k8s_cluster_name label from the correct file (netdata#13062) * Initialize chart label key parameter correctly (netdata#13061) Pass chart label key * [ci skip] Update changelog and version for nightly build: v1.34.0-275-nightly. * update codeowners (netdata#13060) * Fix handling of temp directory in kickstart when uninstalling. (netdata#13056) This makes temporary directory creation idempotent, and ensures it’s invoked in each place that needs it. It also tidies up the cleanup code so that it only tries to remove the temporary directory if one was created. * Fix locking access to chart labels (netdata#13064) No write lock required * [ci skip] Update changelog and version for nightly build: v1.34.0-279-nightly. * Allow usage of the new MQTT 5 implementation (netdata#12838) * adds support for new MQTT5 implementation in agent, currently by default disabled as Tech Preview * Added Alma Linux 9 and RHEL 9 support to CI and packaging. (netdata#13058) * Added Alma Linux 9 and RHEL 9 support to CI and packaging. * Fix RPM deps for Alma 9. * Fix package testing code for Alma 9. * Update default value for "host anomaly rate threshold" (netdata#13075) * Add improved reinstall documentation. (netdata#13047) Including explaining how to do a clean reinstall, and how to switch install types. * Fix handling of DEB package naming in CI. (netdata#13076) * Fix handling of DEB package naming in CI. * Bump repoconfig package versions. To ensure naming consistency. * Fix repoconfig package renaming. * [ci skip] Update changelog and version for nightly build: v1.34.0-285-nightly. * fix(updater): don't produce any output if binpkg update completed successfully (netdata#13081) * don't produce any output when binpkg update succeeded * redirect update repo, update netdata output to 3 * Remove Fedora 34 from CI and package builds. (netdata#12875) * Remove Fedora 34 from CI and package builds. EOL date is 2022-06-07. * Update platform support document. * Fix Coverity errors in mqtt_websockets submodule (netdata#13082) - update mqtt_websockets submodule to bring in the Coverity scan fixes * [ci skip] Update changelog and version for nightly build: v1.34.0-289-nightly. * Update README.md (netdata#13088) * Update README.md (netdata#13089) * fix(updater): return 0 on successful update for native packages when running interactively (netdata#13083) return 0 on success in udpate_binpkg * [ci skip] Minor release v1.35.0. * [ci skip] Update changelog and version for nightly build: v1.35.0-1-nightly. * Temporarily disable updates for static builds. (netdata#13100) * Temporarily disable updates for static builds. * Fix typo in warning message. * Remove unnescesary ‘cleanup’ code. (netdata#13103) This was causing the updter to be unconditionally disabled on static builds, and we don’t actually _need_ any of it anymore since we’ve already decoupled the auto-update handling state from the installation process (in fact, this should have been removed then, but it got missed somehow). * [ci skip] Update changelog and version for nightly build: v1.35.0-4-nightly. * Re-enable updates for systems using static builds. (netdata#13110) Revert "Temporarily disable updates for static builds. (netdata#13100)" This reverts commit b67c102. * docs(statsd.plugin): fix indentation (netdata#13096) fix indentation * Add openSUSE Leap 15.4 to CI and package builds. (netdata#12270) Add openSUSE Leap 15.4 t CI and package builds. Expected release date 2022-06-08. * [ci skip] Update changelog and version for nightly build: v1.35.0-8-nightly. * fix virtualization detection on FreeBSD (netdata#13087) * Add user netdata to secondary group in DEB package (netdata#13109) * Add user netdata to secondary group * Update netdata.postinst * Labels with dictionary (netdata#13070) * squashed and rebased to master * fix overflow and single character bug in sanitize; include rrd.h instead of node_info.h * added unittest for UTF-8 multibyte sanitization * Fix unit test compilation * Fix CMake build * remove double sanitizer for opentsdb; cleanup sanitize_json_string() * rename error_description to error_message to avoid conflict with json-c * revert last and undef error_description from json-c * more unittests; attempt to fix protobuf map issue * get rid of rrdlabels_get() and replace it with a safe version that writes the value to a buffer * added dictionary sorting unittest; rrdlabels_to_buffer() now is sorted * better sorted dictionary checking * proper unittesting for sorted dictionaries * call dictionary deletion callback when destroying the dictionary * remove obsolete variable * Fix exporting unit tests * Fix k8s label parsing test * workaround for cmocka and strdupz() * Bypass cmocka memory allocation check * Revert "Bypass cmocka memory allocation check" This reverts commit 4c49923. * Revert "workaround for cmocka and strdupz()" This reverts commit 7bebee0. * Bypass cmocka memory allocation checks * respect json formatting for chart labels * cloud sends colons * print the value only once * allow parenthesis in values and spaces; make stream sender send quotes for values Co-authored-by: Vladimir Kobal <vlad@prokk.net> * 73x times faster metrics correlations at the agent (netdata#13107) * faster correlations * 4x times faster correlations * a little bit more help * 10x times faster metrics correlations * 6 digits precision; better comments * enabled metrics correlations by default * abstracted DIFFS_NUMBER to allow easily changing it * reworked the entire logic to have more accuracy and support a baseline that is power of two multiple of highlight * properly calculate shifts * even more improved version * added support for timeout; fixed another memory leak; skipped hidden dimensions * default timeout 1min * reduce memory even further * use dictionary for the list of charts and optimize locks * return 403 forbidden, when mc is not enabled * added query options * dont process zero dimensions * added volume method as an option to metric correlations ; now metric correlations can support multiple implementations * make sure we will never crash * spread results evenly for both kstwo and volume * fixed bug in query engine that was missing misaligned queries when a single point was requested from the db; improved comments; improved query flags * updated swagger and added sane defaults; query options are now supported, including anomaly-bit * added "raw" option to allow cross node correlations; added "group" option to allow different time aggregations; allowed calling metric correlations without any parameters; allowed calling metric correlations with relative timestamps; added timeout to volume method; properly handled timeout on ks2 method; json output now sends all parameters back - same for json_wrap; modified query engine to use present time for relative timestamps; modified "allow_past" to mean both past backwards and forwards * emulate the old behaviour about zero points * 100% accuracy against python ks_2samp(); now the default is volume and the default points are 500 * added config option to change default metric correlations method * removed work-arounds now that rrdlabels are merged * [ci skip] Update changelog and version for nightly build: v1.35.0-13-nightly. * Ci coverage (netdata#13118) * add check to verify autoupdate is still enabled after netdata update * add script to install CI support packages before installing netdata * Fix typo in release base prep script. (netdata#13098) * fixed coveriry 379136 379135 379134 379133 (netdata#13123) * allow label names to have slashes (netdata#13125) * allow label names to have slashes * fixed flipped comment * buffer overflow detected by the compiler (netdata#13120) * buffer overflow detected by the compiler * also increase the buffer +1 * ACLK statistics on bytes recvd and sent (netdata#13091) * fix(python.d/mongodb): set `serverSelectionTimeoutMS` for pymongo4+ (netdata#13135) fix server_selection_timeout for pymongo4+ * fix(python.d): urllib3 import collection for py3.10+ (netdata#13136) * [ci skip] Update changelog and version for nightly build: v1.35.0-22-nightly. * Use a separate thread for slow mountpoints in the diskspace plugin (netdata#13067) * use ks2 as MC default (netdata#13131) * fix crashes due to misaligned allocations (netdata#13137) * Add missing control to streaming (netdata#13112) fix_tls_stream: Add call to SSL_get_error to avoid an infinite loop * Add an option to use malloc for page cache instead of mmap (netdata#13142) Add an option to switch to using malloc for page cache instead of mmap * Update docs on what to do if collector not there (netdata#13152) * [ci skip] Update changelog and version for nightly build: v1.35.0-29-nightly. * Remove pinned page reference (netdata#13108) * Disable reference to prev_descr as we do not keep two pages pinned * Remove extra pinned page from page cache calculations * Removed invalid comment * Remove unused variable * Add cargo/rustc/bazel/buck to apps_groups.conf (netdata#13143) * revert to default of `host anomaly rate threshold=0.01` (netdata#13150) * fix conditions for nightly build triggers (netdata#13145) * Fix labels unit test (netdata#13156) * add CAP_SYS_RAWIO to Netdata's systemd unit CapabilityBoundingSet (netdata#13154) * Add mem.available chart to FreeBSD (netdata#13140) * Configurable storage engine for Netdata agents: step 3 (netdata#12892) * storage engine: add host context API Add a new API to allow storage engines to manage host contexts. * Replace single global context with per-engine global context * Context is full managed by storage engines: a storage engine can use no context, a global engine context, per host contexts, or a mix of these. * Currently, only dbengine uses contexts. Following the current logic, legacy hosts use their own context, while non-legacy hosts share the global context. * storage engine: use empty function instead of null for context ops * rrdhost: don't check return value for void call * rrdhost: create context with host * storage engine: move rrddim ops to rrddim_mem.{c,h} * storage engine: don't use NULL for end-of-list marker * storage engine: fallback to default engine * Use memset to mark the empty words in the quoted_strings_splitter function (netdata#13161) Use memset to terminate the words * Don’t manipulate positional parameters in DEB postinst script. (netdata#13169) * Fix coverity issues (netdata#13168) * [ci skip] Update changelog and version for nightly build: v1.35.0-41-nightly. * enable ml by default (netdata#13158) * enabled ml by default * Fix data query on stale chart (netdata#13159) * Fix data query on stale chart * Remove more checks vs the last timestamp of a point collected * feat(proc/proc_net_dev): add dim per operstate to the "Interface Operational State" chart (netdata#13167) * Revert "Configurable storage engine for Netdata agents: step 3 (netdata#12892)" (netdata#13171) This reverts commit 100a12c. A couple parent/child startup/shutdown scenarios can lead to crashes. * feat(proc/proc_net_dev): add dim per duplex state to the "Interface Duplex State" chart (netdata#13165) * allow traversing null-value dictionaries (netdata#13162) * allow traversing null-value dictionaries * fix lgtm report * void the value too * removed NEVERNULL directive * Fix coverity 378587 (netdata#13024) * check for return value of sysconf * if sysconf fails set OWA_NATURAL_PAGE_SIZE to 4096 * [ci skip] Update changelog and version for nightly build: v1.35.0-49-nightly. * feat(proc/proc_net_dev): add dim per phys link state to the "Interface Physical Link State" chart (netdata#13176) * add dim per carrier state * fix down state * Update dashboard to version v2.26.2. (netdata#13177) Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> * set default for `minimum num samples to train` to `900` (netdata#13174) This will enable first set of initial models to be trained quicker and makes sense now that ml is enabled by default. * add k8s_state dashboard_info (netdata#13181) * [ci skip] Update changelog and version for nightly build: v1.35.0-54-nightly. * Add ml alerts examples (netdata#13173) * add ml alarm examples * Update Makefile.am * add hyperlinks and node level AR example * fix(freebsd.plugin): fix wired/cached/avail memory calculation on FreeBSD with ZFS (netdata#13183) * [ci skip] Update changelog and version for nightly build: v1.35.0-57-nightly. * Update dashboard to version v2.26.5. (netdata#13192) Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> * Query Engine multi-granularity support (and MC improvements) (netdata#13155) * set grouping functions * storage engine should check the validity of timestamps, not the query engine * calculate and store in RRDR anomaly rates for every query * anomaly rate used by volume metric correlations * mc volume should use absolute data, to avoid cancelling effect * return anomaly-rates in jasonwrap with jw-anomaly-rates option to data queries * dont return null on anomaly rates * allow passing group query options from the URL * added countif to the query engine and used it in metric correlations * fix configure * fix countif and anomaly rate percentages * added group_options to metric correlations; updated swagger * added newline at the end of yaml file * always check the time the highlighted window was above/below the highlighted window * properly track time in memory queries * error for internal checks only * moved pack_storage_number() into the storage engines * moved unpack_storage_number() inside the storage engines * remove old comment * pass unit tests * properly detect zero or subnormal values in pack_storage_number() * fill nulls before the value, not after * make sure math.h is included * workaround for isfinite() * fix for isfinite() * faster isfinite() alternative * fix for faster isfinite() alternative * next_metric() now returns end_time too * variable step implemented in a generic way * remove left-over variables * ensure we always complete the wanted number of points * fixes * ensure no infinite loop * mc-volume-improvements: Add information about invalid condition * points should have a duration in the past * removed unneeded info() line * Fix unit tests for exporting engine * new_point should only be checked when it is fetched from the db; better comment about the premature breaking of the main query loop Co-authored-by: Thiago Marques <thiagoftsm@gmail.com> Co-authored-by: Vladimir Kobal <vlad@prokk.net> * Add type label for network interfaces (netdata#13187) * feat(python.d/smartd_log): add 2nd job that tries to read from '/var/lib/smartmontools/' (netdata#13188) * Fix cgroups netdev chart labels (netdata#13200) * Allow for an easy way to do metadata migrations (netdata#13196) Allow for an easy way to migrate metadata to a new database schema (versioning) * Clean sqlite prepared statements on thread shutdown (netdata#13193) * Clean sqlite prepared statements on thread shutdown * Add note for pending statements to finalize * [ci skip] Update changelog and version for nightly build: v1.35.0-65-nightly. * Add configuration for dbengine page fetch timeout and retry count (netdata#13194) * Add configuration for page cache fetch timeout and retry count Change page cache wait default timeout to 3 seconds * Issue info message in the error.log if values not within expected lower range * Fix compilation errors with --disable-dbengine * Print INTERNAL BUG messages only when NETDATA_INTERNAL_CHECKS is enabled (netdata#13207) put INTERNAL BUG messages inside NETDATA_INTERNAL_CHECKS * chore(netdata-installer): remove a call to 'cleanup_old_netdata_updater()' because it is no longer exists (netdata#13189) * [ci skip] Update changelog and version for nightly build: v1.35.0-69-nightly. * Add user plugin dirs to environment (netdata#13203) Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * [ci skip] Update changelog and version for nightly build: v1.35.0-71-nightly. * make configuration example clearer (netdata#13182) make ml configuration example cleare * [ci skip] Update changelog and version for nightly build: v1.35.0-73-nightly. * deduplicate mountinfo based on mount point (netdata#13215) * [ci skip] Update changelog and version for nightly build: v1.35.0-75-nightly. * Use new mqtt implementation as default (netdata#13132) flips mqtt5 to be default instead of mqtt3 * fix(updater): fix updating when using `--force-update` and new version of the updater script is available (netdata#13104) * Ensure tmpdir is set for every function that uses it. (netdata#13206) The set_tmpdir function is idempotent specifically to allow for this type of usage, and this should help avoid bugs involving temporary directory handling going forwards. * Don’t use realpath to find kickstart source path. (netdata#13208) Don’t use realpath to find kcisktart source path. It doesn’t work on macOS and leads to a bogus error message being printed. * Properly handle interactivity in the updater code. (netdata#13209) * Clean up updater option parsing. This will make the remaining changes simpler, and also ease maintenance long-term. * Add updater options to override interactivity detection. * Pass interactivity from kickstart to updater when invoking updater. * Removes Legacy JSON Cloud Protocol Support In Agent (netdata#13111) * removes old protocol support (cloud removed support already) * ci: add issues to the Agent Board project workflow (netdata#13225) * add add issues to Agent Board project workflow * add 'transferred' event * fix(cgroups.plugin): fix qemu VMs and LXC containers name resolution (netdata#13220) * [ci skip] Update changelog and version for nightly build: v1.35.0-84-nightly. * feat(python.d): load modules from user plugin directories (NETDATA_USER_PLUGINS_DIRS) (netdata#13214) * Add more sqlite unittests (netdata#13227) * Bump repoconfig package version used in kickstart.sh (netdata#13235) * Update netdata commands (netdata#13080) * Update netdata commands Adding the `-W buildinfo` options. * Update README.md * Update README.md * Update daemon/README.md Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * Update daemon/README.md Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * also add the change in the daemon command line help message Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud> * remove whitespace Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud> Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * Use correct PAT for add-to-project workflow. (netdata#13234) * Implement PackageCloud cleanup (netdata#13236) * add script for netdata packagecloud cleanup * add script for netdata packagecloud cleanup * remove old package-cloud cleanup code * add workflow for packagecloud cleanup * Debug msg * allow individual matrix jobs to complete independently to each other * perform cleanup without dry-run * remove debug msg * remove cleanup line from the slack notification * Upload packages to new self-hosted repository infrastructure. (netdata#13240) * Match packaging arches to distro architecture names. This will simplify handling of uploads to our new self-hosted packaging infrastructure. * Add package uploads for RPM packages on new infrastructure. * Add package upload support for DEB packages. * Saner tmpdir handling. * Upload fixes. * Differentiate different DEB distros in package names. * Workflow fixes. * netdata doubles (netdata#13217) * netdata doubles * fix cmocka test * fix cmocka test again * fix left-overs of long double to NETDATA_DOUBLE * RRDDIM detached from disk representation; db settings in [db] section of netdata.conf * update the memory before saving * rrdset is now detached from file structures too * on memory mode map, update the memory mapped structures on every iteration * allow RRD_ID_LENGTH_MAX to be changed * granularity secs, back to update every * fix formatting * more formatting * Dictionaries with reference counters and full deletion support during traversal (netdata#13195) * dont use atomic operations when not needed; detect misuse of the the unsafe functions * use relaxed atomic operations for statistics * use relaxed atomic operations for statistics * dictionaries now use reference counters, allowing deletetions of any item while traversing it * added acquire/release interface to dictionaries * added unittest for reference counters * added NETDATA_INTERNAL_CHECKS logs to detect non-exclusive access to crusial parts of the dictionaries * dictionaries cannot be deleted while there are referenced items in them - they will be deleted once the last item gets unreferenced * cleanup * properly cleanup released items * maintain counters for readers and writers; defer all deletes on sorted walkthrough; cleaner internal_error(); * somewhat faster reference counters on single threaded dictionaries * minor optimizations; allow compiling without internal checks * Delay health until obsoletions check is complete (netdata#13239) * wait until obsoletions check is complete to run health * run the checks 5 minutes after agent connect time * Updates the sqlite version in the agent (netdata#13233) New version sqlite Enabling also #define SQLITE_UDL_CAPABLE_PARSER 1 #define SQLITE_ENABLE_UPDATE_DELETE_LIMIT 1 #define SQLITE_OMIT_LOAD_EXTENSION 1 #define SQLITE_ENABLE_DBSTAT_VTAB 1 * [ci skip] Update changelog and version for nightly build: v1.35.0-96-nightly. * Fix build matrix generation for build workflow. (netdata#13254) * Fix build matrix generation for build workflow. * Fix build matrix sorting. * Fix further typos. * fix: fix a base64_encode bug (netdata#13074) * Use old mqtt implementation as default (netdata#13258) mqtt5 as default off * Add fstype labels to disk charts (netdata#13245) Co-authored-by: vkalintiris <vasilis@netdata.cloud> Co-authored-by: Vladimir Kobal <vlad@prokk.net> Co-authored-by: Netdata bot <43409846+netdatabot@users.noreply.github.com> Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> Co-authored-by: netdatabot <bot@netdata.cloud> Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Emmanuel Vasilakis <mrzammler@mm.st> Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com> Co-authored-by: Timotej S <6674623+underhood@users.noreply.github.com> Co-authored-by: Chris Akritidis <43294513+cakrit@users.noreply.github.com> Co-authored-by: Costa Tsaousis <costa@netdata.cloud> Co-authored-by: Konstantin Shalygin <k0ste@k0ste.ru> Co-authored-by: Austin S. Hemmelgarn <austin@netdata.cloud> Co-authored-by: Andika Triwidada <andika@gmail.com> Co-authored-by: Marian Savchuk <Marian.Savchuk@cypress.com> Co-authored-by: maneamarius <locomotion.itservices@gmail.com> Co-authored-by: thiagoftsm <thiagoftsm@gmail.com> Co-authored-by: kklionz <85677209+kklionz@users.noreply.github.com> Co-authored-by: Andrew Maguire <andrewm4894@gmail.com> Co-authored-by: FQX <qfuxiang@gmail.com> Co-authored-by: Luca Olivetti <luca@ventoso.org> Co-authored-by: Shyam Sreevalsan <shyam@netdata.cloud> Co-authored-by: Igor Karpov <iigorkarpov@users.noreply.github.com> Co-authored-by: Adrien Béraud <adrien.beraud@savoirfairelinux.com> Co-authored-by: Tasos Katsoulas <tasos@netdata.cloud>
commit b10bdba0e814bab158ef378f04ef6eb2317ef346 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Oct 25 12:53:28 2022 +0100 Squashed commit of merging 'master' into logs-manag-master: commit 64386fa3839d87a252d24513cd1bdae7dd1c4896 Merge: 96246c9ed 4b9f02da2 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Oct 25 12:44:32 2022 +0100 Merge branch 'master' into logs-manag-master commit 96246c9edf9e4d3f73e2d5fa76653817095fb262 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Oct 21 18:40:49 2022 +0100 Complete refactoring of parser.c and parser.h commit 9d3a7a6aefd4465b1a36af9d9fa2bfcb6e0c5d55 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Oct 21 15:08:15 2022 +0100 Remove log_parser from p_file_info commit d0d8f647005cd1b0b8cbc47fb75d7d2a013b2414 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Oct 21 14:50:28 2022 +0100 Remove all stream-related files commit ce0fb29223f27f8e28dc513184753aa3754ae5ca Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 20 20:48:34 2022 +0100 Refactor Log_parser_metrics_t to use a union to reduce memory consumption and code complexity commit 89ada4e49e72eba0da06b2512564a28339d44bce Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 20 18:55:51 2022 +0100 Change logsmanagement_conf.h name and structure commit 0ca25f062958811511dcac16c7a9c258bfc9222e Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 20 15:54:25 2022 +0100 Fix wrong timer interval after 'update_every' changes commit fa6fa09f3bd0eef75510c946a4fed418eb61fe61 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 13 19:48:31 2022 +0100 Add update every option per log source commit 003fb181af71d1bf3acc0a1b7bd5e6832a0adc2b Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 13 19:11:29 2022 +0100 Add logs management global update every config option commit 41cef31e61b2658f6774c2df22b604e0663eb3a3 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 13 18:34:19 2022 +0100 Escape potential backslashes in regex to ensure dimenion name in chart is displayed correctly commit b5b3c168be7310039173d6e678093fd784bd3e11 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Oct 10 22:26:13 2022 +0100 Add ability to define names for regex dimensions commit 1416f0900c789e914dbca2bc104cea8b291ddc7b Author: Dim-P <dimitris1703@gmail.com> Date: Mon Oct 10 17:50:30 2022 +0100 Silence compiler truncation warnings commit 2af527a566435cc47f8f8a56cfa3b110b0c243d6 Author: Dim-P <dimitris1703@gmail.com> Date: Sat Oct 8 03:38:14 2022 +0100 Fix wrong timestamps on 32-bit OS commit 477338bfe48d65793c1312580fe4f5b962f0702d Author: Dim-P <dimitris1703@gmail.com> Date: Sat Oct 8 03:04:09 2022 +0100 Fix failing build on Rasberry Pi commit 475ed2c6db3a6d83d15ef69f464b1268c5f62d3d Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 6 20:34:54 2022 +0100 Add custom regex-based charts commit 27ecb125cf30d854e2593d91be5c43428f8f4bfd Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 6 15:21:30 2022 +0100 Add generic log charts commit b7ae826e9677ce6cf51ebfc67d0166b5ab16cdc0 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 6 13:47:32 2022 +0100 Add metrics extraction for generic log parser commit 9f71f6ff676258253277d1e3c4d9b9e7b59550d0 Author: Dim-P <dimitris1703@gmail.com> Date: Wed Oct 5 23:23:01 2022 +0100 Fix SIGSEGV due to uninitialised regex commit ed3715928f57834490fc559dd46c6c4ada543317 Author: Dim-P <dimitris1703@gmail.com> Date: Wed Oct 5 22:56:25 2022 +0100 WIP: Add regex for generic log parser commit 9a4349067e17d2c46924600eda319710792a4866 Author: Dim-P <dimitris1703@gmail.com> Date: Wed Oct 5 20:46:48 2022 +0100 Replace last \n with \0 in collected log batches commit 0ed60ae1f89847fffd6bd25a9211b586a378c0b1 Author: Dim-P <dimitris1703@gmail.com> Date: Wed Oct 5 18:47:49 2022 +0100 Remove some debug prints commit 35e6045d642530d11798f09cf4e4d61a3232182f Author: Dim-P <dimitris1703@gmail.com> Date: Fri Sep 30 16:00:25 2022 +0100 Add generic log support. Add custom chart config init commit b4a601338735822c565db34bd56a4ec1ba413222 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 29 21:45:08 2022 +0100 Comment out some debug log prints commit 1268b0c43258b00e7ad0a9df529ff2da6686f38d Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 29 21:20:12 2022 +0100 Complete refactoring of logsmanagement.plugin directory commit 06018c5f848b3089f8e152762d9600703af68399 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 29 12:09:54 2022 +0100 Move vhosts and ports metrics collection commit c769d3e9badae3573fbf9ef6537edb3f6c8d8ebd Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 27 22:37:26 2022 +0100 WIP: Further moving / refactoring of plugin_logsmanagement.c code commit e74c8361506f3e76978240402591a02066b58dad Author: Dim-P <dimitris1703@gmail.com> Date: Fri Sep 23 20:07:28 2022 +0100 WIP: Breakdown plugin_logsmanagement.c in multiple files and start copying commit 910eaa39c0f0ec7a42014932dcaab7299534a0b2 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 22 21:12:29 2022 +0100 Add global config option to change default logs management db dir commit 94c51d851619f760fad0022b7f605e7667a95092 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 22 19:56:36 2022 +0100 Add global config option to disable logs management commit c828be0ec17cfc50e16dd1e1bbe6a763eb4b8dd2 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 22 19:15:16 2022 +0100 Add logsmanagement.plugin worker commit 713ccf9b98f9e846ce62477763204ebaba31f3ea Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 22 17:42:35 2022 +0100 Fix conversion of STRING to str in logsmanagement.plugin commit 11b356b2fa7e06213b434af36b35dabc5f226b78 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 20 21:23:13 2022 +0100 Add count of log records matching keyword in query results commit 7fab02c027de27b2cd5918a8f87ae8d82c69ea2f Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 20 16:50:10 2022 +0100 Add measure of query execution time commit ef9d5c1df848638e8c4371cbac75443965608e8a Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 20 15:56:04 2022 +0100 Working db_search_compound() queries commit 2385a12659aea49e2387e208388f4b6043219fde Author: Dim-P <dimitris1703@gmail.com> Date: Fri Sep 16 22:15:35 2022 +0100 WIP: Resolve issues with database detaching failures commit 1a9e5348d9fb37c4fe6b2a3fa2c93c10c3d4cffc Author: Dim-P <dimitris1703@gmail.com> Date: Fri Sep 16 16:13:41 2022 +0100 Working TEMP VIEW for db_search_compound() commit 4301cadf8fa83378f3f3b922cd5747736471d6e1 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 15 21:40:27 2022 +0100 WIP: Create TEMP VIEW db_search_compound() query commit 3f7e6823c798158810626b638f7e8158fcb24205 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 15 12:36:52 2022 +0100 Databases attach / detach works commit 108040285abe74696112301ae21a5c86c3d3406e Author: Dim-P <dimitris1703@gmail.com> Date: Mon Sep 12 21:55:23 2022 +0100 WIP: Compound queries commit f387e1a9ada7b12a56f35faf3eee1cfb0726126e Merge: cab26f071 eddb7125f Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 22 16:03:31 2022 +0100 Merge branch 'master' into logs-manag-master commit cab26f071d42a9e3f8f7f55693d2f8db51bf5461 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 6 21:18:55 2022 +0100 Fix backslashes not escaped correctly in JSON query response. commit 5baa4ef0e149cfadeab219abdab1c301c26a434b Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 6 20:30:36 2022 +0100 Fix double quotes in logs not escaped correctly in query response JSON. commit 8edd975dd39ac4a361dfd24d81b1506339534967 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 6 14:47:18 2022 +0100 Update main.db schema to include Machine_GUID and log type commit 37e171415abca17f4c2c21276639f07e7039e702 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 6 14:46:36 2022 +0100 Fix bug of docker events and systemd conflicting DBs if using default paths commit f5466d210bea8f5bd120bc0884258d755abe4efe Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 6 14:45:40 2022 +0100 Fix wrong log type for FLB_DOCKER_EV. commit 46f22124eb81568310a78855f3672d32a30a6edc Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 6 00:23:45 2022 +0100 Fix bug of conflicting tags for Fluent-Bit inputs/outputs commit 3de72c88e3ad6154e5c636fbd9e2269220adac98 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Sep 5 19:59:26 2022 +0100 Add option to pass on log source path for docker events and systemd logs commit 2d9598f3b39d8542a6204745752a12877b8539ad Author: Dim-P <dimitris1703@gmail.com> Date: Mon Sep 5 16:11:28 2022 +0100 Re-add git clean and submodule update commit a738e0cd152120554120ea77e220a49ebfbcb6d4 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Sep 5 15:36:03 2022 +0100 Update run_stress_test.sh help commit e0075c649764413a7b6949928c83c14d7bb8c1dc Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 1 21:50:24 2022 +0100 Set default access.log log type to flb_web_log commit f2dfb6746321ea2962cdce601f6dc7b1b1691fc1 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 1 19:26:58 2022 +0100 Fix SIGSEGV on web log auto format detection. Add more unit test cases commit 04b162f237037e9a416aa0e2118fd09ef2cf47d8 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 1 17:21:23 2022 +0100 Do not run stress_test unless explicitly asked to do so commit b18d643dc94aa1b130541dda1627a955c656e1d4 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 1 17:20:13 2022 +0100 Comment out docker_ev_action_string[] since not used yet commit 9a5742eca9ce620155c5e438b7c9ce8521124aba Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 1 17:17:35 2022 +0100 Improve loading of logsmanagement.conf Load stock logsmanagement.conf if user-defined one is not available. If stock logsmanagement.conf is not found, then do not start logs management engine at all. commit 230ab034e3c4580351d0bebea38cf5d4568f1843 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 29 20:00:08 2022 +0100 Fix possible null pointer dereference commit dab670b8d079574d0db4fa0241e9aeb42b38d5fb Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 29 19:42:24 2022 +0100 Remove code related to streaming as it's non-functioning yet commit 96ad161e55f6409458c64050b31a21fcc9adb321 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 29 19:38:07 2022 +0100 Disable Docker Event log collection by default - needs some more testing commit 6903a23c2a762b4a740c5eabce528e54d2f6d105 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 29 19:34:09 2022 +0100 Add collection of docker events attributes commit 1a831afeec4512efbfe2b90919921c3b8d5a0945 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 29 16:07:18 2022 +0100 Add docker events id to log message commit 9fd1c7f79344b7530d71b0870c5323dfb2bfdaa6 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 29 15:10:54 2022 +0100 Fix bugs in docker events logs and add actions collection. Fix a bug of docker events logs not being saved to DB. Fix a bug of docker events timestamp being incorrect when nanoseconds start with zeroes. Add docker events action to log messages. commit b106c9fa00411cd2ce9181b8699d8458b58432fb Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 29 11:41:15 2022 +0100 Exit if netdata build fails commit 01598dae4b7707d34cf5003b2ae8abe9dff77ec3 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 26 17:20:58 2022 +0100 Add collection, parsing and conversion (to datetime) of docker events timestamp commit fb0bdfdcf7601a959f5e3eeef4a5030f11a6cb36 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Aug 25 22:55:48 2022 +0100 WIP: Docker Events Logs: Add some of the metrics exctraction and charts generation functionality for docker events types commit 1200fca057ff3c81ce1127b9ed131c968a48309c Author: Dim-P <dimitris1703@gmail.com> Date: Thu Aug 25 21:01:43 2022 +0100 WIP: Docker Events Logs: Work on adding JSON parser to docker events input commit fd0e9cf5a22f512bddd172878ef4b01a1d8856ec Author: Dim-P <dimitris1703@gmail.com> Date: Wed Aug 24 04:02:33 2022 +0100 WIP: Some more work on docker events logs commit 4d4433538d56ffe3c25ab7ee061d7dc3c56cd8fa Author: Dim-P <dimitris1703@gmail.com> Date: Wed Aug 24 00:06:38 2022 +0100 WIP: Add docker events 'add input' code commit b274b1d61895b677fdaa8d310011ab7d8f08b873 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 19 18:57:47 2022 +0100 Fix some Codacity warnings commit 83545bf2d5b2df3e13a473f6531ee47f017e2fcf Author: Dim-P <dimitris1703@gmail.com> Date: Thu Aug 18 23:51:17 2022 +0100 Add database disk usage chart commit d3e8540413cc37c80b7549bd27cb78f63fabd6ce Author: Dim-P <dimitris1703@gmail.com> Date: Thu Aug 18 21:11:12 2022 +0100 Add explicit FATAL error if a log source DB is not found in main.db commit 2294f618575f96bba6db7482bbbafe985c682e8a Author: Dim-P <dimitris1703@gmail.com> Date: Thu Aug 18 20:51:37 2022 +0100 Small DB code optimisations commit acb11d7da92b9b9a09b0f2d65ae96bc7d604ab71 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Aug 18 17:21:54 2022 +0100 Add const to several pointers that shouldn't be modified commit f429a306848fe291208b2f8b2105f3cfc429f3ad Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 16 19:44:34 2022 +0100 Clean up run_stress_test.sh comments commit 736cbbdf0c2d05cd4a5169c4456556cba8787db7 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 16 19:10:55 2022 +0100 Fix commit revision of mqtt_websockets and aclk/aclk-schemas submodules commit e65847c46bbc5a07f05c93dd800091b7d9a4cc49 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 16 19:01:30 2022 +0100 Add aclk/aclk-schemas submodule that was accidentally removed commit c844758ad0b3c1eb17258decd67fbe7d34374e8b Merge: e533115ac 69ad057b0 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 16 18:54:21 2022 +0100 Merge branch 'master' into logs-manag-master commit e533115ac619e2df881de8cd89a8ba9fe064603d Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 16 17:46:30 2022 +0100 Various improvements to run_Stress_test.sh (see details) Use £INSTALL_PATH wherever possible. Do not attempt to run logs management tests if netdata is configured without the ability to run them. Rename some variables and some constants. commit ea03d9126564b282be0ae914270ee12cf5e7b256 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 16 13:45:24 2022 +0100 Add --help option to run_stress_test.sh commit dfb16a8e7a68dc946cce8dd5b18af16d9666696b Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 16 13:38:43 2022 +0100 Change circ_buff_item_status_t to atomic commit 3cedfb30d3d89f5c13cd04e9e51a38d613396dd6 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 5 16:54:33 2022 +0100 Do not assert in case of failed input initialisation commit 5e9589ec4b523db4bc53e8714a174e7d79d048a8 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 5 16:52:33 2022 +0100 Log flb_init() errors but do not fatal. Fix missing symbol error. commit 8f4b7085875049a203f6ecc500df5a4fe291dc3b Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 5 13:50:33 2022 +0100 Do not fatal() if Fluent-Bit library cannot be accessed commit efa229fd8bfb16c70f4f5e24b8618cbff6230137 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 5 13:37:49 2022 +0100 Remove Flex and Bison library dependencies commit 2539fe7121a75d21300aa008599e4d44c151d801 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 5 01:36:52 2022 +0100 Install Fluent-Bit shared library in /tmp/netdata/ commit cf1ecba22ad48a1c57467d811d3732336c982073 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 5 00:45:43 2022 +0100 Fix msgpack-c linking issues in flb_plugin.c commit 97768d1542b18e05656e079127920350866d5ea2 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 2 20:25:49 2022 +0100 Use dlopen() instead of installed Fluent-Bit shared library commit 6aed3c52557587f0830b222c89086ed8ffa358fa Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 1 22:27:11 2022 +0100 Move log file tailing code to separate source code file and improve documentation commit 34b1c51fcca88acd5595c692c93bf9f8fd80e805 Author: Dim-P <dimitris1703@gmail.com> Date: Wed Jul 6 00:22:38 2022 +0300 Add web API to get list of all log sources commit cf9b10facc589ba18925f0644d4c5a70e9e16750 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Jul 5 20:08:21 2022 +0300 Add more tests for sanitise_string() commit 43ecb8d89ccb1194f2bb357fad69bec8941c03db Author: Dim-P <dimitris1703@gmail.com> Date: Tue Jul 5 19:32:08 2022 +0300 Add missing include of query.h in web api source commit ff970de898884e5a5300049b520089b308eae865 Author: Dim-P <dimitris1703@gmail.com> Date: Sat Jul 2 02:41:30 2022 +0300 Fix web_api_v1.c merge conflicts correctly commit 9c96d432c76e9be5edb841513cf3ba3f8f2b71a3 Merge: 1141c5935 74e547417 Author: Dim-P <dimitris1703@gmail.com> Date: Sat Jul 2 02:31:14 2022 +0300 Merge branch 'netdata-master' into rebase-on-master commit 1141c593570a837c371cd70fd2d1410c1bd8c9a8 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 20:59:46 2022 +0300 Working sanitise_string() test commit 97f0212e0486813bf66da61af8c5d36efdc4f2e5 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 19:58:56 2022 +0300 Add --enable-logsmanagement-tests option to installer to enable unit tests commit ab0cbdc78800692e67a3fda93a4ac08528935ab6 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 19:16:57 2022 +0300 Further reduce supported Fluent-Bit plugins commit 342813932d0a20106d33c93ff4b9e6167343caf7 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 18:59:17 2022 +0300 Add explicit Fluent-Bit configuration and reduce supported plugins. Use max number of CPU cores during Fluent-Bit build. commit 3ab2ef676903703e42ffbc6e1e23cd32f9405988 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 17:27:42 2022 +0300 Add some missing files to CMakeLists.txt commit e3135e221effc8cf21cfbbae450669e9b6e12ace Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 16:00:57 2022 +0300 WIP: Add sanitise_string() unit test commit eb32fb9fc21a8421759ffcfb147186a1324933c1 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 14:05:19 2022 +0300 Add installation of libmsgpack-dev dependency in run_stress_test.sh commit 99d9ea41e51fcb6855f30d039ca0d7883fab181e Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 13:57:25 2022 +0300 Add extra tests cases in test_auto_detect_parse_config() commit 0d560b801ff1b6ea8e5102a3ab3dee6acbd8242f Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 13:21:31 2022 +0300 Add csvVhostCombined log format autodetection and respective unit tests commit 7005c30903135051a4daa0512a6c656fb4309bbe Author: Dim-P <dimitris1703@gmail.com> Date: Thu Jun 30 20:49:21 2022 +0300 Add test_auto_detect_parse_config() unit tests commit 23e9662560f0733d689c42e19a288ea5fa1047c9 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Jun 30 20:48:15 2022 +0300 Fix web log config autodetection not working. Add combined log format in autodetect matrix commit 55844c3bd335a7f296dd9b42a73b6c4189d4ed92 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Jun 30 17:34:40 2022 +0300 Add compression / decompression unit tests commit f6bdbabeb02a26be98fd249c762d6fa85883ae93 Author: Dim-P <Dim-P@users.noreply.github.com> Date: Thu Jun 30 15:47:44 2022 +0300 Remove obsolete --use-system-lws option from netdata-installer.sh help (#13272) Remove obsolete --use-system-lws option from netdata-installer.sh help commit 02b17db85947b495bc7948335a7600900dece821 Author: Dim-P <Dim-P@users.noreply.github.com> Date: Thu Jun 30 14:18:25 2022 +0300 Update to latest netdata/master (#5) * Expose anomaly-bit option to health. (#12835) * Consider ZFS ARC shrinkable as cache on FreeBSD (#12879) * Update dashboard to version v2.25.2. (#12884) Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> * [ci skip] Update changelog and version for nightly build: v1.34.0-148-nightly. * feat(proc/cgroups.plugin): add PSI stall time charts (#12869) * Fix compilation warnings (#12886) * chore(deps): bump docker/setup-buildx-action from 1 to 2 (#12848) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 1 to 2. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](https://github.com/docker/setup-buildx-action/compare/v1...v2) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump docker/setup-qemu-action from 1 to 2 (#12849) Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 1 to 2. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](https://github.com/docker/setup-qemu-action/compare/v1...v2) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump docker/login-action from 1 to 2 (#12850) Bumps [docker/login-action](https://github.com/docker/login-action) from 1 to 2. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](https://github.com/docker/login-action/compare/v1...v2) --- updated-dependencies: - dependency-name: docker/login-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump docker/build-push-action from 2 to 3 (#12851) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 2 to 3. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/v2...v3) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Pause alert pushes to the cloud (#12852) * pause and unpause alert pushes to the cloud * move the check to when creating opcode * check for worker * remove previous checks for dbsync_workers. queue and clean aclk_alert tables even if no workers are up. Get wc then check before setting pause * remove sync_syncronize * remove sync_synchronize_2 * Take into account the in queue wait time when executing a data query (#12885) Take into account the in queue wait time when executing a query with a timeout * [ci skip] Update changelog and version for nightly build: v1.34.0-157-nightly. * Return stable or nightly based on version if the file check fails (#12894) * Implements new capability fields in aclk_schemas (#12602) use new capability fields * Remove CPU-specific info from cpuidle dimensions (#12898) * Update reconnect node with kickstart info (#12891) * Update reconnect node with kickstart info * Split stop from start Second change done because restart may very easily result in the agent failing to connect because of problematic handling of reconnections. * fix(cgroups.plugin): use correct identifier when registering the main thread "chart" worker job (#12902) use main thread chart job registration * chore(worker_utilization): log an error when re-registering an already registered job (#12903) * [ci skip] Update changelog and version for nightly build: v1.34.0-164-nightly. * Fix release channel in the node info message (#12905) Fix release channel in the node info message (was hardcoded) * Fix compilation warnings in FreeBSD (#12887) * [ci skip] Update changelog and version for nightly build: v1.34.0-167-nightly. * added worker jobs for cgroup-rename, cgroup-network and cgroup-first-time (#12910) * chore(streaming): bump default "buffer size bytes" to 10MB (#12913) * [ci skip] Update changelog and version for nightly build: v1.34.0-170-nightly. * fix(cgroups.plugin): improve check for uninitialized containers in k8s (#12912) * fix `[global statistics]` section in netdata.conf (#12916) * fix virtualization detection when `systemd-detect-virt` is not available (#12911) * Fix the log entry for incoming cloud start streaming commands (#12908) Add the correct requested chart sequence id from the cloud and also record the local one we have * user configurable sqlite PRAGMAs (#12917) * user configurable sqlite PRAGMAs * added cache size * chore(cgroups.plugin): remove "enable new cgroups detected at run time" config option (#12906) * Restore a broken symbolic link (#12923) * chore(fping.plugin): bump default fping version to 5.1 (#12930) * chore: add links to SQLite init options in the src code (#12920) * collectors: apps.plugin: apps_groups: update net, aws, ha groups (#12921) * [ci skip] Update changelog and version for nightly build: v1.34.0-181-nightly. * Adjust alarms count (#12896) * check for is_available when counting * remove empty line * Remove Alpine 3.12 from CI. (#12919) It went EOL upstream on 2022-05-01. * Adjust the dimension liveness status check (#12933) * Mark a chart to be exposed only if dimension is created or metadata changes * Add a calculate liveness for the dimension for collected to non collected (live -> stale) and vice versa * queue_dimension_to_aclk will have the rrdset and either 0 or last collected time If 0 then it will be marked as live else it will be marked as stale and last collected time will be sent to the cloud * Add an extra parameter to indicate if the payload check should be done in the database or it has been done already * Queue dimension sets dimension liveness and queues the exact payload to store in the database * Fix compilation error when --disable-cloud is specified * Reduce timeout to 1 second for getting cloud instance info (#12941) * feat: move dirs, logs, and env vars config options to separate sections (#12935) * include proper package dependency (#12518) * Add support for installing static builds on systems without usable internet connections. (#12809) * Add code to prepare an offline installation source directory. * Add support to `download` function for `file://` URLs. This will be used to reference local files instead of having to set up a web server or complicate the code for fetching files. * Modify static archive URL handling to use offline install source if present. * Make check for offline install source support more robust. * Fix fetching of files for offline installs. * Fixed installer stub script. * Mark release channel in offline install source directory. To ensure that users are getting what they expect. * Fix handling of option ordering. * Assorted fixes. * Don’t enable auto-updates by defualt for offline installs. * Update documentation. * Add dry run mode support. * Fix checking of SHA256 checksums. * Fix handling of custom install version selection. * Fix CPU architecture selection. * Fix typo. * Fix typos in user-facing messages * Fix broken merge. * Fix incorrect variable name. * Move kickstart argument parsing code to a function. (#12805) Move kickstart argument arsing code to a function. This will make it slightly more maintainable in the long term, and slightly easier for newcomers to understand. * Fix and improve netdata-updater.sh script (#12757) * minor improvement in compatible DISTRO detection * revert distro detection code; added almalinux and rocky distros as centos compatibility distro * [ci skip] Update changelog and version for nightly build: v1.34.0-191-nightly. * Autodetect channel for specific version (#12856) * autodetect channel * fix kickstart.sh * Prevent command_to_be_logged from overflowing (#12947) * prevent command_to_be_logged from overflowing * dont access with size * Update libbpf version (#12945) * detailed dbengine stats (#12948) * Optimize the dimensions option store to the metadata database (#12952) * Add a flag to "cache" the latest hidden status written in the database * rrddim hide and unhide will check "cached" state, update the database if needed and set the cache flag accordingly * Check the dimension option and only do the database update if the cached state is different * Defer the dimension payload check to the ACLK sync thread (#12951) Defer payload check to the aclk sync thread * [ci skip] Update changelog and version for nightly build: v1.34.0-198-nightly. * Suppress warning when freeing a NULL pointer in onewayalloc_freez (#12955) * Cleanup chart hash and map tables on startup (#12956) * statsd sets should count unique values (#12963) statsd set should count unique values * [ci skip] Update changelog and version for nightly build: v1.34.0-202-nightly. * Apply some logic to possible streaming destinations (#12866) * replace connect_to_one_of with connect_to_one_of_destinations * move functions from socket.c * use sizeof * move current destination pointer to host * formatting * use snprintfz * get entries in same order * handle single destination as before (or when it is the last of the list), instead of skiping it every other loop * try other destinations on ssl problem * fix: keep virtualization unknown if all used commands are not available (#12964) * cleanup and optimize rrdeng_load_metric_next() (#12966) * cleanup and optimize rrdeng_load_metric_next() * fixed typo * Don't permanetly disable a destination because of denied access (#12971) dont make denied destination permanent * feat(charts.d/apcupds): add load usage chart (Watts) (#12965) * chore: check link local address before querying cloud instance metadata (#12973) check link local address before querying cloud providers data * [ci skip] Update changelog and version for nightly build: v1.34.0-209-nightly. * optimize poll_events() to spread the work over the threads more evenly (#12975) * optimize poll_events() to spread the work over the threads more evenly * fixed typos, code cleanup * better error handling * prevent crash in case callbacks manipulate the sockets arrays - added warnings * query engine optimizations and cleanup (#12978) * move number unpacking close to next_metric * dont miss group value flags * [ci skip] Update changelog and version for nightly build: v1.34.0-212-nightly. * chore(apps.plugin): change cpu_guest chart context (#12983) * Add automatic retries fo static builds during nightly and release builds. (#12961) * fix(kickstart.sh): handle the case when `tput colors` doesn't return a number (#12979) * fix: don't kill Netdata PIDs if successfully stopped Netdata (#12982) * modify code to resolve compile warning issue (#12969) * Make heartbeat a static chart (#12986) * Improve reconnect node instructions (#12987) * [ci skip] Update changelog and version for nightly build: v1.34.0-220-nightly. * Faster queries (#12988) * faster rrdeng_load_metric_next() * no need to check validity for number - already done at the query side * solve discrepancy between query create and free * inline unpack_storage_number * Stream and advertise metric correlations to the cloud (#12940) * stream and advertise mc to the cloud * better reporting * remove log * remove aclk debug * Return rc->last_update from alarms_values api (#12968) * add rc last_update as when in alarms values api * rename when to last_updated * update swagger * Fix compilation warnings (#12993) * Fix nanosleep on platforms other than Linux (#12991) * Run the /net/dev module of the proc plugin in a separate thread (#12996) * Update dashboard to version v2.25.4. (#13000) Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> * Don't expose the chart definition to streaming if there is no metadata change (#12990) * Only clear the RRDSET_FLAG_UPSTREAM_EXPOSED chart flag if metadata has changed * Handle modification of units as well * Initialize old_units in the chart state * add dictionary support to statsd (#12980) * add dictionary support to statsd * faster statsd sets and dictionaries; disabled events dimensions by default * properly detect tags, even without a sampling rate * statsd assumes a pipe between fields * missing param * allow names without values and support more unknown fields * more parser fixes * support multiple tags; remove the sum from the dimensions of histograms and timers, but keep it for synthetic charts * Parse statsd tags and support changing units of private charts * remove debug code * added support for naming dimensions too * updated docs * added support for family tags * updated docs * [ci skip] Update changelog and version for nightly build: v1.34.0-230-nightly. * Delay children chart obsoletion check (#12992) * wait untill after 2 minutes of last chart received to run obsoletion check * turn write to read locks * Use printf instead of echo for printing collected warnings in kickstart.sh. (#13002) Use printf instead of echo for printing collected warnings. This ensures that the formatting actually happens correctly. * Autodetect coverity install path to increase robustness (#12995) * [ci skip] Update changelog and version for nightly build: v1.34.0-234-nightly. * Alarms py collector add filtering (#12972) add ability to filter alarms in `alarms.conf` * Add Alpine 3.16 to CI and platform support document. (#12997) * fix COVERITY_PATH added with INSTALL_DIR into PATH (#13014) * Cleanup Challenge Response Code (#11730) - Use OpenSSL for Base64 encode/decode - general cleanup of code * Pass the child machine's guid to the goto_url link (#12609) * pass child machine guid * pass parameter * Only try to update repo metadata in updater script if needed. (#13009) * Cache invariant components in static builds to reduce build times. (#12877) * Add basic build caching support to static builds. Cache is store din `artifacts/cache/${BUILDARCH}`. Each third-party component utilizes a separate build cache. Invalidation is only done for version changes (more rigorous invalidation is expected to be handled externally). * Integrate static build caching with CI. * Fix fping cache handling. * Test caching in CI. * Properly skip rebuilds on cache hits. * Remove static build container when done with it. * Reuse existing image automatically if it’s for the correct platform. * Test CI build caching. * Fix static build job names. * Fix coverity issue 378598 (#13022) move appconfig_get before aclk lock * Initialize a pointer and add a check for it (#13023) * prevent gap filling on dbengine gaps (#13027) * Update dashboard to version v2.25.6. (#13028) Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> * [ci skip] Update changelog and version for nightly build: v1.34.0-246-nightly. * add hostname to mirrored hosts (#13030) * [ci skip] Update changelog and version for nightly build: v1.34.0-248-nightly. * Update apps_groups.conf (#13033) * Trigger queue removed alerts on health log exchange with cloud (#12954) trigger queue removed on health log exchange with cloud * Check return value and log an error on failure (#13037) * [ci skip] Update changelog and version for nightly build: v1.34.0-252-nightly. * Fix coverity issue 378617,378615 (#13021) * Fix CID 378617 * Fix CID 378615 * Make sure the ST rrdr lock indicator is set/reset while holding a lock * Switch to int * Skip collecting network interface speed and duplex if carrier is down (#13019) * When sending a dimension for the first time, make sure there is a non zero created_at timestamp (#13035) * Treat dimensions as normal when we don't have enough/valid data. (#13005) Ideally, we'd log such cases but this is not currently feasible because we have to process thousands of dimensions per second. * replace `history` with relevant `dbengine` params (#13041) * Fix the retry count and netdata_exit check when running an sqlite3_step command (#13040) * Move retry count to the header file * Add SQL_MAX_RETRY count and fix the netdata_exit check * Write pid file (needed by logrotate) (#9377) * Add additional metadata to the data response (#13036) * Consolidate query params * Add new option to show full dimensions in the json header (this will include dimensions, charts and chart labels) * Group and pass parameters with query_params * [ci skip] Update changelog and version for nightly build: v1.34.0-261-nightly. * Fix disabled apps (ebpf.plugin) (#13044) * add note about anomaly advisor (#13042) * Schedule retention message calculation to a worker thread (#13039) * Move aclk_update_retention to the proper header file * Do a scan but avoid going through all the dimensions if we have too much to delete -- do not generate a retention message in that case * Schedule the retention calculation to a worker * Adjust messages in the access log * Fix compilation errors with --disable-cloud * Dictionary with JudyHS and double linked list (#13032) * dictionary internals isolation * more dictionary cleanups * added unit test * we should use DICT internally * disable cups in cmake * implement DICTIONARY with Judy arrays * operational JUDY implementation * JUDY cleanup * JUDY summary added * JudyHS implementation with double linked list * test negative searches too * optimize destruction * optimize set to insert first without lookup * updated stats * code cleanup; better organization; updated info * more code cleanup and commenting * more cleanup, renames and comments * fix rename * more cleanups * use Judy.h from system paths * added foreach traversal; added flag to add item in front; isolated locks to their own functions; destruction returns the number of bytes freed * more comments; flags are now 16-bit * completed unittesting * addressed comments and added reference counters maintainance * added unittest in main; tested removal of items in front, back and middle * added read/write walkthrough and foreach; allowed walkthrough and foreach in write mode to delete the current element (used by cups.plugin); referenced counters removed from the API * DICTFE.name should be const too * added API calls for exposing all statistics * dictionary flags as enum and reference counters as atomic operations * more comments; improved error handling at unit tests * added functions to allow unsafe access while traversing the dictionary with locks in place * check for libcups in cmake * added delete callback; implemented statsd with this dictionary * added missing dfe_done() * added alternative implementation with AVL * added documentation * added comments and warning about AVL * dictionary walktrhough on new code * simplified foreach; updated docs * updated docs * AVL is much faster without hashes * AVL should follow DBENGINE * coverity fixes about statsd; removal of strsame (#13049) * Fix dictionary crash walkthrough empty (#13051) fix dictionary walkthrouhg crash when the dictionary is empty * [ci skip] Update changelog and version for nightly build: v1.34.0-268-nightly. * dictionary improvements (#13052) * fix typo in foreach write; added unit tests to traverse empty dictionaries * rename variable dfe in macro to be uniform with name variable * Check for host labels when linking alerts for children (#13053) check for host labels when linking alerts * Fix coverity 378625 (#13055) free jsonb buffer * add the ability to merge dictionary items (#13054) * add the ability to merge old value and new value * docs * merge to conflict * fix(cgroup.plugin): read k8s_cluster_name label from the correct file (#13062) * Initialize chart label key parameter correctly (#13061) Pass chart label key * [ci skip] Update changelog and version for nightly build: v1.34.0-275-nightly. * update codeowners (#13060) * Fix handling of temp directory in kickstart when uninstalling. (#13056) This makes temporary directory creation idempotent, and ensures it’s invoked in each place that needs it. It also tidies up the cleanup code so that it only tries to remove the temporary directory if one was created. * Fix locking access to chart labels (#13064) No write lock required * [ci skip] Update changelog and version for nightly build: v1.34.0-279-nightly. * Allow usage of the new MQTT 5 implementation (#12838) * adds support for new MQTT5 implementation in agent, currently by default disabled as Tech Preview * Added Alma Linux 9 and RHEL 9 support to CI and packaging. (#13058) * Added Alma Linux 9 and RHEL 9 support to CI and packaging. * Fix RPM deps for Alma 9. * Fix package testing code for Alma 9. * Update default value for "host anomaly rate threshold" (#13075) * Add improved reinstall documentation. (#13047) Including explaining how to do a clean reinstall, and how to switch install types. * Fix handling of DEB package naming in CI. (#13076) * Fix handling of DEB package naming in CI. * Bump repoconfig package versions. To ensure naming consistency. * Fix repoconfig package renaming. * [ci skip] Update changelog and version for nightly build: v1.34.0-285-nightly. * fix(updater): don't produce any output if binpkg update completed successfully (#13081) * don't produce any output when binpkg update succeeded * redirect update repo, update netdata output to 3 * Remove Fedora 34 from CI and package builds. (#12875) * Remove Fedora 34 from CI and package builds. EOL date is 2022-06-07. * Update platform support document. * Fix Coverity errors in mqtt_websockets submodule (#13082) - update mqtt_websockets submodule to bring in the Coverity scan fixes * [ci skip] Update changelog and version for nightly build: v1.34.0-289-nightly. * Update README.md (#13088) * Update README.md (#13089) * fix(updater): return 0 on successful update for native packages when running interactively (#13083) return 0 on success in udpate_binpkg * [ci skip] Minor release v1.35.0. * [ci skip] Update changelog and version for nightly build: v1.35.0-1-nightly. * Temporarily disable updates for static builds. (#13100) * Temporarily disable updates for static builds. * Fix typo in warning message. * Remove unnescesary ‘cleanup’ code. (#13103) This was causing the updter to be unconditionally disabled on static builds, and we don’t actually _need_ any of it anymore since we’ve already decoupled the auto-update handling state from the installation process (in fact, this should have been removed then, but it got missed somehow). * [ci skip] Update changelog and version for nightly build: v1.35.0-4-nightly. * Re-enable updates for systems using static builds. (#13110) Revert "Temporarily disable updates for static builds. (#13100)" This reverts commit b67c1026bae0411e863a227214fd4175ff109d0f. * docs(statsd.plugin): fix indentation (#13096) fix indentation * Add openSUSE Leap 15.4 to CI and package builds. (#12270) Add openSUSE Leap 15.4 t CI and package builds. Expected release date 2022-06-08. * [ci skip] Update changelog and version for nightly build: v1.35.0-8-nightly. * fix virtualization detection on FreeBSD (#13087) * Add user netdata to secondary group in DEB package (#13109) * Add user netdata to secondary group * Update netdata.postinst * Labels with dictionary (#13070) * squashed and rebased to master * fix overflow and single character bug in sanitize; include rrd.h instead of node_info.h * added unittest for UTF-8 multibyte sanitization * Fix unit test compilation * Fix CMake build * remove double sanitizer for opentsdb; cleanup sanitize_json_string() * rename error_description to error_message to avoid conflict with json-c * revert last and undef error_description from json-c * more unittests; attempt to fix protobuf map issue * get rid of rrdlabels_get() and replace it with a safe version that writes the value to a buffer * added dictionary sorting unittest; rrdlabels_to_buffer() now is sorted * better sorted dictionary checking * proper unittesting for sorted dictionaries * call dictionary deletion callback when destroying the dictionary * remove obsolete variable * Fix exporting unit tests * Fix k8s label parsing test * workaround for cmocka and strdupz() * Bypass cmocka memory allocation check * Revert "Bypass cmocka memory allocation check" This reverts commit 4c49923839d9229bea23ca914dd8a0be1ebe2bf4. * Revert "workaround for cmocka and strdupz()" This reverts commit 7bebee04801db1865c748a7896d5fa54bb7104a5. * Bypass cmocka memory allocation checks * respect json formatting for chart labels * cloud sends colons * print the value only once * allow parenthesis in values and spaces; make stream sender send quotes for values Co-authored-by: Vladimir Kobal <vlad@prokk.net> * 73x times faster metrics correlations at the agent (#13107) * faster correlations * 4x times faster correlations * a little bit more help * 10x times faster metrics correlations * 6 digits precision; better comments * enabled metrics correlations by default * abstracted DIFFS_NUMBER to allow easily changing it * reworked the entire logic to have more accuracy and support a baseline that is power of two multiple of highlight * properly calculate shifts * even more improved version * added support for timeout; fixed another memory leak; skipped hidden dimensions * default timeout 1min * reduce memory even further * use dictionary for the list of charts and optimize locks * return 403 forbidden, when mc is not enabled * added query options * dont process zero dimensions * added volume method as an option to metric correlations ; now metric correlations can support multiple implementations * make sure we will never crash * spread results evenly for both kstwo and volume * fixed bug in query engine that was missing misaligned queries when a single point was requested from the db; improved comments; improved query flags * updated swagger and added sane defaults; query options are now supported, including anomaly-bit * added "raw" option to allow cross node correlations; added "group" option to allow different time aggregations; allowed calling metric correlations without any parameters; allowed calling metric correlations with relative timestamps; added timeout to volume method; properly handled timeout on ks2 method; json output now sends all parameters back - same for json_wrap; modified query engine to use present time for relative timestamps; modified "allow_past" to mean both past backwards and forwards * emulate the old behaviour about zero points * 100% accuracy against python ks_2samp(); now the default is volume and the default points are 500 * added config option to change default metric correlations method * removed work-arounds now that rrdlabels are merged * [ci skip] Update changelog and version for nightly build: v1.35.0-13-nightly. * Ci coverage (#13118) * add check to verify autoupdate is still enabled after netdata update * add script to install CI support packages before installing netdata * Fix typo in release base prep script. (#13098) * fixed coveriry 379136 379135 379134 379133 (#13123) * allow label names to have slashes (#13125) * allow label names to have slashes * fixed flipped comment * buffer overflow detected by the compiler (#13120) * buffer overflow detected by the compiler * also increase the buffer +1 * ACLK statistics on bytes recvd and sent (#13091) * fix(python.d/mongodb): set `serverSelectionTimeoutMS` for pymongo4+ (#13135) fix server_selection_timeout for pymongo4+ * fix(python.d): urllib3 import collection for py3.10+ (#13136) * [ci skip] Update changelog and version for nightly build: v1.35.0-22-nightly. * Use a separate thread for slow mountpoints in the diskspace plugin (#13067) * use ks2 as MC default (#13131) * fix crashes due to misaligned allocations (#13137) * Add missing control to streaming (#13112) fix_tls_stream: Add call to SSL_get_error to avoid an infinite loop * Add an option to use malloc for page cache instead of mmap (#13142) Add an option to switch to using malloc for page cache instead of mmap * Update docs on what to do if collector not there (#13152) * [ci skip] Update changelog and version for nightly build: v1.35.0-29-nightly. * Remove pinned page reference (#13108) * Disable reference to prev_descr as we do not keep two pages pinned * Remove extra pinned page from page cache calculations * Removed invalid comment * Remove unused variable * Add cargo/rustc/bazel/buck to apps_groups.conf (#13143) * revert to default of `host anomaly rate threshold=0.01` (#13150) * fix conditions for nightly build triggers (#13145) * Fix labels unit test (#13156) * add CAP_SYS_RAWIO to Netdata's systemd unit CapabilityBoundingSet (#13154) * Add mem.available chart to FreeBSD (#13140) * Configurable storage engine for Netdata agents: step 3 (#12892) * storage engine: add host context API Add a new API to allow storage engines to manage host contexts. * Replace single global context with per-engine global context * Context is full managed by storage engines: a storage engine can use no context, a global engine context, per host contexts, or a mix of these. * Currently, only dbengine uses contexts. Following the current logic, legacy hosts use their own context, while non-legacy hosts share the global context. * storage engine: use empty function instead of null for context ops * rrdhost: don't check return value for void call * rrdhost: create context with host * storage engine: move rrddim ops to rrddim_mem.{c,h} * storage engine: don't use NULL for end-of-list marker * storage engine: fallback to default engine * Use memset to mark the empty words in the quoted_strings_splitter function (#13161) Use memset to terminate the words * Don’t manipulate positional parameters in DEB postinst script. (#13169) * Fix coverity issues (#13168) * [ci skip] Update changelog and version for nightly build: v1.35.0-41-nightly. * enable ml by default (#13158) * enabled ml by default * Fix data query on stale chart (#13159) * Fix data query on stale chart * Remove more checks vs the last timestamp of a point collected * feat(proc/proc_net_dev): add dim per operstate to the "Interface Operational State" chart (#13167) * Revert "Configurable storage engine for Netdata agents: step 3 (#12892)" (#13171) This reverts commit 100a12c6cc01222b1518e5e50d2147f592d8a111. A couple parent/child startup/shutdown scenarios can lead to crashes. * feat(proc/proc_net_dev): add dim per duplex state to the "Interface Duplex State" chart (#13165) * allow traversing null-value dictionaries (#13162) * allow traversing null-value dictionaries * fix lgtm report * void the value too * removed NEVERNULL directive * Fix coverity 378587 (#13024) * check for return value of sysconf * if sysconf fails set OWA_NATURAL_PAGE_SIZE to 4096 * [ci skip] Update changelog and version for nightly build: v1.35.0-49-nightly. * feat(proc/proc_net_dev): add dim per phys link state to the "Interface Physical Link State" chart (#13176) * add dim per carrier state * fix down state * Update dashboard to version v2.26.2. (#13177) Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> * set default for `minimum num samples to train` to `900` (#13174) This will enable first set of initial models to be trained quicker and makes sense now that ml is enabled by default. * add k8s_state dashboard_info (#13181) * [ci skip] Update changelog and version for nightly build: v1.35.0-54-nightly. * Add ml alerts examples (#13173) * add ml alarm examples * Update Makefile.am * add hyperlinks and node level AR example * fix(freebsd.plugin): fix wired/cached/avail memory calculation on FreeBSD with ZFS (#13183) * [ci skip] Update changelog and version for nightly build: v1.35.0-57-nightly. * Update dashboard to version v2.26.5. (#13192) Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> * Query Engine multi-granularity support (and MC improvements) (#13155) * set grouping functions * storage engine should check the validity of timestamps, not the query engine * calculate and store in RRDR anomaly rates for every query * anomaly rate used by volume metric correlations * mc volume should use absolute data, to avoid cancelling effect * return anomaly-rates in jasonwrap with jw-anomaly-rates option to data queries * dont return null on anomaly rates * allow passing group query options from the URL * added countif to the query engine and used it in metric correlations * fix configure * fix countif and anomaly rate percentages * added group_options to metric correlations; updated swagger * added newline at the end of yaml file * always check the time the highlighted window was above/below the highlighted window * properly track time in memory queries * error for internal checks only * moved pack_storage_number() into the storage engines * moved unpack_storage_number() inside the storage engines * remove old comment * pass unit tests * properly detect zero or subnormal values in pack_storage_number() * fill nulls before the value, not after * make sure math.h is included * workaround for isfinite() * fix for isfinite() * faster isfinite() alternative * fix for faster isfinite() alternative * next_metric() now returns end_time too * variable step implemented in a generic way * remove left-over variables * ensure we always complete the wanted number of points * fixes * ensure no infinite loop * mc-volume-improvements: Add information about invalid condition * points should have a duration in the past * removed unneeded info() line * Fix unit tests for exporting engine * new_point should only be checked when it is fetched from the db; better comment about the premature breaking of the main query loop Co-authored-by: Thiago Marques <thiagoftsm@gmail.com> Co-authored-by: Vladimir Kobal <vlad@prokk.net> * Add type label for network interfaces (#13187) * feat(python.d/smartd_log): add 2nd job that tries to read from '/var/lib/smartmontools/' (#13188) * Fix cgroups netdev chart labels (#13200) * Allow for an easy way to do metadata migrations (#13196) Allow for an easy way to migrate metadata to a new database schema (versioning) * Clean sqlite prepared statements on thread shutdown (#13193) * Clean sqlite prepared statements on thread shutdown * Add note for pending statements to finalize * [ci skip] Update changelog and version for nightly build: v1.35.0-65-nightly. * Add configuration for dbengine page fetch timeout and retry count (#13194) * Add configuration for page cache fetch timeout and retry count Change page cache wait default timeout to 3 seconds * Issue info message in the error.log if values not within expected lower range * Fix compilation errors with --disable-dbengine * Print INTERNAL BUG messages only when NETDATA_INTERNAL_CHECKS is enabled (#13207) put INTERNAL BUG messages inside NETDATA_INTERNAL_CHECKS * chore(netdata-installer): remove a call to 'cleanup_old_netdata_updater()' because it is no longer exists (#13189) * [ci skip] Update changelog and version for nightly build: v1.35.0-69-nightly. * Add user plugin dirs to environment (#13203) Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * [ci skip] Update changelog and version for nightly build: v1.35.0-71-nightly. * make configuration example clearer (#13182) make ml configuration example cleare * [ci skip] Update changelog and version for nightly build: v1.35.0-73-nightly. * deduplicate mountinfo based on mount point (#13215) * [ci skip] Update changelog and version for nightly build: v1.35.0-75-nightly. * Use new mqtt implementation as default (#13132) flips mqtt5 to be default instead of mqtt3 * fix(updater): fix updating when using `--force-update` and new version of the updater script is available (#13104) * Ensure tmpdir is set for every function that uses it. (#13206) The set_tmpdir function is idempotent specifically to allow for this type of usage, and this should help avoid bugs involving temporary directory handling going forwards. * Don’t use realpath to find kickstart source path. (#13208) Don’t use realpath to find kcisktart source path. It doesn’t work on macOS and leads to a bogus error message being printed. * Properly handle interactivity in the updater code. (#13209) * Clean up updater option parsing. This will make the remaining changes simpler, and also ease maintenance long-term. * Add updater options to override interactivity detection. * Pass interactivity from kickstart to updater when invoking updater. * Removes Legacy JSON Cloud Protocol Support In Agent (#13111) * removes old protocol support (cloud removed support already) * ci: add issues to the Agent Board project workflow (#13225) * add add issues to Agent Board project workflow * add 'transferred' event * fix(cgroups.plugin): fix qemu VMs and LXC containers name resolution (#13220) * [ci skip] Update changelog and version for nightly build: v1.35.0-84-nightly. * feat(python.d): load modules from user plugin directories (NETDATA_USER_PLUGINS_DIRS) (#13214) * Add more sqlite unittests (#13227) * Bump repoconfig package version used in kickstart.sh (#13235) * Update netdata commands (#13080) * Update netdata commands Adding the `-W buildinfo` options. * Update README.md * Update README.md * Update daemon/README.md Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * Update daemon/README.md Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * also add the change in the daemon command line help message Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud> * remove whitespace Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud> Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * Use correct PAT for add-to-project workflow. (#13234) * Implement PackageCloud cleanup (#13236) * add script for netdata packagecloud cleanup * add script for netdata packagecloud cleanup * remove old package-cloud cleanup code * add workflow for packagecloud cleanup * Debug msg * allow individual matrix jobs to complete independently to each other * perform cleanup without dry-run * remove debug msg * remove cleanup line from the slack notification * Upload packages to new self-hosted repository infrastructure. (#13240) * Match packaging arches to distro architecture names. This will simplify handling of uploads to our new self-hosted packaging infrastructure. * Add package uploads for RPM packages on new infrastructure. * Add package upload support for DEB packages. * Saner tmpdir handling. * Upload fixes. * Differentiate different DEB distros in package names. * Workflow fixes. * netdata doubles (#13217) * netdata doubles * fix cmocka test * fix cmocka test again * fix left-overs of long double to NETDATA_DOUBLE * RRDDIM detached from disk representation; db settings in [db] section of netdata.conf * update the memory before saving * rrdset is now detached from file structures too * on memory mode map, update the memory mapped structures on every iteration * allow RRD_ID_LENGTH_MAX to be changed * granularity secs, back to update every * fix formatting * more formatting * Dictionaries with reference counters and full deletion support during traversal (#13195) * dont use atomic operations when not needed; detect misuse of the the unsafe functions * use relaxed atomic operations for statistics * use relaxed atomic operations for statistics * dictionaries now use reference counters, allowing deletetions of any item while traversing it * added acquire/release interface to dictionaries * added unittest for reference counters * added NETDATA_INTERNAL_CHECKS logs to detect non-exclusive access to crusial parts of the dictionaries * dictionaries cannot be deleted while there are referenced items in them - they will be deleted once the last item gets unreferenced * cleanup * properly cleanup released items * maintain counters for readers and writers; defer all deletes on sorted walkthrough; cleaner internal_error(); * somewhat faster reference counters on single threaded dictionaries * minor optimizations; allow compiling without internal checks * Delay health until obsoletions check is complete (#13239) * wait until obsoletions check is complete to run health * run the checks 5 minutes after agent connect time * Updates the sqlite version in the agent (#13233) New version sqlite Enabling also #define SQLITE_UDL_CAPABLE_PARSER 1 #define SQLITE_ENABLE_UPDATE_DELETE_LIMIT 1 #define SQLITE_OMIT_LOAD_EXTENSION 1 #define SQLITE_ENABLE_DBSTAT_VTAB 1 * [ci skip] Update changelog and version for nightly build: v1.35.0-96-nightly. * Fix build matrix generation for build workflow. (#13254) * Fix build matrix generation for build workflow. * Fix build matrix sorting. * Fix further typos. * fix: fix a base64_encode bug (#13074) * Use old mqtt implementation as default (#13258) mqtt5 as default off * Add fstype labels to disk charts (#13245) Co-authored-by: vkalintiris <vasilis@netdata.cloud> Co-authored-by: Vladimir Kobal <vlad@prokk.net> Co-authored-by: Netdata bot <43409846+netdatabot@users.noreply.github.com> Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> Co-authored-by: netdatabot <bot@netdata.cloud> Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Emmanue…
commit b10bdba0e814bab158ef378f04ef6eb2317ef346 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Oct 25 12:53:28 2022 +0100 Squashed commit of merging 'master' into logs-manag-master: commit 64386fa3839d87a252d24513cd1bdae7dd1c4896 Merge: 96246c9ed 4b9f02da2 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Oct 25 12:44:32 2022 +0100 Merge branch 'master' into logs-manag-master commit 96246c9edf9e4d3f73e2d5fa76653817095fb262 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Oct 21 18:40:49 2022 +0100 Complete refactoring of parser.c and parser.h commit 9d3a7a6aefd4465b1a36af9d9fa2bfcb6e0c5d55 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Oct 21 15:08:15 2022 +0100 Remove log_parser from p_file_info commit d0d8f647005cd1b0b8cbc47fb75d7d2a013b2414 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Oct 21 14:50:28 2022 +0100 Remove all stream-related files commit ce0fb29223f27f8e28dc513184753aa3754ae5ca Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 20 20:48:34 2022 +0100 Refactor Log_parser_metrics_t to use a union to reduce memory consumption and code complexity commit 89ada4e49e72eba0da06b2512564a28339d44bce Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 20 18:55:51 2022 +0100 Change logsmanagement_conf.h name and structure commit 0ca25f062958811511dcac16c7a9c258bfc9222e Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 20 15:54:25 2022 +0100 Fix wrong timer interval after 'update_every' changes commit fa6fa09f3bd0eef75510c946a4fed418eb61fe61 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 13 19:48:31 2022 +0100 Add update every option per log source commit 003fb181af71d1bf3acc0a1b7bd5e6832a0adc2b Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 13 19:11:29 2022 +0100 Add logs management global update every config option commit 41cef31e61b2658f6774c2df22b604e0663eb3a3 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 13 18:34:19 2022 +0100 Escape potential backslashes in regex to ensure dimenion name in chart is displayed correctly commit b5b3c168be7310039173d6e678093fd784bd3e11 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Oct 10 22:26:13 2022 +0100 Add ability to define names for regex dimensions commit 1416f0900c789e914dbca2bc104cea8b291ddc7b Author: Dim-P <dimitris1703@gmail.com> Date: Mon Oct 10 17:50:30 2022 +0100 Silence compiler truncation warnings commit 2af527a566435cc47f8f8a56cfa3b110b0c243d6 Author: Dim-P <dimitris1703@gmail.com> Date: Sat Oct 8 03:38:14 2022 +0100 Fix wrong timestamps on 32-bit OS commit 477338bfe48d65793c1312580fe4f5b962f0702d Author: Dim-P <dimitris1703@gmail.com> Date: Sat Oct 8 03:04:09 2022 +0100 Fix failing build on Rasberry Pi commit 475ed2c6db3a6d83d15ef69f464b1268c5f62d3d Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 6 20:34:54 2022 +0100 Add custom regex-based charts commit 27ecb125cf30d854e2593d91be5c43428f8f4bfd Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 6 15:21:30 2022 +0100 Add generic log charts commit b7ae826e9677ce6cf51ebfc67d0166b5ab16cdc0 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Oct 6 13:47:32 2022 +0100 Add metrics extraction for generic log parser commit 9f71f6ff676258253277d1e3c4d9b9e7b59550d0 Author: Dim-P <dimitris1703@gmail.com> Date: Wed Oct 5 23:23:01 2022 +0100 Fix SIGSEGV due to uninitialised regex commit ed3715928f57834490fc559dd46c6c4ada543317 Author: Dim-P <dimitris1703@gmail.com> Date: Wed Oct 5 22:56:25 2022 +0100 WIP: Add regex for generic log parser commit 9a4349067e17d2c46924600eda319710792a4866 Author: Dim-P <dimitris1703@gmail.com> Date: Wed Oct 5 20:46:48 2022 +0100 Replace last \n with \0 in collected log batches commit 0ed60ae1f89847fffd6bd25a9211b586a378c0b1 Author: Dim-P <dimitris1703@gmail.com> Date: Wed Oct 5 18:47:49 2022 +0100 Remove some debug prints commit 35e6045d642530d11798f09cf4e4d61a3232182f Author: Dim-P <dimitris1703@gmail.com> Date: Fri Sep 30 16:00:25 2022 +0100 Add generic log support. Add custom chart config init commit b4a601338735822c565db34bd56a4ec1ba413222 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 29 21:45:08 2022 +0100 Comment out some debug log prints commit 1268b0c43258b00e7ad0a9df529ff2da6686f38d Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 29 21:20:12 2022 +0100 Complete refactoring of logsmanagement.plugin directory commit 06018c5f848b3089f8e152762d9600703af68399 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 29 12:09:54 2022 +0100 Move vhosts and ports metrics collection commit c769d3e9badae3573fbf9ef6537edb3f6c8d8ebd Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 27 22:37:26 2022 +0100 WIP: Further moving / refactoring of plugin_logsmanagement.c code commit e74c8361506f3e76978240402591a02066b58dad Author: Dim-P <dimitris1703@gmail.com> Date: Fri Sep 23 20:07:28 2022 +0100 WIP: Breakdown plugin_logsmanagement.c in multiple files and start copying commit 910eaa39c0f0ec7a42014932dcaab7299534a0b2 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 22 21:12:29 2022 +0100 Add global config option to change default logs management db dir commit 94c51d851619f760fad0022b7f605e7667a95092 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 22 19:56:36 2022 +0100 Add global config option to disable logs management commit c828be0ec17cfc50e16dd1e1bbe6a763eb4b8dd2 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 22 19:15:16 2022 +0100 Add logsmanagement.plugin worker commit 713ccf9b98f9e846ce62477763204ebaba31f3ea Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 22 17:42:35 2022 +0100 Fix conversion of STRING to str in logsmanagement.plugin commit 11b356b2fa7e06213b434af36b35dabc5f226b78 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 20 21:23:13 2022 +0100 Add count of log records matching keyword in query results commit 7fab02c027de27b2cd5918a8f87ae8d82c69ea2f Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 20 16:50:10 2022 +0100 Add measure of query execution time commit ef9d5c1df848638e8c4371cbac75443965608e8a Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 20 15:56:04 2022 +0100 Working db_search_compound() queries commit 2385a12659aea49e2387e208388f4b6043219fde Author: Dim-P <dimitris1703@gmail.com> Date: Fri Sep 16 22:15:35 2022 +0100 WIP: Resolve issues with database detaching failures commit 1a9e5348d9fb37c4fe6b2a3fa2c93c10c3d4cffc Author: Dim-P <dimitris1703@gmail.com> Date: Fri Sep 16 16:13:41 2022 +0100 Working TEMP VIEW for db_search_compound() commit 4301cadf8fa83378f3f3b922cd5747736471d6e1 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 15 21:40:27 2022 +0100 WIP: Create TEMP VIEW db_search_compound() query commit 3f7e6823c798158810626b638f7e8158fcb24205 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 15 12:36:52 2022 +0100 Databases attach / detach works commit 108040285abe74696112301ae21a5c86c3d3406e Author: Dim-P <dimitris1703@gmail.com> Date: Mon Sep 12 21:55:23 2022 +0100 WIP: Compound queries commit f387e1a9ada7b12a56f35faf3eee1cfb0726126e Merge: cab26f071 eddb7125f Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 22 16:03:31 2022 +0100 Merge branch 'master' into logs-manag-master commit cab26f071d42a9e3f8f7f55693d2f8db51bf5461 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 6 21:18:55 2022 +0100 Fix backslashes not escaped correctly in JSON query response. commit 5baa4ef0e149cfadeab219abdab1c301c26a434b Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 6 20:30:36 2022 +0100 Fix double quotes in logs not escaped correctly in query response JSON. commit 8edd975dd39ac4a361dfd24d81b1506339534967 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 6 14:47:18 2022 +0100 Update main.db schema to include Machine_GUID and log type commit 37e171415abca17f4c2c21276639f07e7039e702 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 6 14:46:36 2022 +0100 Fix bug of docker events and systemd conflicting DBs if using default paths commit f5466d210bea8f5bd120bc0884258d755abe4efe Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 6 14:45:40 2022 +0100 Fix wrong log type for FLB_DOCKER_EV. commit 46f22124eb81568310a78855f3672d32a30a6edc Author: Dim-P <dimitris1703@gmail.com> Date: Tue Sep 6 00:23:45 2022 +0100 Fix bug of conflicting tags for Fluent-Bit inputs/outputs commit 3de72c88e3ad6154e5c636fbd9e2269220adac98 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Sep 5 19:59:26 2022 +0100 Add option to pass on log source path for docker events and systemd logs commit 2d9598f3b39d8542a6204745752a12877b8539ad Author: Dim-P <dimitris1703@gmail.com> Date: Mon Sep 5 16:11:28 2022 +0100 Re-add git clean and submodule update commit a738e0cd152120554120ea77e220a49ebfbcb6d4 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Sep 5 15:36:03 2022 +0100 Update run_stress_test.sh help commit e0075c649764413a7b6949928c83c14d7bb8c1dc Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 1 21:50:24 2022 +0100 Set default access.log log type to flb_web_log commit f2dfb6746321ea2962cdce601f6dc7b1b1691fc1 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 1 19:26:58 2022 +0100 Fix SIGSEGV on web log auto format detection. Add more unit test cases commit 04b162f237037e9a416aa0e2118fd09ef2cf47d8 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 1 17:21:23 2022 +0100 Do not run stress_test unless explicitly asked to do so commit b18d643dc94aa1b130541dda1627a955c656e1d4 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 1 17:20:13 2022 +0100 Comment out docker_ev_action_string[] since not used yet commit 9a5742eca9ce620155c5e438b7c9ce8521124aba Author: Dim-P <dimitris1703@gmail.com> Date: Thu Sep 1 17:17:35 2022 +0100 Improve loading of logsmanagement.conf Load stock logsmanagement.conf if user-defined one is not available. If stock logsmanagement.conf is not found, then do not start logs management engine at all. commit 230ab034e3c4580351d0bebea38cf5d4568f1843 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 29 20:00:08 2022 +0100 Fix possible null pointer dereference commit dab670b8d079574d0db4fa0241e9aeb42b38d5fb Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 29 19:42:24 2022 +0100 Remove code related to streaming as it's non-functioning yet commit 96ad161e55f6409458c64050b31a21fcc9adb321 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 29 19:38:07 2022 +0100 Disable Docker Event log collection by default - needs some more testing commit 6903a23c2a762b4a740c5eabce528e54d2f6d105 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 29 19:34:09 2022 +0100 Add collection of docker events attributes commit 1a831afeec4512efbfe2b90919921c3b8d5a0945 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 29 16:07:18 2022 +0100 Add docker events id to log message commit 9fd1c7f79344b7530d71b0870c5323dfb2bfdaa6 Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 29 15:10:54 2022 +0100 Fix bugs in docker events logs and add actions collection. Fix a bug of docker events logs not being saved to DB. Fix a bug of docker events timestamp being incorrect when nanoseconds start with zeroes. Add docker events action to log messages. commit b106c9fa00411cd2ce9181b8699d8458b58432fb Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 29 11:41:15 2022 +0100 Exit if netdata build fails commit 01598dae4b7707d34cf5003b2ae8abe9dff77ec3 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 26 17:20:58 2022 +0100 Add collection, parsing and conversion (to datetime) of docker events timestamp commit fb0bdfdcf7601a959f5e3eeef4a5030f11a6cb36 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Aug 25 22:55:48 2022 +0100 WIP: Docker Events Logs: Add some of the metrics exctraction and charts generation functionality for docker events types commit 1200fca057ff3c81ce1127b9ed131c968a48309c Author: Dim-P <dimitris1703@gmail.com> Date: Thu Aug 25 21:01:43 2022 +0100 WIP: Docker Events Logs: Work on adding JSON parser to docker events input commit fd0e9cf5a22f512bddd172878ef4b01a1d8856ec Author: Dim-P <dimitris1703@gmail.com> Date: Wed Aug 24 04:02:33 2022 +0100 WIP: Some more work on docker events logs commit 4d4433538d56ffe3c25ab7ee061d7dc3c56cd8fa Author: Dim-P <dimitris1703@gmail.com> Date: Wed Aug 24 00:06:38 2022 +0100 WIP: Add docker events 'add input' code commit b274b1d61895b677fdaa8d310011ab7d8f08b873 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 19 18:57:47 2022 +0100 Fix some Codacity warnings commit 83545bf2d5b2df3e13a473f6531ee47f017e2fcf Author: Dim-P <dimitris1703@gmail.com> Date: Thu Aug 18 23:51:17 2022 +0100 Add database disk usage chart commit d3e8540413cc37c80b7549bd27cb78f63fabd6ce Author: Dim-P <dimitris1703@gmail.com> Date: Thu Aug 18 21:11:12 2022 +0100 Add explicit FATAL error if a log source DB is not found in main.db commit 2294f618575f96bba6db7482bbbafe985c682e8a Author: Dim-P <dimitris1703@gmail.com> Date: Thu Aug 18 20:51:37 2022 +0100 Small DB code optimisations commit acb11d7da92b9b9a09b0f2d65ae96bc7d604ab71 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Aug 18 17:21:54 2022 +0100 Add const to several pointers that shouldn't be modified commit f429a306848fe291208b2f8b2105f3cfc429f3ad Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 16 19:44:34 2022 +0100 Clean up run_stress_test.sh comments commit 736cbbdf0c2d05cd4a5169c4456556cba8787db7 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 16 19:10:55 2022 +0100 Fix commit revision of mqtt_websockets and aclk/aclk-schemas submodules commit e65847c46bbc5a07f05c93dd800091b7d9a4cc49 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 16 19:01:30 2022 +0100 Add aclk/aclk-schemas submodule that was accidentally removed commit c844758ad0b3c1eb17258decd67fbe7d34374e8b Merge: e533115ac 69ad057b0 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 16 18:54:21 2022 +0100 Merge branch 'master' into logs-manag-master commit e533115ac619e2df881de8cd89a8ba9fe064603d Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 16 17:46:30 2022 +0100 Various improvements to run_Stress_test.sh (see details) Use £INSTALL_PATH wherever possible. Do not attempt to run logs management tests if netdata is configured without the ability to run them. Rename some variables and some constants. commit ea03d9126564b282be0ae914270ee12cf5e7b256 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 16 13:45:24 2022 +0100 Add --help option to run_stress_test.sh commit dfb16a8e7a68dc946cce8dd5b18af16d9666696b Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 16 13:38:43 2022 +0100 Change circ_buff_item_status_t to atomic commit 3cedfb30d3d89f5c13cd04e9e51a38d613396dd6 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 5 16:54:33 2022 +0100 Do not assert in case of failed input initialisation commit 5e9589ec4b523db4bc53e8714a174e7d79d048a8 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 5 16:52:33 2022 +0100 Log flb_init() errors but do not fatal. Fix missing symbol error. commit 8f4b7085875049a203f6ecc500df5a4fe291dc3b Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 5 13:50:33 2022 +0100 Do not fatal() if Fluent-Bit library cannot be accessed commit efa229fd8bfb16c70f4f5e24b8618cbff6230137 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 5 13:37:49 2022 +0100 Remove Flex and Bison library dependencies commit 2539fe7121a75d21300aa008599e4d44c151d801 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 5 01:36:52 2022 +0100 Install Fluent-Bit shared library in /tmp/netdata/ commit cf1ecba22ad48a1c57467d811d3732336c982073 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Aug 5 00:45:43 2022 +0100 Fix msgpack-c linking issues in flb_plugin.c commit 97768d1542b18e05656e079127920350866d5ea2 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Aug 2 20:25:49 2022 +0100 Use dlopen() instead of installed Fluent-Bit shared library commit 6aed3c52557587f0830b222c89086ed8ffa358fa Author: Dim-P <dimitris1703@gmail.com> Date: Mon Aug 1 22:27:11 2022 +0100 Move log file tailing code to separate source code file and improve documentation commit 34b1c51fcca88acd5595c692c93bf9f8fd80e805 Author: Dim-P <dimitris1703@gmail.com> Date: Wed Jul 6 00:22:38 2022 +0300 Add web API to get list of all log sources commit cf9b10facc589ba18925f0644d4c5a70e9e16750 Author: Dim-P <dimitris1703@gmail.com> Date: Tue Jul 5 20:08:21 2022 +0300 Add more tests for sanitise_string() commit 43ecb8d89ccb1194f2bb357fad69bec8941c03db Author: Dim-P <dimitris1703@gmail.com> Date: Tue Jul 5 19:32:08 2022 +0300 Add missing include of query.h in web api source commit ff970de898884e5a5300049b520089b308eae865 Author: Dim-P <dimitris1703@gmail.com> Date: Sat Jul 2 02:41:30 2022 +0300 Fix web_api_v1.c merge conflicts correctly commit 9c96d432c76e9be5edb841513cf3ba3f8f2b71a3 Merge: 1141c5935 74e547417 Author: Dim-P <dimitris1703@gmail.com> Date: Sat Jul 2 02:31:14 2022 +0300 Merge branch 'netdata-master' into rebase-on-master commit 1141c593570a837c371cd70fd2d1410c1bd8c9a8 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 20:59:46 2022 +0300 Working sanitise_string() test commit 97f0212e0486813bf66da61af8c5d36efdc4f2e5 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 19:58:56 2022 +0300 Add --enable-logsmanagement-tests option to installer to enable unit tests commit ab0cbdc78800692e67a3fda93a4ac08528935ab6 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 19:16:57 2022 +0300 Further reduce supported Fluent-Bit plugins commit 342813932d0a20106d33c93ff4b9e6167343caf7 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 18:59:17 2022 +0300 Add explicit Fluent-Bit configuration and reduce supported plugins. Use max number of CPU cores during Fluent-Bit build. commit 3ab2ef676903703e42ffbc6e1e23cd32f9405988 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 17:27:42 2022 +0300 Add some missing files to CMakeLists.txt commit e3135e221effc8cf21cfbbae450669e9b6e12ace Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 16:00:57 2022 +0300 WIP: Add sanitise_string() unit test commit eb32fb9fc21a8421759ffcfb147186a1324933c1 Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 14:05:19 2022 +0300 Add installation of libmsgpack-dev dependency in run_stress_test.sh commit 99d9ea41e51fcb6855f30d039ca0d7883fab181e Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 13:57:25 2022 +0300 Add extra tests cases in test_auto_detect_parse_config() commit 0d560b801ff1b6ea8e5102a3ab3dee6acbd8242f Author: Dim-P <dimitris1703@gmail.com> Date: Fri Jul 1 13:21:31 2022 +0300 Add csvVhostCombined log format autodetection and respective unit tests commit 7005c30903135051a4daa0512a6c656fb4309bbe Author: Dim-P <dimitris1703@gmail.com> Date: Thu Jun 30 20:49:21 2022 +0300 Add test_auto_detect_parse_config() unit tests commit 23e9662560f0733d689c42e19a288ea5fa1047c9 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Jun 30 20:48:15 2022 +0300 Fix web log config autodetection not working. Add combined log format in autodetect matrix commit 55844c3bd335a7f296dd9b42a73b6c4189d4ed92 Author: Dim-P <dimitris1703@gmail.com> Date: Thu Jun 30 17:34:40 2022 +0300 Add compression / decompression unit tests commit f6bdbabeb02a26be98fd249c762d6fa85883ae93 Author: Dim-P <Dim-P@users.noreply.github.com> Date: Thu Jun 30 15:47:44 2022 +0300 Remove obsolete --use-system-lws option from netdata-installer.sh help (#13272) Remove obsolete --use-system-lws option from netdata-installer.sh help commit 02b17db85947b495bc7948335a7600900dece821 Author: Dim-P <Dim-P@users.noreply.github.com> Date: Thu Jun 30 14:18:25 2022 +0300 Update to latest netdata/master (#5) * Expose anomaly-bit option to health. (#12835) * Consider ZFS ARC shrinkable as cache on FreeBSD (#12879) * Update dashboard to version v2.25.2. (#12884) Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> * [ci skip] Update changelog and version for nightly build: v1.34.0-148-nightly. * feat(proc/cgroups.plugin): add PSI stall time charts (#12869) * Fix compilation warnings (#12886) * chore(deps): bump docker/setup-buildx-action from 1 to 2 (#12848) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 1 to 2. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](https://github.com/docker/setup-buildx-action/compare/v1...v2) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump docker/setup-qemu-action from 1 to 2 (#12849) Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 1 to 2. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](https://github.com/docker/setup-qemu-action/compare/v1...v2) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump docker/login-action from 1 to 2 (#12850) Bumps [docker/login-action](https://github.com/docker/login-action) from 1 to 2. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](https://github.com/docker/login-action/compare/v1...v2) --- updated-dependencies: - dependency-name: docker/login-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump docker/build-push-action from 2 to 3 (#12851) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 2 to 3. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/v2...v3) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Pause alert pushes to the cloud (#12852) * pause and unpause alert pushes to the cloud * move the check to when creating opcode * check for worker * remove previous checks for dbsync_workers. queue and clean aclk_alert tables even if no workers are up. Get wc then check before setting pause * remove sync_syncronize * remove sync_synchronize_2 * Take into account the in queue wait time when executing a data query (#12885) Take into account the in queue wait time when executing a query with a timeout * [ci skip] Update changelog and version for nightly build: v1.34.0-157-nightly. * Return stable or nightly based on version if the file check fails (#12894) * Implements new capability fields in aclk_schemas (#12602) use new capability fields * Remove CPU-specific info from cpuidle dimensions (#12898) * Update reconnect node with kickstart info (#12891) * Update reconnect node with kickstart info * Split stop from start Second change done because restart may very easily result in the agent failing to connect because of problematic handling of reconnections. * fix(cgroups.plugin): use correct identifier when registering the main thread "chart" worker job (#12902) use main thread chart job registration * chore(worker_utilization): log an error when re-registering an already registered job (#12903) * [ci skip] Update changelog and version for nightly build: v1.34.0-164-nightly. * Fix release channel in the node info message (#12905) Fix release channel in the node info message (was hardcoded) * Fix compilation warnings in FreeBSD (#12887) * [ci skip] Update changelog and version for nightly build: v1.34.0-167-nightly. * added worker jobs for cgroup-rename, cgroup-network and cgroup-first-time (#12910) * chore(streaming): bump default "buffer size bytes" to 10MB (#12913) * [ci skip] Update changelog and version for nightly build: v1.34.0-170-nightly. * fix(cgroups.plugin): improve check for uninitialized containers in k8s (#12912) * fix `[global statistics]` section in netdata.conf (#12916) * fix virtualization detection when `systemd-detect-virt` is not available (#12911) * Fix the log entry for incoming cloud start streaming commands (#12908) Add the correct requested chart sequence id from the cloud and also record the local one we have * user configurable sqlite PRAGMAs (#12917) * user configurable sqlite PRAGMAs * added cache size * chore(cgroups.plugin): remove "enable new cgroups detected at run time" config option (#12906) * Restore a broken symbolic link (#12923) * chore(fping.plugin): bump default fping version to 5.1 (#12930) * chore: add links to SQLite init options in the src code (#12920) * collectors: apps.plugin: apps_groups: update net, aws, ha groups (#12921) * [ci skip] Update changelog and version for nightly build: v1.34.0-181-nightly. * Adjust alarms count (#12896) * check for is_available when counting * remove empty line * Remove Alpine 3.12 from CI. (#12919) It went EOL upstream on 2022-05-01. * Adjust the dimension liveness status check (#12933) * Mark a chart to be exposed only if dimension is created or metadata changes * Add a calculate liveness for the dimension for collected to non collected (live -> stale) and vice versa * queue_dimension_to_aclk will have the rrdset and either 0 or last collected time If 0 then it will be marked as live else it will be marked as stale and last collected time will be sent to the cloud * Add an extra parameter to indicate if the payload check should be done in the database or it has been done already * Queue dimension sets dimension liveness and queues the exact payload to store in the database * Fix compilation error when --disable-cloud is specified * Reduce timeout to 1 second for getting cloud instance info (#12941) * feat: move dirs, logs, and env vars config options to separate sections (#12935) * include proper package dependency (#12518) * Add support for installing static builds on systems without usable internet connections. (#12809) * Add code to prepare an offline installation source directory. * Add support to `download` function for `file://` URLs. This will be used to reference local files instead of having to set up a web server or complicate the code for fetching files. * Modify static archive URL handling to use offline install source if present. * Make check for offline install source support more robust. * Fix fetching of files for offline installs. * Fixed installer stub script. * Mark release channel in offline install source directory. To ensure that users are getting what they expect. * Fix handling of option ordering. * Assorted fixes. * Don’t enable auto-updates by defualt for offline installs. * Update documentation. * Add dry run mode support. * Fix checking of SHA256 checksums. * Fix handling of custom install version selection. * Fix CPU architecture selection. * Fix typo. * Fix typos in user-facing messages * Fix broken merge. * Fix incorrect variable name. * Move kickstart argument parsing code to a function. (#12805) Move kickstart argument arsing code to a function. This will make it slightly more maintainable in the long term, and slightly easier for newcomers to understand. * Fix and improve netdata-updater.sh script (#12757) * minor improvement in compatible DISTRO detection * revert distro detection code; added almalinux and rocky distros as centos compatibility distro * [ci skip] Update changelog and version for nightly build: v1.34.0-191-nightly. * Autodetect channel for specific version (#12856) * autodetect channel * fix kickstart.sh * Prevent command_to_be_logged from overflowing (#12947) * prevent command_to_be_logged from overflowing * dont access with size * Update libbpf version (#12945) * detailed dbengine stats (#12948) * Optimize the dimensions option store to the metadata database (#12952) * Add a flag to "cache" the latest hidden status written in the database * rrddim hide and unhide will check "cached" state, update the database if needed and set the cache flag accordingly * Check the dimension option and only do the database update if the cached state is different * Defer the dimension payload check to the ACLK sync thread (#12951) Defer payload check to the aclk sync thread * [ci skip] Update changelog and version for nightly build: v1.34.0-198-nightly. * Suppress warning when freeing a NULL pointer in onewayalloc_freez (#12955) * Cleanup chart hash and map tables on startup (#12956) * statsd sets should count unique values (#12963) statsd set should count unique values * [ci skip] Update changelog and version for nightly build: v1.34.0-202-nightly. * Apply some logic to possible streaming destinations (#12866) * replace connect_to_one_of with connect_to_one_of_destinations * move functions from socket.c * use sizeof * move current destination pointer to host * formatting * use snprintfz * get entries in same order * handle single destination as before (or when it is the last of the list), instead of skiping it every other loop * try other destinations on ssl problem * fix: keep virtualization unknown if all used commands are not available (#12964) * cleanup and optimize rrdeng_load_metric_next() (#12966) * cleanup and optimize rrdeng_load_metric_next() * fixed typo * Don't permanetly disable a destination because of denied access (#12971) dont make denied destination permanent * feat(charts.d/apcupds): add load usage chart (Watts) (#12965) * chore: check link local address before querying cloud instance metadata (#12973) check link local address before querying cloud providers data * [ci skip] Update changelog and version for nightly build: v1.34.0-209-nightly. * optimize poll_events() to spread the work over the threads more evenly (#12975) * optimize poll_events() to spread the work over the threads more evenly * fixed typos, code cleanup * better error handling * prevent crash in case callbacks manipulate the sockets arrays - added warnings * query engine optimizations and cleanup (#12978) * move number unpacking close to next_metric * dont miss group value flags * [ci skip] Update changelog and version for nightly build: v1.34.0-212-nightly. * chore(apps.plugin): change cpu_guest chart context (#12983) * Add automatic retries fo static builds during nightly and release builds. (#12961) * fix(kickstart.sh): handle the case when `tput colors` doesn't return a number (#12979) * fix: don't kill Netdata PIDs if successfully stopped Netdata (#12982) * modify code to resolve compile warning issue (#12969) * Make heartbeat a static chart (#12986) * Improve reconnect node instructions (#12987) * [ci skip] Update changelog and version for nightly build: v1.34.0-220-nightly. * Faster queries (#12988) * faster rrdeng_load_metric_next() * no need to check validity for number - already done at the query side * solve discrepancy between query create and free * inline unpack_storage_number * Stream and advertise metric correlations to the cloud (#12940) * stream and advertise mc to the cloud * better reporting * remove log * remove aclk debug * Return rc->last_update from alarms_values api (#12968) * add rc last_update as when in alarms values api * rename when to last_updated * update swagger * Fix compilation warnings (#12993) * Fix nanosleep on platforms other than Linux (#12991) * Run the /net/dev module of the proc plugin in a separate thread (#12996) * Update dashboard to version v2.25.4. (#13000) Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> * Don't expose the chart definition to streaming if there is no metadata change (#12990) * Only clear the RRDSET_FLAG_UPSTREAM_EXPOSED chart flag if metadata has changed * Handle modification of units as well * Initialize old_units in the chart state * add dictionary support to statsd (#12980) * add dictionary support to statsd * faster statsd sets and dictionaries; disabled events dimensions by default * properly detect tags, even without a sampling rate * statsd assumes a pipe between fields * missing param * allow names without values and support more unknown fields * more parser fixes * support multiple tags; remove the sum from the dimensions of histograms and timers, but keep it for synthetic charts * Parse statsd tags and support changing units of private charts * remove debug code * added support for naming dimensions too * updated docs * added support for family tags * updated docs * [ci skip] Update changelog and version for nightly build: v1.34.0-230-nightly. * Delay children chart obsoletion check (#12992) * wait untill after 2 minutes of last chart received to run obsoletion check * turn write to read locks * Use printf instead of echo for printing collected warnings in kickstart.sh. (#13002) Use printf instead of echo for printing collected warnings. This ensures that the formatting actually happens correctly. * Autodetect coverity install path to increase robustness (#12995) * [ci skip] Update changelog and version for nightly build: v1.34.0-234-nightly. * Alarms py collector add filtering (#12972) add ability to filter alarms in `alarms.conf` * Add Alpine 3.16 to CI and platform support document. (#12997) * fix COVERITY_PATH added with INSTALL_DIR into PATH (#13014) * Cleanup Challenge Response Code (#11730) - Use OpenSSL for Base64 encode/decode - general cleanup of code * Pass the child machine's guid to the goto_url link (#12609) * pass child machine guid * pass parameter * Only try to update repo metadata in updater script if needed. (#13009) * Cache invariant components in static builds to reduce build times. (#12877) * Add basic build caching support to static builds. Cache is store din `artifacts/cache/${BUILDARCH}`. Each third-party component utilizes a separate build cache. Invalidation is only done for version changes (more rigorous invalidation is expected to be handled externally). * Integrate static build caching with CI. * Fix fping cache handling. * Test caching in CI. * Properly skip rebuilds on cache hits. * Remove static build container when done with it. * Reuse existing image automatically if it’s for the correct platform. * Test CI build caching. * Fix static build job names. * Fix coverity issue 378598 (#13022) move appconfig_get before aclk lock * Initialize a pointer and add a check for it (#13023) * prevent gap filling on dbengine gaps (#13027) * Update dashboard to version v2.25.6. (#13028) Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> * [ci skip] Update changelog and version for nightly build: v1.34.0-246-nightly. * add hostname to mirrored hosts (#13030) * [ci skip] Update changelog and version for nightly build: v1.34.0-248-nightly. * Update apps_groups.conf (#13033) * Trigger queue removed alerts on health log exchange with cloud (#12954) trigger queue removed on health log exchange with cloud * Check return value and log an error on failure (#13037) * [ci skip] Update changelog and version for nightly build: v1.34.0-252-nightly. * Fix coverity issue 378617,378615 (#13021) * Fix CID 378617 * Fix CID 378615 * Make sure the ST rrdr lock indicator is set/reset while holding a lock * Switch to int * Skip collecting network interface speed and duplex if carrier is down (#13019) * When sending a dimension for the first time, make sure there is a non zero created_at timestamp (#13035) * Treat dimensions as normal when we don't have enough/valid data. (#13005) Ideally, we'd log such cases but this is not currently feasible because we have to process thousands of dimensions per second. * replace `history` with relevant `dbengine` params (#13041) * Fix the retry count and netdata_exit check when running an sqlite3_step command (#13040) * Move retry count to the header file * Add SQL_MAX_RETRY count and fix the netdata_exit check * Write pid file (needed by logrotate) (#9377) * Add additional metadata to the data response (#13036) * Consolidate query params * Add new option to show full dimensions in the json header (this will include dimensions, charts and chart labels) * Group and pass parameters with query_params * [ci skip] Update changelog and version for nightly build: v1.34.0-261-nightly. * Fix disabled apps (ebpf.plugin) (#13044) * add note about anomaly advisor (#13042) * Schedule retention message calculation to a worker thread (#13039) * Move aclk_update_retention to the proper header file * Do a scan but avoid going through all the dimensions if we have too much to delete -- do not generate a retention message in that case * Schedule the retention calculation to a worker * Adjust messages in the access log * Fix compilation errors with --disable-cloud * Dictionary with JudyHS and double linked list (#13032) * dictionary internals isolation * more dictionary cleanups * added unit test * we should use DICT internally * disable cups in cmake * implement DICTIONARY with Judy arrays * operational JUDY implementation * JUDY cleanup * JUDY summary added * JudyHS implementation with double linked list * test negative searches too * optimize destruction * optimize set to insert first without lookup * updated stats * code cleanup; better organization; updated info * more code cleanup and commenting * more cleanup, renames and comments * fix rename * more cleanups * use Judy.h from system paths * added foreach traversal; added flag to add item in front; isolated locks to their own functions; destruction returns the number of bytes freed * more comments; flags are now 16-bit * completed unittesting * addressed comments and added reference counters maintainance * added unittest in main; tested removal of items in front, back and middle * added read/write walkthrough and foreach; allowed walkthrough and foreach in write mode to delete the current element (used by cups.plugin); referenced counters removed from the API * DICTFE.name should be const too * added API calls for exposing all statistics * dictionary flags as enum and reference counters as atomic operations * more comments; improved error handling at unit tests * added functions to allow unsafe access while traversing the dictionary with locks in place * check for libcups in cmake * added delete callback; implemented statsd with this dictionary * added missing dfe_done() * added alternative implementation with AVL * added documentation * added comments and warning about AVL * dictionary walktrhough on new code * simplified foreach; updated docs * updated docs * AVL is much faster without hashes * AVL should follow DBENGINE * coverity fixes about statsd; removal of strsame (#13049) * Fix dictionary crash walkthrough empty (#13051) fix dictionary walkthrouhg crash when the dictionary is empty * [ci skip] Update changelog and version for nightly build: v1.34.0-268-nightly. * dictionary improvements (#13052) * fix typo in foreach write; added unit tests to traverse empty dictionaries * rename variable dfe in macro to be uniform with name variable * Check for host labels when linking alerts for children (#13053) check for host labels when linking alerts * Fix coverity 378625 (#13055) free jsonb buffer * add the ability to merge dictionary items (#13054) * add the ability to merge old value and new value * docs * merge to conflict * fix(cgroup.plugin): read k8s_cluster_name label from the correct file (#13062) * Initialize chart label key parameter correctly (#13061) Pass chart label key * [ci skip] Update changelog and version for nightly build: v1.34.0-275-nightly. * update codeowners (#13060) * Fix handling of temp directory in kickstart when uninstalling. (#13056) This makes temporary directory creation idempotent, and ensures it’s invoked in each place that needs it. It also tidies up the cleanup code so that it only tries to remove the temporary directory if one was created. * Fix locking access to chart labels (#13064) No write lock required * [ci skip] Update changelog and version for nightly build: v1.34.0-279-nightly. * Allow usage of the new MQTT 5 implementation (#12838) * adds support for new MQTT5 implementation in agent, currently by default disabled as Tech Preview * Added Alma Linux 9 and RHEL 9 support to CI and packaging. (#13058) * Added Alma Linux 9 and RHEL 9 support to CI and packaging. * Fix RPM deps for Alma 9. * Fix package testing code for Alma 9. * Update default value for "host anomaly rate threshold" (#13075) * Add improved reinstall documentation. (#13047) Including explaining how to do a clean reinstall, and how to switch install types. * Fix handling of DEB package naming in CI. (#13076) * Fix handling of DEB package naming in CI. * Bump repoconfig package versions. To ensure naming consistency. * Fix repoconfig package renaming. * [ci skip] Update changelog and version for nightly build: v1.34.0-285-nightly. * fix(updater): don't produce any output if binpkg update completed successfully (#13081) * don't produce any output when binpkg update succeeded * redirect update repo, update netdata output to 3 * Remove Fedora 34 from CI and package builds. (#12875) * Remove Fedora 34 from CI and package builds. EOL date is 2022-06-07. * Update platform support document. * Fix Coverity errors in mqtt_websockets submodule (#13082) - update mqtt_websockets submodule to bring in the Coverity scan fixes * [ci skip] Update changelog and version for nightly build: v1.34.0-289-nightly. * Update README.md (#13088) * Update README.md (#13089) * fix(updater): return 0 on successful update for native packages when running interactively (#13083) return 0 on success in udpate_binpkg * [ci skip] Minor release v1.35.0. * [ci skip] Update changelog and version for nightly build: v1.35.0-1-nightly. * Temporarily disable updates for static builds. (#13100) * Temporarily disable updates for static builds. * Fix typo in warning message. * Remove unnescesary ‘cleanup’ code. (#13103) This was causing the updter to be unconditionally disabled on static builds, and we don’t actually _need_ any of it anymore since we’ve already decoupled the auto-update handling state from the installation process (in fact, this should have been removed then, but it got missed somehow). * [ci skip] Update changelog and version for nightly build: v1.35.0-4-nightly. * Re-enable updates for systems using static builds. (#13110) Revert "Temporarily disable updates for static builds. (#13100)" This reverts commit b67c1026bae0411e863a227214fd4175ff109d0f. * docs(statsd.plugin): fix indentation (#13096) fix indentation * Add openSUSE Leap 15.4 to CI and package builds. (#12270) Add openSUSE Leap 15.4 t CI and package builds. Expected release date 2022-06-08. * [ci skip] Update changelog and version for nightly build: v1.35.0-8-nightly. * fix virtualization detection on FreeBSD (#13087) * Add user netdata to secondary group in DEB package (#13109) * Add user netdata to secondary group * Update netdata.postinst * Labels with dictionary (#13070) * squashed and rebased to master * fix overflow and single character bug in sanitize; include rrd.h instead of node_info.h * added unittest for UTF-8 multibyte sanitization * Fix unit test compilation * Fix CMake build * remove double sanitizer for opentsdb; cleanup sanitize_json_string() * rename error_description to error_message to avoid conflict with json-c * revert last and undef error_description from json-c * more unittests; attempt to fix protobuf map issue * get rid of rrdlabels_get() and replace it with a safe version that writes the value to a buffer * added dictionary sorting unittest; rrdlabels_to_buffer() now is sorted * better sorted dictionary checking * proper unittesting for sorted dictionaries * call dictionary deletion callback when destroying the dictionary * remove obsolete variable * Fix exporting unit tests * Fix k8s label parsing test * workaround for cmocka and strdupz() * Bypass cmocka memory allocation check * Revert "Bypass cmocka memory allocation check" This reverts commit 4c49923839d9229bea23ca914dd8a0be1ebe2bf4. * Revert "workaround for cmocka and strdupz()" This reverts commit 7bebee04801db1865c748a7896d5fa54bb7104a5. * Bypass cmocka memory allocation checks * respect json formatting for chart labels * cloud sends colons * print the value only once * allow parenthesis in values and spaces; make stream sender send quotes for values Co-authored-by: Vladimir Kobal <vlad@prokk.net> * 73x times faster metrics correlations at the agent (#13107) * faster correlations * 4x times faster correlations * a little bit more help * 10x times faster metrics correlations * 6 digits precision; better comments * enabled metrics correlations by default * abstracted DIFFS_NUMBER to allow easily changing it * reworked the entire logic to have more accuracy and support a baseline that is power of two multiple of highlight * properly calculate shifts * even more improved version * added support for timeout; fixed another memory leak; skipped hidden dimensions * default timeout 1min * reduce memory even further * use dictionary for the list of charts and optimize locks * return 403 forbidden, when mc is not enabled * added query options * dont process zero dimensions * added volume method as an option to metric correlations ; now metric correlations can support multiple implementations * make sure we will never crash * spread results evenly for both kstwo and volume * fixed bug in query engine that was missing misaligned queries when a single point was requested from the db; improved comments; improved query flags * updated swagger and added sane defaults; query options are now supported, including anomaly-bit * added "raw" option to allow cross node correlations; added "group" option to allow different time aggregations; allowed calling metric correlations without any parameters; allowed calling metric correlations with relative timestamps; added timeout to volume method; properly handled timeout on ks2 method; json output now sends all parameters back - same for json_wrap; modified query engine to use present time for relative timestamps; modified "allow_past" to mean both past backwards and forwards * emulate the old behaviour about zero points * 100% accuracy against python ks_2samp(); now the default is volume and the default points are 500 * added config option to change default metric correlations method * removed work-arounds now that rrdlabels are merged * [ci skip] Update changelog and version for nightly build: v1.35.0-13-nightly. * Ci coverage (#13118) * add check to verify autoupdate is still enabled after netdata update * add script to install CI support packages before installing netdata * Fix typo in release base prep script. (#13098) * fixed coveriry 379136 379135 379134 379133 (#13123) * allow label names to have slashes (#13125) * allow label names to have slashes * fixed flipped comment * buffer overflow detected by the compiler (#13120) * buffer overflow detected by the compiler * also increase the buffer +1 * ACLK statistics on bytes recvd and sent (#13091) * fix(python.d/mongodb): set `serverSelectionTimeoutMS` for pymongo4+ (#13135) fix server_selection_timeout for pymongo4+ * fix(python.d): urllib3 import collection for py3.10+ (#13136) * [ci skip] Update changelog and version for nightly build: v1.35.0-22-nightly. * Use a separate thread for slow mountpoints in the diskspace plugin (#13067) * use ks2 as MC default (#13131) * fix crashes due to misaligned allocations (#13137) * Add missing control to streaming (#13112) fix_tls_stream: Add call to SSL_get_error to avoid an infinite loop * Add an option to use malloc for page cache instead of mmap (#13142) Add an option to switch to using malloc for page cache instead of mmap * Update docs on what to do if collector not there (#13152) * [ci skip] Update changelog and version for nightly build: v1.35.0-29-nightly. * Remove pinned page reference (#13108) * Disable reference to prev_descr as we do not keep two pages pinned * Remove extra pinned page from page cache calculations * Removed invalid comment * Remove unused variable * Add cargo/rustc/bazel/buck to apps_groups.conf (#13143) * revert to default of `host anomaly rate threshold=0.01` (#13150) * fix conditions for nightly build triggers (#13145) * Fix labels unit test (#13156) * add CAP_SYS_RAWIO to Netdata's systemd unit CapabilityBoundingSet (#13154) * Add mem.available chart to FreeBSD (#13140) * Configurable storage engine for Netdata agents: step 3 (#12892) * storage engine: add host context API Add a new API to allow storage engines to manage host contexts. * Replace single global context with per-engine global context * Context is full managed by storage engines: a storage engine can use no context, a global engine context, per host contexts, or a mix of these. * Currently, only dbengine uses contexts. Following the current logic, legacy hosts use their own context, while non-legacy hosts share the global context. * storage engine: use empty function instead of null for context ops * rrdhost: don't check return value for void call * rrdhost: create context with host * storage engine: move rrddim ops to rrddim_mem.{c,h} * storage engine: don't use NULL for end-of-list marker * storage engine: fallback to default engine * Use memset to mark the empty words in the quoted_strings_splitter function (#13161) Use memset to terminate the words * Don’t manipulate positional parameters in DEB postinst script. (#13169) * Fix coverity issues (#13168) * [ci skip] Update changelog and version for nightly build: v1.35.0-41-nightly. * enable ml by default (#13158) * enabled ml by default * Fix data query on stale chart (#13159) * Fix data query on stale chart * Remove more checks vs the last timestamp of a point collected * feat(proc/proc_net_dev): add dim per operstate to the "Interface Operational State" chart (#13167) * Revert "Configurable storage engine for Netdata agents: step 3 (#12892)" (#13171) This reverts commit 100a12c6cc01222b1518e5e50d2147f592d8a111. A couple parent/child startup/shutdown scenarios can lead to crashes. * feat(proc/proc_net_dev): add dim per duplex state to the "Interface Duplex State" chart (#13165) * allow traversing null-value dictionaries (#13162) * allow traversing null-value dictionaries * fix lgtm report * void the value too * removed NEVERNULL directive * Fix coverity 378587 (#13024) * check for return value of sysconf * if sysconf fails set OWA_NATURAL_PAGE_SIZE to 4096 * [ci skip] Update changelog and version for nightly build: v1.35.0-49-nightly. * feat(proc/proc_net_dev): add dim per phys link state to the "Interface Physical Link State" chart (#13176) * add dim per carrier state * fix down state * Update dashboard to version v2.26.2. (#13177) Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> * set default for `minimum num samples to train` to `900` (#13174) This will enable first set of initial models to be trained quicker and makes sense now that ml is enabled by default. * add k8s_state dashboard_info (#13181) * [ci skip] Update changelog and version for nightly build: v1.35.0-54-nightly. * Add ml alerts examples (#13173) * add ml alarm examples * Update Makefile.am * add hyperlinks and node level AR example * fix(freebsd.plugin): fix wired/cached/avail memory calculation on FreeBSD with ZFS (#13183) * [ci skip] Update changelog and version for nightly build: v1.35.0-57-nightly. * Update dashboard to version v2.26.5. (#13192) Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> * Query Engine multi-granularity support (and MC improvements) (#13155) * set grouping functions * storage engine should check the validity of timestamps, not the query engine * calculate and store in RRDR anomaly rates for every query * anomaly rate used by volume metric correlations * mc volume should use absolute data, to avoid cancelling effect * return anomaly-rates in jasonwrap with jw-anomaly-rates option to data queries * dont return null on anomaly rates * allow passing group query options from the URL * added countif to the query engine and used it in metric correlations * fix configure * fix countif and anomaly rate percentages * added group_options to metric correlations; updated swagger * added newline at the end of yaml file * always check the time the highlighted window was above/below the highlighted window * properly track time in memory queries * error for internal checks only * moved pack_storage_number() into the storage engines * moved unpack_storage_number() inside the storage engines * remove old comment * pass unit tests * properly detect zero or subnormal values in pack_storage_number() * fill nulls before the value, not after * make sure math.h is included * workaround for isfinite() * fix for isfinite() * faster isfinite() alternative * fix for faster isfinite() alternative * next_metric() now returns end_time too * variable step implemented in a generic way * remove left-over variables * ensure we always complete the wanted number of points * fixes * ensure no infinite loop * mc-volume-improvements: Add information about invalid condition * points should have a duration in the past * removed unneeded info() line * Fix unit tests for exporting engine * new_point should only be checked when it is fetched from the db; better comment about the premature breaking of the main query loop Co-authored-by: Thiago Marques <thiagoftsm@gmail.com> Co-authored-by: Vladimir Kobal <vlad@prokk.net> * Add type label for network interfaces (#13187) * feat(python.d/smartd_log): add 2nd job that tries to read from '/var/lib/smartmontools/' (#13188) * Fix cgroups netdev chart labels (#13200) * Allow for an easy way to do metadata migrations (#13196) Allow for an easy way to migrate metadata to a new database schema (versioning) * Clean sqlite prepared statements on thread shutdown (#13193) * Clean sqlite prepared statements on thread shutdown * Add note for pending statements to finalize * [ci skip] Update changelog and version for nightly build: v1.35.0-65-nightly. * Add configuration for dbengine page fetch timeout and retry count (#13194) * Add configuration for page cache fetch timeout and retry count Change page cache wait default timeout to 3 seconds * Issue info message in the error.log if values not within expected lower range * Fix compilation errors with --disable-dbengine * Print INTERNAL BUG messages only when NETDATA_INTERNAL_CHECKS is enabled (#13207) put INTERNAL BUG messages inside NETDATA_INTERNAL_CHECKS * chore(netdata-installer): remove a call to 'cleanup_old_netdata_updater()' because it is no longer exists (#13189) * [ci skip] Update changelog and version for nightly build: v1.35.0-69-nightly. * Add user plugin dirs to environment (#13203) Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * [ci skip] Update changelog and version for nightly build: v1.35.0-71-nightly. * make configuration example clearer (#13182) make ml configuration example cleare * [ci skip] Update changelog and version for nightly build: v1.35.0-73-nightly. * deduplicate mountinfo based on mount point (#13215) * [ci skip] Update changelog and version for nightly build: v1.35.0-75-nightly. * Use new mqtt implementation as default (#13132) flips mqtt5 to be default instead of mqtt3 * fix(updater): fix updating when using `--force-update` and new version of the updater script is available (#13104) * Ensure tmpdir is set for every function that uses it. (#13206) The set_tmpdir function is idempotent specifically to allow for this type of usage, and this should help avoid bugs involving temporary directory handling going forwards. * Don’t use realpath to find kickstart source path. (#13208) Don’t use realpath to find kcisktart source path. It doesn’t work on macOS and leads to a bogus error message being printed. * Properly handle interactivity in the updater code. (#13209) * Clean up updater option parsing. This will make the remaining changes simpler, and also ease maintenance long-term. * Add updater options to override interactivity detection. * Pass interactivity from kickstart to updater when invoking updater. * Removes Legacy JSON Cloud Protocol Support In Agent (#13111) * removes old protocol support (cloud removed support already) * ci: add issues to the Agent Board project workflow (#13225) * add add issues to Agent Board project workflow * add 'transferred' event * fix(cgroups.plugin): fix qemu VMs and LXC containers name resolution (#13220) * [ci skip] Update changelog and version for nightly build: v1.35.0-84-nightly. * feat(python.d): load modules from user plugin directories (NETDATA_USER_PLUGINS_DIRS) (#13214) * Add more sqlite unittests (#13227) * Bump repoconfig package version used in kickstart.sh (#13235) * Update netdata commands (#13080) * Update netdata commands Adding the `-W buildinfo` options. * Update README.md * Update README.md * Update daemon/README.md Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * Update daemon/README.md Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * also add the change in the daemon command line help message Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud> * remove whitespace Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud> Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * Use correct PAT for add-to-project workflow. (#13234) * Implement PackageCloud cleanup (#13236) * add script for netdata packagecloud cleanup * add script for netdata packagecloud cleanup * remove old package-cloud cleanup code * add workflow for packagecloud cleanup * Debug msg * allow individual matrix jobs to complete independently to each other * perform cleanup without dry-run * remove debug msg * remove cleanup line from the slack notification * Upload packages to new self-hosted repository infrastructure. (#13240) * Match packaging arches to distro architecture names. This will simplify handling of uploads to our new self-hosted packaging infrastructure. * Add package uploads for RPM packages on new infrastructure. * Add package upload support for DEB packages. * Saner tmpdir handling. * Upload fixes. * Differentiate different DEB distros in package names. * Workflow fixes. * netdata doubles (#13217) * netdata doubles * fix cmocka test * fix cmocka test again * fix left-overs of long double to NETDATA_DOUBLE * RRDDIM detached from disk representation; db settings in [db] section of netdata.conf * update the memory before saving * rrdset is now detached from file structures too * on memory mode map, update the memory mapped structures on every iteration * allow RRD_ID_LENGTH_MAX to be changed * granularity secs, back to update every * fix formatting * more formatting * Dictionaries with reference counters and full deletion support during traversal (#13195) * dont use atomic operations when not needed; detect misuse of the the unsafe functions * use relaxed atomic operations for statistics * use relaxed atomic operations for statistics * dictionaries now use reference counters, allowing deletetions of any item while traversing it * added acquire/release interface to dictionaries * added unittest for reference counters * added NETDATA_INTERNAL_CHECKS logs to detect non-exclusive access to crusial parts of the dictionaries * dictionaries cannot be deleted while there are referenced items in them - they will be deleted once the last item gets unreferenced * cleanup * properly cleanup released items * maintain counters for readers and writers; defer all deletes on sorted walkthrough; cleaner internal_error(); * somewhat faster reference counters on single threaded dictionaries * minor optimizations; allow compiling without internal checks * Delay health until obsoletions check is complete (#13239) * wait until obsoletions check is complete to run health * run the checks 5 minutes after agent connect time * Updates the sqlite version in the agent (#13233) New version sqlite Enabling also #define SQLITE_UDL_CAPABLE_PARSER 1 #define SQLITE_ENABLE_UPDATE_DELETE_LIMIT 1 #define SQLITE_OMIT_LOAD_EXTENSION 1 #define SQLITE_ENABLE_DBSTAT_VTAB 1 * [ci skip] Update changelog and version for nightly build: v1.35.0-96-nightly. * Fix build matrix generation for build workflow. (#13254) * Fix build matrix generation for build workflow. * Fix build matrix sorting. * Fix further typos. * fix: fix a base64_encode bug (#13074) * Use old mqtt implementation as default (#13258) mqtt5 as default off * Add fstype labels to disk charts (#13245) Co-authored-by: vkalintiris <vasilis@netdata.cloud> Co-authored-by: Vladimir Kobal <vlad@prokk.net> Co-authored-by: Netdata bot <43409846+netdatabot@users.noreply.github.com> Co-authored-by: netdatabot <netdatabot@users.noreply.github.com> Co-authored-by: netdatabot <bot@netdata.cloud> Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Emmanue…
Summary
The aim of this PR is to use
ks2
as the default for metric correlations instead ofvolume
.Here is a video of some examples why i think ks2 should be the default over volume.
Basically, ks2 will capture a wider range of changes than volume will be able to. As such just using volume will bias towards a subset of changes that the ks2 algo will capture anyway.
So while i like that we should have volume as a dropdown option in the FE and it will be the default at first for cross node MC where we have a larger amount of selected nodes. I don't think it should be the default and replace ks2.
Test Plan
Additional Information
For users: How does this change affect me?