Release Notes

Oliver Gorwits edited this page Jun 17, 2018 · 28 revisions

Introduction

This document will list only the most significant changes with each release of Netdisco. You are STRONGLY recommended to read this document each time you install and upgrade. Also see the Changes file, for more information.

Migrating from Netdisco 1.x

This distribution (App::Netdisco) is a complete rewrite of the Netdisco application. Users often ask whether they can run both versions at the same time, and whether the database must be copied. Here are the guidelines for migrating from Netdisco 1.x:

  • You can run both Netdisco 1.x and App::Netdisco web frontends at the same time, using the same database (if “safe_password_store” is set to “false” in the config file).

  • You can share a single database between Netdisco 1.x and App::Netdisco. The deploy script for App::Netdisco will make some schema changes to the database, but they are backwards compatible.

  • Only enable the backend daemon and discovery jobs from either Netdisco 1.x or App::Netdisco.

2.039031

This release fixes a bug in the job queue picker which can result in backend workers becoming tied up with jobs they cannot complete. Upgrade is strongly recommended for all installations.

2.039015

A bug fix in the network diagram component in this release will result in the loss of link speeds in the display until a discover job is run for all devices. You can either wait a day for this to happen, or trigger discoverall from the CLI or Admin menu.

2.039012

In this release we gather and display additional inventory information about connected nodes and devices. This allows smarter signing of Phone and Wireless AP existence on device ports. You may need to force-reload the Device Ports page to load the new sidebar, and/or click the "reset to defaults" icon to enable display of inventory data.

2.039011

This release harmonises the behaviour of netdisco-do and the scheduler so they can accept the same requests. netdisco-do is also now able to send jobs to the queue rather than execute them immediately, and also to suppress logs, for better compatibility with crontab users. For example:

schedule:
  # standard schedule
  macsuck:
    when: 20 * * * *
  # duplicate that action for increased frequency for a specific device
  macsuck_specific:
    action: macsuck
    when: 15,45 * * * *
    device: 192.0.2.1
  # run a specific stage of an action
  refresh_neighbors:
    action: 'discover::neighbors'
    when: 20 8,11,14,17 * * *
  # limit action to an IP prefix
  arpnip_specific:
    action: arpnip
    when: 25 * * * *
    device: 192.0.2.0/29
netdisco-do discover -d 192.0.2.0/24 --enqueue --quiet

2.039003

The timeout setting which appeared in version 2.039000 has moved to within the workers configuration setting (rather than being global). For example:

workers:
  timeout: 1200 # 20 minutes

Futhermore, the nbtstat_timeout setting is renamed to nbtstat_response_timeout.

Previously when the backend daemon was restarted, Netdisco reset its count of SNMP connect failures to each device. You will recall this is used to stop the backend trying to connect after many failed attempts (default 10). With this release, the count is not reset, but decremented by one, saving a lot of time and effort and allowing you to edit deployment.yml and restore connectivity to a device if needed.

2.039000

All actions (jobs) now have a default timeout of 10 minutes. After this time the job will abort with an error status. You can change this with the new timeout setting (default 600 seconds), which can also be set to zero to disable the feature. For any specific action you can also set “<actionname>_timeout” (e.g. macsuck_timeout).

As well as autodiscovering devices on your network using layer two discovery protocols, Netdisco will now optimistically try to discover routing peers for the protocols BGP and OSPF. This should help discover WAN topologies more easily.

There is a new replacement for the netdisco-rancid-export script. Instead, you can run the makerancidconf worker within your backend schedule. See the documentation for further details.

The IP archiving behaviour of Netdisco 2 has until now been accidentally different to that in Netdisco 1. This has now been fixed. See the new “expire_nodeip_freshness” configuration setting if you wish to retain the behaviour of Netdisco 2. As a consequence of this fix you may notice some values in your statistics table reducing.

As always, you are advised to take a backup of your database regularly and especially before upgrade, as we cannot unfortunately anticipate all bugs.

2.038006

This release introduces the jobs_stale_after setting which allows the backend manager to assume jobs are stale and duplicates can be started. The default is 50 minutes (configured as a number of seconds: 3000).

2.038003

This is a bug fix release that addresses a longstanding issue with the expiry job - it was not removing all records that it should have from the database. As a consequence of this fix you may notice some values in your statistics table reducing.

As always, you are advised to take a backup of your database regularly and especially before upgrade, as we cannot unfortunately anticipate all bugs.

2.038000

This release introduces a new device neighbors map visualisation. You are advised to read the User Guide and to configure the host_group and host_group_displaynames settings to get maximum benefit.

Feedback and suggestions on the new neighbors map are very welcome 😀.

2.037004

This is a bug fix release, and includes better support for customising the job schedule configuration. As mentioned below, you can disable the scheduler by setting schedule: null in your configuration. Your own configuration is merged with Netdisco’s default, so you always get discoverall, macwalk, arpwalk, and nbtwalk jobs. If you wish to disable any of these, assign them the null value, for example:

schedule:
  nbtwalk: null

2.037000

This is a major point release bringing a rewrite of the core of the backend poller daemon. The only action you may wish to take as a result, is to comment out the schedule configuration in your deployment.yml configuration file, if you are using the default settings. This will ensure you always get the "best" schedule configuration including new features/actions.

To disable job scheduling but still run a backend poller (to handle jobs queued by another daemon, or from the web user interface), then use this configuration:

schedule: null

Other improvements in this release include rewriting the sidebar settings handling in the web front end, so they are remembered properly (in the Device Ports page). Also we have rewritten the job queue to further eliminate race conditions and duplicate devices leaking into your setup.

2.036002

This is a bug fix release since 2.036000. Please also read the 2.036000 release notes below, in full. Notable changes:

  • Device Port report is much faster when displaying nodes

  • netdisco-do psql now supports NETDISCO_DBNAME environment variable

  • snmp_auth configuration now supports only and no ACLs per stanza

  • New Duplicate Devices Report in case you get these appearing

  • Neighbor L2 topology map will show sysName if DNS is not available

  • Speed up ACLs featuring regualr expressions

Plus lots more mentioned in the Changes file.

2.036000

This release has many significant new features and changes. Please read all the release notes before upgrading.

  • A new setting host_groups allows for creating named Access Control Lists which can be referred to in other host groups or in any of the settings taking an ACL.

  • The new setting device_identity allows configuring rules to select the interface to use as a canonical (friendly) identity of a device in Netdisco.

  • The new settings devices_no and devices_only are shorthand for setting discover_*, macsuck_*, arpnip_*, and nbtstat_* at once.

  • Netdisco now tracks SNMP connect failures and after 10 failed attempts will pause trying to connect, for one week (see the max_deferrals and retry_after settings). See also the "SNMP Connect Failures" admin report.

  • Documentation and support for access control lists has been overhauled. Most “*_no”, “*_only”, and “only” settings will accept ACLs as single items or lists. ACLs now support negation and OR/AND modifier options.

  • A new setting site_local_files is a shorthand for confguring paths in which to install local Perl, template, javascript, and images files for overriding or enhancing Netdisco.

  • The topology import script (nd-import-topology) will now queue a "discover" job for each new device it imports.

  • The netdisco-daemon and netdisco-daemon-fg scripts have been renamed to netdisco-backend and netdisco-backend-fg respectively.

The old commands will still work but we recommend packagers to use the new names to remain consistent with documentation. Run the following on upgrade:

ln -s ~/perl5/bin/{localenv,netdisco-*} ~/bin/
~/bin/netdisco-daemon stop
~/bin/netdisco-backend restart
  • SSL library headers are required to build Netdisco now that we retrieve support files via HTTPS.

On Ubuntu/Debian:

root:~# apt-get install libssl-dev

On Fedora/Red-Hat:

root:~# yum install openssl-devel

On BSD these headers are usually installed with the openssl port itself.

Netdisco will otherwise fail to upgrade/install (it will fail building IO::Socket::SSL or Net::SSLeay). If you get stuck or confused, you are looking for the package including the file openssl/err.h.

2.034000

This release changes the way the application tracks web sessions for logged-in users, from on-disk files, to encrypted browser cookies. As a result, on upgrade (after running netdisco-deploy and restarting netdisco-web), all users will need to log in again.

There may be a pause after restarting netdisco-web as old web session files on disk are purged.

2.032003

The algorithm for selecting the canonical IP/name of a device has changed in this release. No longer is the OSPF Router ID taken into account. The default IP/name of a device will be either the IP specified for manual discovery, or the IP reported to a neighbor port during automatic discovery. For the latter you can often influence this through device configuration (LLDP advertise…​).

2.032000

The identification of IP Phone hansets and Wireless APs is now configurable, using the CDP/LLDP information from the device. See documentation for:

phone_capabilities
phone_platforms
wap_capabilities
wap_platforms

2.031006

When displaying device ports, Netdisco will now avoid showing VLAN Membership if it looks like there are a large number of VLANs on many ports. This is an average of the VLANs per port, configurable in devport_vlan_limit. The default is 150.

2.031005

The netdisco-do command’s delete option now uses the -p parameter to set node archive mode (previously it was a hack on -e). For example:

~netdisco/bin/netdisco-do delete -d 192.0.2.1 -e 'older than the sun' -p yes

2.031003

Health Advice

This release will once again remove from the database spurious Node (workstation, printer, etc) entries on vlan 0, which were causing dupliate entries in the web interface. We advise that you back up the database prior to upgrade:

/usr/bin/pg_dump -F c --create -f netdisco-pgsql.dump netdisco

General Notices

The database schema can be fully redeployed (even over an existing installation, in a safe way) using the following command:

~netdisco/bin/netdisco-db-deploy --redeploy-all

2.031002

Netdisco web and backend daemons will now rotate their log files (“\~netdisco/logs/netdisco-{web,daemon}.log”). This happens when they reach about 10MB in size and seven historical log files will be maintained in the same directory. The first time this happens you may notice the daemons restarting due to having to deal with the large initial logfile.

Two missing features from Netdisco 1 have been implemented: CLI device delete and renumber (canonical IP change). They are available using the netdisco-do utility.

The Device Port Log comment feature from 2.030000 has been disabled as it is incomplete, pending a review of how to handle authorization to the feature.

2.029014

The node archiving behaviour of Netdisco 2 has until now been accidentally different to that in Netdisco 1. This has now been fixed. See the new “node_freshness” configuration setting if you wish to revert or tune this behaviour.

2.029010

When upgrading you will encounter a current incompatibility between Netdisco and one of its components. To work around this, issue the following command:

~/bin/localenv cpanm --notest --force Dancer@1.3126 DBIx::Class@0.08270

2.029008

When upgrading you will encounter a current incompatibility between Netdisco and one of its components. To work around this, issue the following command:

~/bin/localenv cpanm --notest --force Dancer@1.3126

2.029002

The backend polling daemon has been rewritten and as a result your configuration can be simplified. Some keys have also been renamed. Our advice is to remove (or comment out) the complete workers configuration which enables auto-tuning. If you do wish to control the number of worker processes, follow this pattern:

workers:
  tasks: 'AUTO * 2'  # this is the default, twice the number of CPUs

2.029001

Health Advice

This release will remove from the database spurious Node (workstation, printer, etc) entries on vlan 0, which were causing dupliate entries in the web interface. We advise that you back up the database prior to upgrade:

/usr/bin/pg_dump -F c --create -f netdisco-pgsql.dump netdisco

General Notices

The configuration item reports is now a list (used to be a dictionary). Each item in the list must have a tag entry which was previously the dictionary key. For example, now use:

reports:
  - tag: power_inventory
    category: Device
    label: 'Power Supply Inventory'
    columns:
      - {name: 'Name'}
      - {ps1_type: 'PS1 Type'}
      - {ps1_status: 'PS1 Status'}
    query: |
      SELECT d.name, d.ps1_type, d.ps1_status
        FROM device d
        WHERE d.ps1_type IS NOT NULL
      ORDER BY name

Old configuration will be continue to work, but we recommend you reconfigure anyway.

2.028000

Incompatible Changes

The daemons can be started from init scripts, as root. They will drop back from the root user to netdisco before opening logs. However a limitation is that the web frontend might temporarily keep root status to bind to a specific port (e.g. 80) - the logs will then be created as root user. Sorry about that.

You might also find when upgrading that previous logs were owned by root and Netdisco now wants to write to them as non-root (netdisco) user. Please either remove the logs before restarting, or alter their ownership.

Logs can be found in the logs subdirectory of Netdisco’s home area.

General Notices

The configuration item housekeeping has been renamed to schedule. Old configuration will continue to work, but we recommend you now rename this key in your configuration anyway.

2.025001

The Web and Backend daemons (netdisco-web and netdisco-daemon respectively) will now watch your deployment.yml configuration file, and restart themselves whenever it is changed.

The Web and Backend daemons will also now drop privilege to the same user and group as their files on disk. This allows use of run control (init) scripts whilst maintaining non-root privilege status (see install tips for details).

The housekeeping task expiry has been renamed to expire. Old configuration will continue to work, but we recommend you rename this part of your housekeeping configuration anyway.

2.023000

Incompatible Changes

This release will automatically migrate user passwords to have stronger hashing in the database (a good thing!). This is incompatible with Netdisco 1.x web frontend, so if you must maintain backward-compatibility, set the following in your deployment.yml file:

safe_password_store: false

General Notices

The number of parallel DNS queries running during node discovery has been reduced to 10 for maximum safety, but resulting in lower macsuck performance. If you have a robust DNS infrastructure, you can probably put it back up to something like 50 or 100:

dns:
 max_outstanding: 100

2.021000

Incompatible Changes

SNMP community strings provided in the community_rw configuration setting will no longer be used for read actions on a device (despite having “rw” in the setting name).

If you have the same community string for read and write access, then you must set both community and community_rw in your deployment.yml file. In any case, we recommend using the new snmp_auth configuration format which supercedes both these settings.

Health Advice

This release includes support for Device and Node expiry from your database. This is an important part of housekeeping for your installation, and our recommendation is to enable this feature such that suitably old Devices and Nodes are expired nightly.

Add the following to your “housekeeping” configuration in deployment.yml, to have a nightly check at 11:20pm:

housekeeping:
  expire:
    when: '20 23 * * *'

You should also configure one or more of expire_devices, expire_nodes, and expire_nodes_archive to a number of days. See the documentation for further details.

General Notices

If you use an Apache reverse proxy, we recomment increasing the timeout from our previous example of 5 seconds to, perhaps 60. This is because some reports do take more time to run their queries on the database. See documentation for details.

2.020000

If you were using the X::Observium plugin, you’ll now need to install the separate distribution https://metacpan.org/pod/App::NetdiscoX::Web::Plugin::Observium.

2.019000

This release fixes a number of issues with the poller, and is a recommended upgrade.

During Arpnip, Node IPs are resolved to DNS names in parallel. See the dns configuration option for details. Note that the nodenames configuration items from release 2.018000 are no longer available.

This release includes new support for SNMPv3 via the snmp_auth configuration option. Please provide feedback to the developers on your experience.

2.018000

The previous mentioned bug in Macsuck is now fixed.

2.017000

There is a bug in Macsuck whereby in rare circumstances some invalid SQL is generated. The root cause is known but we want to take more time to get the fix right. It should only be a few more days.

The no_port_control configuration setting is now called check_userlog and its logic is inverted. Don’t worry if this is not familiar to you - the option is only used by Netdisco Developers.

2.016000

The dangerous action log messages are now saved to the database. In a future version there will be a way to display them in the web interface.

2.015000

Some of the "dangerous action" confirmation dialogs offer to take a log message (e.g. Port Control, Device Delete). Currently the log messages are not saved. This feature will be added in the next release.

2.014000

The backend poller daemon is now considered stable. You can uncomment the housekeeping section of the example configuration and thereby enable regular device (re-)discovery, arpnip and macsuck.

2.013000

You can now configure LDAP authentication for users.

2.012000

The read-write SNMP community is now stored in the database, when used for the first time on a device. If you don’t want the web frontend to be able to access this, you need to:

  • Have separate deployment.yml files for web frontend and daemon, such that only the daemon config contains any community strings.

  • Use separate PostgreSQL users for web frontend and daemon, such that the web frontend user cannot SELECT from the community DB table.

2.011000

Users can be managed through the web interface (by admins only).

2.010000

You can now simplify database configuration to just the following, instead of the more verbose plugins/DBIC setting which was there before:

database:
  name: 'netdisco'
  host: 'localhost'
  user: 'someuser'
  pass: 'somepass'

Also, the REMOTE_USER environment variable and X-REMOTE_USER HTTP Header are now supported for delegating authentication to another web server. See the Deployment and Configuration documentation for further details.

2.008000

Health Advice

This release contains the first version of our new poller, which handles device and node discovery. Please make sure to backup any existing Netdisco database before trying it out.

General Notices

You can remove any settings from ~/environments/deployment.yml which you didn’t edit or add to the file yourself. All defaults are now properly embedded within the application. See the new deployment.yml sample which ships with this distribution for an example.

2.006000

Incompatible Changes

The default environment configuration file develpment.yml has been renamed to deployment.yml. This better reflects that users are not developers, and also fits with the default for PSGI compatible cloud deployment services.

Please rename or copy your environment file:

mv ~/environments/development.yml ~/environments/deployment.yml

General Notices

The installation is now relocateable outside of a user’s home directory by setting the NETDISCO_HOME environment variable. This defaults to your own home directory.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.