New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DNS record management #166

Open
ryanmerolle opened this Issue Jul 1, 2016 · 48 comments

Comments

Projects
None yet
@ryanmerolle
Contributor

ryanmerolle commented Jul 1, 2016

Basic support for DNS resolution for mapping individual IPs to provide additional metadata/logic to assist with searches and investigation of individual IP usage.

@shaneshort

This comment has been minimized.

shaneshort commented Jul 6, 2016

At this point I'd be happy to just have a hostname field for the specific IP address, I think that would handle my needs.

@smcguane

This comment has been minimized.

smcguane commented Jul 6, 2016

I think that adding the host name field will be needed as part of the solution anyway. As you will need to be able to place the dns record somewhere within the platform. Probably would not hurt to add that now and then tie the hostname field into the dns integration later when this gets looked at.

@ventris

This comment has been minimized.

ventris commented Jul 11, 2016

Adding a hostname field would improve the use for the IPAM by itself and would simplify adding new records.

@jeremystretch

This comment has been minimized.

Contributor

jeremystretch commented Jul 11, 2016

I'm happy to add a hostname field to the IPAddress model if it won't conflict with any future plans for DNS support. I haven't given much thought to how that would be implemented though.

@rdujardin

This comment has been minimized.

rdujardin commented Jul 11, 2016

What we could do for instance :

  • Having a char field 'hostname' in IPAddress, for reverse DNS
  • Having an object DNSZone storing SOA data, with fields : name (charfield), ttl (integer), soaname (char), soacontact (char), serial (char), refresh (integer), retry (integer), expire (integer), minimum (integer) ; all fields required
  • Having an object DNSRecord with fields : name (charfield), type (charfield : MX, A, AAAA, CNAME, etc), priority (integer), a foreign key on DNSZone, a foreign key on IPAddress, and if IP is not needed for instance in a CNAME record, a value charfield ; only type, zone and either ip or char value are required

A question is : in DNS zone files, records can have empty name, which means the record designate the origin of the zone, how can we take this into account ? we can allow name to be empty but then how records could be selected in list ? or we can enter '@' in name to mean origin.

What do you think about it ?

@smcguane

This comment has been minimized.

smcguane commented Jul 12, 2016

The hostname field at the very least will not impact future dns implementations as long as you want to connect the DNS to the device name and not have it entirely as its own module under a new submenu, This really plays out on how you would like it to work flow visually.

I have an idea in response to @rdujardin which i think is really beneficial to us as service providers.
Bare with me as i explain as i dump my thoughts onto this post.

We should do something that no one else does well or like us have to use custom scripts to cron and check records to auto create zones for customers that are not provisioned. That is to auto generate PTR records and insert them into DNS.

How to handle default PTR record creation?
Using Subnet 10.0.0.0/24, An example of this would be as follows;

  • Add Subnet to DCIM
  • Have option to auto provision PTR records for subnet and insert them into DNS platform
  • Generation based on a set standard of subnet in reverse and a domain name that is user configured.
    An example of how this may look for the host .1 would be 1.0.0.10-in.addr.usersetdomain.com

When adding a subnet into DCIM have an option to automatically setup default PTR records
This then would automatically login to DNS, Check for existing record and if none exist, then create a default set of records for the subnet, if exist - perhaps ignore or show popup modal with some options on how to handle that.

How do we handle linking to devices and assigning / un-assigning resolution?
From there it gets a bit more involved as you will need to have certain checks and references back to existing records and default records within DNS. If we said that the best way to manage work flow is by using the hostname field on the device as being the main insertion point for PTR records, which i am not sure how this would work yet as this would need to be able to support multiple PTR records for multiple IPs assigned to the device, but say this is what was used as a reference.

Adding a PTR record
Edit the device that is located in the rack and add custom PTR record into hostname field. This maybe smtp.mailserver.com. This process would need to check if any record exists if none exist then create as per hostname field entry.

Removing a PTR record - Removing Device
If you are to remove the device from the rack this would need to remove the PTR record as part of this removal process. I would suggest handling this either way;

If subnet has the auto generate default records selected, then removing this device would remove this PTR entry and repopulate this with the default record used for that IP.

if subnet does not have the auto generate records selected, removing this device should remove the PTR record from DNS and leave this record empty in the DNS server.

Changing a PTR record
Changing a record should be as simple as value is X please change to Y. If value does not exist in DNS server then populate with Y anyway.

That is all i have thought about thus far - and i am by no means an expert on how this could look and feel and work throughout DCIM, perhaps others may want to add some value to this topic as there maybe a different way to look at this?

@rdujardin

This comment has been minimized.

rdujardin commented Jul 12, 2016

I don't know if I have well understood your message, but a DNS should be associated to an IPAddress, not to a device. The device has a primary IP and so it's already indirectly associated to a hostname. In the "reality", i.e. in bind zone files, we only associate hostnames and IPs, there are no devices.

We can make a new DNS menu, it's a menu more but DNS is essential for potentially a big part of users. IPAM menu is already quite full.

About the PTR auto populating, it's a good idea. About the automatically connecting to DNS server, it's maybe quite overkill ? Netbox provides a readonly API, which can be used in external scripts to export configs, it may be sufficient ?

What do you think ?

@smcguane

This comment has been minimized.

smcguane commented Jul 13, 2016

My comments are based on my opinion and do not necessarily represent the industry as a whole, so with that in mind - see below :)

Having Netbox connect to the PowerDNS API - is a major bonus when you are working with thousands of ip addresses. As it just removes a single step of administration which is unnecessary if you use it the way we do. My suggestion while this would benefit us greatly was thought about and taken into consideration that others would benefit from this too if they were using a similar Netbox / PowerDNS setup.

With exporting of files, you maybe also assuming that one would be using bind and/or the non database driven version that uses config files in for example /var/named. I know some providers still may use file based configuration but we personally don't.

If you look at this from a work flow chart from top to bottom, Ideally I would like to see Netbox as the top product in that chart and it can make connections to other platforms below such as DNS / ROUTERS / FIREWALLS, etc not just to have a place to store all your data/assets on your network but to really integrate and manage them. This is obviously a different conversation for a different comment but if you have Netbox making the connections and not the other way around this software becomes an integral part of day to day management and deployment in your business.

Within regards to how it flows within Netbox - I am not sure either which is why i left that open for discussion, other products are attaching ip to device or a way of linking them to the device in the rack which allows you to find most configuration in one view. Netbox may not follow this path and other people may have a better idea on how this may work?

Thoughts?

@shaneshort

This comment has been minimized.

shaneshort commented Jul 13, 2016

I think that this is probably best left to some kind of plugin based solution. I'd rather something that's a little bit more flexible than hunkering netbox down with stuff that's probably specific to people's own workflow.

I'd suggest not putting too much specific dns zone related information into netbox itself, but rather enough information that the rest of my infrastructure can nut it out based on information inside netbox.

@ryanmerolle

This comment has been minimized.

Contributor

ryanmerolle commented Jul 13, 2016

Yea I would agree with @shaneshort . @rackcentral, you feature seems like we would be getting ahead of ourselves with provisioning functionality before creating the DCIM/IPAM source of truth Stretch set forth to first create. Maybe this is a separate feature later down the road, if not a plugin as @shaneshort mentioned.

@Armadill0

This comment has been minimized.

Armadill0 commented Jul 13, 2016

An integration of the Powerdns API would be a great feature! Core or plugin doesn't matter for me, as long as it works.

And I think the better solution is to attach an IP to an interface. In this way we can have multiple interfaces with different IPs and their associated DNS records. Those are linked to the devices, which allows us to identify a single system via multiple connections.

@smcguane

This comment has been minimized.

smcguane commented Jul 13, 2016

While i respect what you both have to say, @shaneshort @ryanmerolle.

I think that you can not call Netbox a DCIM, (DataCentre Infrastructure Management) tool, if you do not consider that management is important enough to include in the design of your overall software.

DNS is essential to anyone running a network using a DCIM software solution. You have built the heart of this solution and possibly what could be the main central management tool for service providers moving forward.

This could be an amazing software if you consider the possibility of allowing it to be more involved in the management role of provider networks.

Thanks for your time.

@shaneshort

This comment has been minimized.

shaneshort commented Jul 13, 2016

@rackcentral I'm not suggesting that what you're chasing isn't useful nor wanted, I'm simply stating that building-in use-case specific things like powerdns isn't the right course of action. Making plugins for those use cases, would be

@rdujardin

This comment has been minimized.

rdujardin commented Jul 13, 2016

@rackcentral it's true that netbox should be a true management tool and not just a manual database, but for the moment the project seems not to be following this path but to focus on core features. a writable API is planned for version 2.0, so we can imagine for instance making a simple core simply handling the data model, then making more powerful/specific I/O features in plugins using the API.

@smcguane

This comment has been minimized.

smcguane commented Jul 13, 2016

@rdujardin @shaneshort - No problems,

Thanks for your time - I look forward to seeing the development path to see where it takes Netbox for future releases.

@rdujardin

This comment has been minimized.

rdujardin commented Jul 13, 2016

Ok. So what about my proposal (of having a field hostname in ip for PTR, new objects DNSZone and DNSRecord and a new DNS menu) ?

@rdujardin

This comment has been minimized.

rdujardin commented Jul 26, 2016

Hello, I'm currently working on DNS and I'm thinking to emit a pull request soon, so I would like to talk a bit about it in order to make some changes if wanted.

As I have proposed it, I created a new app for DNS, with its new menu and its two new objects : Zone and Record. I've also added a host name field to IPAddress to handle the reverse DNS. Zone and Record objects are as follows :

  • Object Zone, storing SOA data, with fields : name (charfield), ttl (integer), soa_name (char), soa_contact (char), soa_serial (char), soa_refresh (integer), soa_retry (integer), soa_expire (integer), soa_minimum (integer), description (charfield) ; all fields required except of description
  • Object Record with fields : name (charfield), type (charfield : MX, A, AAAA, CNAME, etc), priority (integer), a foreign key on a Zone, a foreign key on an IPAddress, and if IP is not needed for instance in a CNAME record, a value charfield ; and a description charfield ; only type, zone and either ip or char value are required

If you want to create a A/AAAA record with an IP you don't want in your database, if you're not in charge of its reverse or if it's an IP you don't own for instance, you can do it by setting the IP in the value charfield instead of setting an IPAddress foreign key.

So I have created these objects and all what goes with : views (list, detail, add, import), filters, API serializers and views, bulk operations, CSV import/export, new documentation page, etc.

The zone detail views lists all related records, zones and records are searchable/filterable with almost all their fields. On some char fields, filters are a multi select field on values present in database, it's for instance the case of record type : if you have A, NS, MX and CNAME records in your database, these will be the four possibilities in the filter.

I also have modified a bit some IPAM objects to handle the reverse DNS : prefixes are considered as reverse zones, so they have SOA fields too (optional), and on IP address detail view I added a list of related DNS records.

Serials are not editable because they are automatically updated as soon as something is modified in zones : the zone itself, or one of its records (edition/adding/deletion), or one of the IP addresses related to one of its records. The serial is formatted like this : YYYYMMDDN with N a positive integer.

I have also handled exports in BIND format, which is compatible with all DNS servers directly or through an import : each zone can be exported, and reverse DNS is handled too. A prefix, which is a reverse zone, can be exported. DNS reverse zones are /16 or /24 in IPv4 and /n*16 in IPv6, in IPv4 if the prefix is smaller than /24 the whole /24 corresponding zone is exported (possibly with IPs from the other prefixes in the same zone), if it's bigger the several corresponding zones are exported. BIND exports can also be accessed through the API, so it's possible to automate your DNS conf from Netbox.

I have also created a special view in DNS menu which allow to export all the reverse zones in a single shot.

I'm currently working on some additional helpful features : auto-create the corresponding A/AAAA record if applicable when you set or modify the host name field of an IP address, and auto-create the IPs and prefixes which don't exist yet when you import records.

I think I've said most of what I've done and what I'm still working on, tell me your thoughts about it, if you agree or not with a feature, if you have suggestions, ... (you can look at my fork repository if you want to see by yourself what I've already done).

@Armadill0

This comment has been minimized.

Armadill0 commented Jul 27, 2016

Nice work!

My two cents: The hostname field should be renamed to something more generic, because this could also be a Cluster- or an HA-IP. Maybe something like "fqdn" would be a better choice.

@rdujardin

This comment has been minimized.

rdujardin commented Jul 27, 2016

I agree. In fact I already thought of renaming it to something like "reverse dns" or "ptr" because it's basically what the field is. What about it ?

@Armadill0

This comment has been minimized.

Armadill0 commented Jul 27, 2016

I think PTR would be fine and precise.

Btw: auto-creation of corresponding A/AAAA record would be awesome! 😃

@LambdaDriver

This comment has been minimized.

LambdaDriver commented Jul 27, 2016

Taking that approach with the serial may cause problems as it only allows for 10 changes each day (assuming you start the count at 0). Typically I've seen YYYYMMDDNN as a serial format which allows for 100 changes in a day which is sufficient when manually editing as you can batch a bunch of changes together.

I fear incrementing the serial for every individual change automatically would rapidly exhaust the serials for a day.

You could move to a simple incrementing serial which would wrap around when it reaches the max. Another option might be to only increment the serial when that zone file is exported (through the GUI or API).

Thanks for your work on this feature!

@rdujardin

This comment has been minimized.

rdujardin commented Jul 27, 2016

Sorry, I have not correctly described it. For the moment, the serial is in fact formatted YYYYMMDD + a number of unlimited size, there's no constraint on the number of digits, except of the size of the field itself which is 100 characters. Is it okay like this ?

@jallakim

This comment has been minimized.

Contributor

jallakim commented Jul 27, 2016

Keep in mind that the NN part has to be a fixed number of digits (zero padded) from the start, since 20160727101 > 2016072801.

@LambdaDriver

This comment has been minimized.

LambdaDriver commented Jul 27, 2016

@jallakim is correct. The serial needs to be fixed and unfortunately bind9 seems to be restricted to a max length of 4294967295 which only leaves us two digits to increment with.

@jallakim

This comment has been minimized.

Contributor

jallakim commented Jul 27, 2016

Incrementing the serial whenever exporting or accessing through API would be a nice approach, yes. Only if there are changes since last export/poll, though.

@shaneshort

This comment has been minimized.

shaneshort commented Nov 8, 2016

I just wanted to see how this was tracking and to see if it's still on the roadmap for integration?

On a slightly related note, is there a bounty program or some other way for us to contribute?

@Armadill0

This comment has been minimized.

Armadill0 commented Nov 8, 2016

This would also be a very interesting feature for us.

@jsenecal

This comment has been minimized.

Contributor

jsenecal commented Dec 15, 2016

Hey @rdujardin, how's progress on this? It's something I could implement if development stopped in that direction - let us know :)

EDIT - NVM... found #398

@candlerb

This comment has been minimized.

candlerb commented Jan 9, 2017

I would very much like to see "hostname" or "FQDN" be an (optional) attribute of the IP Address object.

The presence of this attribute would imply that you want both a forward (A/AAAA) and reverse (PTR) record to be published, if you are authoritative for the respective forward and reverse zones. How you achieve this doesn't matter - e.g. you could use a PDNS backend that maps the incoming queries appropriately, or export to multiple BIND zone files.

The key thing is, both the 'A' query and the 'PTR' query would ultimately look in the same table, avoiding the data duplication and inconsistencies which can arise if you have to push two entries into two different tables.

Of course there are times where you want an individual A or PTR record, and there are other DNS records you need to provide (e.g. NS, SOA, CNAME). Those belong in separate DNS tables. They would be merged by the the PDNS query or the BIND zone export.

So for example: PDNS receives a query for 4.3.2.1.in-addr.arpa PTR.

  • Look in the IP address table for 1.2.3.4, find the fqdn and turn it into PTR record
  • Look in the DNS tables for 4.3.2.1.in-addr.arpa with PTR record (in case of manual entry)
  • Merge the results

And PDNS receives a query for foo.com A:

  • Look in the IP Address tables for fqdn "foo.com", find the address and turn it into an A record
  • Look in the DNS tables for foo.com A (in case of manual entry)
  • Merge the results

This would give a clean solution which automatically avoids the nasty problems in other systems which keep separate A and PTR tables.

A common one is: I add a host with name foo.com and address 10.0.0.1, and then find afterwards the 0.0.10.in-addr.arpa zone didn't exist. So I add that zone; but the PTR record is missing because there was nowhere to put the PTR record when the host was added.

@candlerb

This comment has been minimized.

candlerb commented Jan 16, 2017

Adding a Custom Field called "FQDN" to the ipam > IP Address object seems to do the trick nicely for now.

Minor inconveniences: I can't bulk import it (#568), and it's not shown as a column when I'm browsing IP addresses.

@VicenteGiles

This comment has been minimized.

VicenteGiles commented Jan 19, 2017

Hello,

I've been manually mergin rdujardin´s version '1.5.3-dev' DNS code to digitalocean version '1.7.4-dev' and I think I got it working (no intensive testing yet...). Hope no one feels bad about it.

rdujardin has done a great effort. From my point of view (and needs) some more work is required (DNS zone delegation, dns configuration parameters, zone trailers or headers to add extra data, etc.) but the foundation seems solid. We are considering to implement what we need from my initial merge, but would prefer to be aligned to digitalocean´s releases.

If @jeremystretch, @rdujardin or anybody is interested on this merge as a base to a future common development, I could provide it (not really experienced with github yet).

Regards,
Vicente

@adamcooley

This comment has been minimized.

adamcooley commented Feb 28, 2017

@rdujardin are you also including a swagger view for the dns module?

@mattschwen

This comment has been minimized.

mattschwen commented Apr 21, 2017

This is literally the only feature missing that I need to deploy NetBox fully in my environment. Is there any way to get rdujardin's changes in my deployment?

@rickmur

This comment has been minimized.

rickmur commented May 11, 2017

I'm also highly interested in seeing the merge being integrated in the NetBox release. DNS management is the only killer feature missing at the moment.

@jsenecal

This comment has been minimized.

Contributor

jsenecal commented Jul 24, 2018

I'm wondering if there would be some interrest in having DNS support based around lexicon and/or based around existing work done in #398 - This might be something that could be combined with the way the reports feature works for updating/fetching records periodically or manualy. (As I believe implementing a celery task runner would be somewhat too complex for the scope of this project.)

If @jeremystretch agrees on this, I could start working on it fairly soon as I need this feature implemented and would rather have it included in netbox than have an external set of scripts to implement this with custom fields.

@digitalocean digitalocean deleted a comment from wrouesnel Jul 24, 2018

@digitalocean digitalocean deleted a comment from ravenblue12 Jul 24, 2018

@digitalocean digitalocean deleted a comment from csuttles Jul 24, 2018

@learn3r

This comment has been minimized.

learn3r commented Sep 19, 2018

@jsenecal @jeremystretch It will be really great, we could have this dns features added

@dBitech

This comment has been minimized.

dBitech commented Sep 19, 2018

@candlerb

This comment has been minimized.

candlerb commented Sep 19, 2018

Not just DNS management, but a bit wider:

  • A domain is an asset, from a given registrar, with an expiry date. (Would be very useful to be warned before domains expire!)
  • Similarly would like to keep tracks of certificates, and be warned of expiry dates
  • DNS record management sits on top, but you need to be careful what you're asking for.

Is Netbox the authoritative source of all your DNS records - you dump zone files from it to your DNS servers periodically? That only works with BIND-type backends.

Or does Netbox make API calls to add / change / delete DNS records in some other system? Maybe Netbox has its own DNS tables, with database triggers to log changes. Those triggers can in turn be picked up by an external DNS update tool which talks to your DNS infrastructure.

Or should Netbox treat the DNS itself as a database, doing live DNS queries and updates??

@eimann

This comment has been minimized.

eimann commented Sep 19, 2018

I'd rather see DNS in netbox as an addition so some kind of existing DNS infrastructure with a few built-in APIs.

I don't want to domain and certificate stuff in a DCIM, that's the wrong place.

@luqasz

This comment has been minimized.

luqasz commented Sep 19, 2018

I don't want to domain and certificate stuff in a DCIM, that's the wrong place.

Neither do I.

@CodingSpiderFox

This comment has been minimized.

CodingSpiderFox commented Sep 19, 2018

I think that you can not call Netbox a DCIM, (DataCentre Infrastructure Management) [...]

thanks for writing out the long form for people like me who are not experienced in this field.. makes life so much easier

@joniknsk

This comment has been minimized.

joniknsk commented Sep 20, 2018

NetBox is not only DCIM and also IPAM tool, and whereas IPAM and DNS management are inextricably linked together, it's seem like that having of basic DNS for network equipment inside NetBox will be a good option.

If this feature is not important for someone, then just do not use it.

@luqasz

This comment has been minimized.

luqasz commented Sep 20, 2018

I just don't want any certificates, PKI etc. in netbox. DNS is fine.

@learn3r

This comment has been minimized.

learn3r commented Sep 21, 2018

NetBox is not only DCIM and also IPAM tool, and whereas IPAM and DNS management are inextricably linked together, it's seem like that having of basic DNS for network equipment inside NetBox will be a good option.

If this feature is not important for someone, then just do not use it.

I fully agree with @joniknsk as netbox is IPAM at least it's good to keep track of DNS records of organisation, but we can use other tools to manage those records with API

@DanSheps

This comment has been minimized.

Contributor

DanSheps commented Sep 21, 2018

I think some people are conflating IPAM and other infrastructure management solutions.

IPAM is IP Address Management, I would encourage everyone to familiarize themselves with the terminology. While some IPAM's do indeed provide DNS management, not all do.

I don't disagree with hostname and perhaps even record management, as that would be part of device management, but running your whole DNS infrastructure in netbox is not something I would care to see and I hope the maintainers do not look at implementing this. Same with integrating PKI infrastructure, your PKI should be managed elsewhere. I am sure when DNS is implemented, there will be a webhook system implemented as well with it that you can use to hook into your DNS infrastructure.

When all else fails, Read the Docs.

NetBox intends to represent the desired state of a network versus its operational state. As such, automated import of live network state is strongly discouraged. All data created in NetBox should first be vetted by a human to ensure its integrity. NetBox can then be used to populate monitoring and provisioning systems with a high degree of confidence.

Changing DNS, by it's vary nature, is a more operational state, versus a desired state.

@shaneshort

This comment has been minimized.

shaneshort commented Nov 8, 2018

Did this ever get anywhere? I'm worried it got bogged down in over-reaching feature requests when half of us were simply asking for a field in the table

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment