Configure automatically the DNS records using lexicon #1194

ppom0 · 2021-03-22T00:39:15Z

The problem

DNS configuration is tedious to apply by hand

Solution

Lexicon is a library that permits to programmaticaly set DNS records with an API key provided by the registrar.

PR Status

Work in Progress!
For now, we didn't begin to use lexicon:
Our PR permits to change some settings of the DNS generated records.

How to test

First install lexicon on yunohost : apt install lexicon.

Copy yunohost/data/other/providers_list.yml to /usr/share/yunohost/other/providers_list.yml

Some commands to try:

add a provider: yunohost domain registrar set <domain> <registrar-name> and answer the interactive questions
see the generated config yunohost domain dns-conf <domain>
fiddle with the settings (interface similar with yunohost app setting) yunohost domain setting ...
once you've changed some settings or added a provider, you can read /etc/yunohost/domains.yml where all the settings are stored.
see the (new) generated config yunohost domain dns-conf <domain>
push your DNS configuration to your registrar (which must be previously added) yunohost domain push_config. Firstly, check the generated config to confirm you it's what you want, and backup your DNS entries, as this function has not been deeply tested.

Look for differences in the actionsmap.yml and domains.py to see all the new available commands! :)

… (and backward-compability)

…/yunohost into enh-dns-autoconf

…tries generation

…/yunohost into enh-dns-autoconf

…ns-autoconf

…/yunohost into enh-dns-autoconf

…ns-autoconf

ppom0 · 2021-05-21T10:19:00Z

💥 Our code is ready to be tested. We'll be happy to discuss bugs, design questions and refactoring proposals.

zamentur

Thanks for your work, i made a lot of suggestion, i hope you wouldn't be afraid.

Don't forget you have 3 different situations:

Situation A : domain + sub domains

toto.com
foo.toto.com
bar.toto.com

Situation B : sub domain

ynh.toto.com

(i.e. the admin owns toto.com but uses that domain for other things than their yunohost server, and in the context of yunohost only use a sudomain ynh.toto.com - and possibly sub-sub domains like foo.ynh.toto.com, bar.ynh.toto.com)

Situation C : several sub domains

foo.toto.com
bar.toto.com

(similar to B but with several of them, but the actual root zone is toto.com)

Situation D : dyndns domain

foo.nohost.me

(This doesn't fall under the usecase of "managing the registrar / zone with lexicon" considering that this is already managed by Yunohost but it should be kept in mind as it may influence the design around the whole dns-zone generation and how-to-detect-the-parent-domain)

And of course you can have a mix with several topdomain:

toto.com
foo.toto.com
bar.toto.com

foo.titi.com
bar.titi.com

foo.tutu.com

data/actionsmap/yunohost.yml

zamentur · 2021-05-24T15:32:44Z

data/other/providers_list.yml

@@ -0,0 +1,218 @@
+aliyun:


This list should use YunoHost Args format to be able to build nice form inside webadmin. We need info like (type, help, default, example...)

Be sure to list keys according to lexicon version (we might add lexicon on our debian package chain to get an earlier version). However, we could have difficult migrations issues with this. I don't know what to do (i set this point for the next meetings https://pad.lqdn.fr/p/yunohost-01-06-2021 )

About i18n, be sure to find a way to avoid to have 100 auth_id to translate in the same way

debian/install

data/actionsmap/yunohost.yml

src/yunohost/domain.py

zamentur · 2021-05-24T19:54:14Z

src/yunohost/domain.py

+        final_lexicon = ConfigResolver().with_dict(dict_object=base_config).with_dict(dict_object=record_config)
+        # print('final_lexicon:', final_lexicon);
+        client = Client(final_lexicon)
+        distant_records[key] = client.execute()


We might want to use dig() function from yunohost.utils.network here
We should think about what we want to do with DNS cache policy.

Remember that we could have several A or several MX for the same key (qname)

zamentur · 2021-05-24T19:58:44Z

src/yunohost/domain.py

+
+        # Finally, push the new record or update the existing one
+        record_config = {
+            "action": "update" if it_exists else "create", # create, list, update, delete


In the case of several answers for a couple (qname, type):

If answers differs for a (qname, type) couple between remote and local, we probably need to delete first and create next all couple needed.

Co-authored-by: ljf (zamentur) <zamentur@users.noreply.github.com>

zamentur · 2021-08-28T16:56:41Z

Hello I’ve standard yunohost environnement I have a domain but I don’t know how to setup this branch in my yunohost and after send review to this pr

If it's not a production server, you can do this:

cd /
curl https://raw.githubusercontent.com/yunohost/ynh-dev/master/deploy.sh | bash
cd /ynh-dev
cd yunohost
git checkout -b MercierCorentin-enh-dns-autoconf dev
git pull https://github.com/MercierCorentin/yunohost.git enh-dns-autoconf
cd ../
./ynh-dev use-git yunohost

… make possible migrations easier idk

andretheolauret · 2021-08-29T15:46:01Z

Ok, that's my first review, if there are missing things please tell me, so first during install my server tells me a traceback :

./ynh-dev use-git yunohost

Traceback (most recent call last):
  File "/ynh-dev/yunohost/data/actionsmap/yunohost_completion.py", line 64, in <module>
    with open(BASH_COMPLETION_FILE, "w") as generated_file:
FileNotFoundError: [Errno 2] No such file or directory: '/ynh-dev/yunohost/data/actionsmap/../bash-completion.d/yunohost'
[ OK ] Now using Git repository for YunoHost

Then after installation here are the different outputs of the associated commands

cp yunohost/data/other/providers_list.yml /usr/share/yunohost/other/providers_list.yml
cp: cannot stat 'yunohost/data/other/providers_list.yml': No such file or directory

But I think the real path is yunohost/data/other/providers_list.yml so now the copy has result okay

cp yunohost/data/other/registrar_list.yml /usr/share/yunohost/other/registrar_list.yml

No output, I guess the copy is good !

yunohost domain registrar catalog

Outputs all supported registrar, no issue.

yunohost domain registrar set portail.awmreunion.re cloudflare

Enter value for 'auth_username':: *******@*****.****
Enter value for 'auth_token':: **********
Enter value for 'zone_id':: *****************
Traceback (most recent call last):
  File "/usr/bin/yunohost", line 72, in <module>
    parser=parser
  File "/usr/lib/moulinette/yunohost/__init__.py", line 25, in cli
    ret = moulinette.cli(args, output_as=output_as, timeout=timeout, top_parser=parser)
  File "/usr/lib/python3/dist-packages/moulinette/__init__.py", line 131, in cli
    args, output_as=output_as, timeout=timeout
  File "/usr/lib/python3/dist-packages/moulinette/interfaces/cli.py", line 495, in run
    ret = self.actionsmap.process(args, timeout=timeout)
  File "/usr/lib/python3/dist-packages/moulinette/actionsmap.py", line 604, in process
    return func(**arguments)
  File "/usr/lib/moulinette/yunohost/domain.py", line 551, in domain_registrar_set
    return yunohost.dns.domain_registrar_set(domain, registrar, args)
  File "/usr/lib/moulinette/yunohost/dns.py", line 463, in domain_registrar_set
    dns_zone = _get_domain_settings(domain)["dns_zone"]
  File "/usr/lib/moulinette/yunohost/domain.py", line 416, in _get_domain_settings
    settings = _default_domain_settings(domain)
  File "/usr/lib/moulinette/yunohost/domain.py", line 387, in _default_domain_settings
    from yunohost.utils.dns import get_dns_zone_from_domain
  File "/usr/lib/moulinette/yunohost/utils/dns.py", line 22, in <module>
    from publicsuffixlist import PublicSuffixList
ModuleNotFoundError: No module named 'publicsuffixlist'

Module not found, trying to install publicsuffixlist..

pip3 install publicsuffixlist
Collecting publicsuffixlist
Downloading https://files.pythonhosted.org/packages/14/b9/6ae2ba427347c1c29eae03fa8e1bb4c50426519c839bf1e0cd1d0caa00a6/publicsuffixlist-0.7.8-py2.py3-none-any.whl (92kB)
100% |████████████████████████████████| 102kB 411kB/s
Installing collected packages: publicsuffixlist
Successfully installed publicsuffixlist-0.7.8

Installed successfully

yunohost domain registrar set portail.awmreunion.re cloudflare
Enter value for 'auth_username':: @.
Enter value for 'auth_token':: ***************
Enter value for 'zone_id':: ***********

No outputs, I think it's okay

yunohost domain dns-conf ***************

Info: This command shows you the *recommended* configuration. It does not actually set up the DNS configuration for you. It is your responsability to configure your DNS zone in your registrar according to this recommendation.
; Basic ipv4/ipv6 records
portail 3600 IN A **********
portail 3600 IN AAAA *************

; XMPP
_xmpp-client._tcp.portail 3600 IN SRV 0 5 5222 *************
_xmpp-server._tcp.portail 3600 IN SRV 0 5 5269 ***************
muc.portail 3600 IN CNAME portail
pubsub.portail 3600 IN CNAME portail
vjud.portail 3600 IN CNAME portail
xmpp-upload.portail 3600 IN CNAME portail

; Mail
portail 3600 IN MX 10 ************
portail 3600 IN TXT "v=spf1 a mx -all"
mail._domainkey.portail 3600 IN TXT "v=DKIM1; h=sha256; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDyO84FWnkYBboZNsT9eDWfpXWYBnrKP09rhvq+sG4RI1OY2Rx/LVjUXwM8ZVrB7hqr3ejibTeIHBv0COhDudbxw7j4uftGunI8m/vMhktVWTSuq0hE4U9KsLpqZ0fMOL3o1v1KbzhN1/W4fUlgbJhYe4pduqG/lyNO917eLuUFSQIDAQAB"
_dmarc.portail 3600 IN TXT "v=DMARC1; p=none"

; Extra
*.portail 3600 IN A ************
*.portail 3600 IN AAAA **********
portail 3600 IN CAA 128 issue "letsencrypt.org"

Outputs look ok

cat /etc/yunohost/domains.yml
cat: /etc/yunohost/domains.yml: No such file or directory

Mmmmh I think it's not okay

cat /etc/yunohost/registrars/***..yml
name: cloudflare
options:
auth_token: **_
auth_username: @.
zone_id: *************

Right now configuration is okay

yunohost domain push_config is now yunohost domain registrar push portail.awmreunion.re

yunohost domain registrar push portail.awmreunion.re

{"result":[{"id":"","name":"awmreunion.re","status":"active","paused":false,"type":"full","development_mode":0,"name_servers":["aida.ns.cloudflare.com","miguel.ns.cloudflare.com"],"original_name_servers":["ns1.digitalocean.com","ns3.digitalocean.com","ns2.digitalocean.com"],"original_registrar":null,"original_dnshost":null,"modified_on":"2021-08-29T15:00:15.706859Z","created_on":"2021-08-29T14:49:39.179627Z","activated_on":"2021-08-29T15:00:15.706859Z","meta":{"step":2,"wildcard_proxiable":false,"custom_certificate_quota":0,"page_rule_quota":3,"phishing_detected":false,"multiple_railguns_allowed":false},"owner":{"id":"","type":"user","email":""},"account":{"id":"","name":""},"permissions":["#access:edit","#access:read","#analytics:read","#app:edit","#auditlogs:read","#billing:edit","#billing:read","#cache_purge:edit","#dns_records:edit","#dns_records:read","#lb:edit","#lb:read","#legal:edit","#legal:read","#logs:edit","#logs:read","#member:edit","#member:read","#organization:edit","#organization:read","#ssl:edit","#ssl:read","#stream:edit","#stream:read","#subscription:edit","#subscription:read","#teams:edit","#teams:pii","#teams:read","#teams:report","#waf:edit","#waf:read","#webhooks:edit","#webhooks:read","#worker:edit","#worker:read","#zone:edit","#zone:read","#zone_settings:edit","#zone_settings:read"],"plan":{"id":"","name":"Free Website","price":0,"currency":"USD","frequency":"","is_subscribed":false,"can_subscribe":false,"legacy_id":"free","legacy_discount":false,"externally_managed":false}}],"result_info":{"page":1,"per_page":20,"total_pages":1,"count":1,"total_count":1},"success":true,"errors":[],"messages":[]}
{"result":[],"success":true,"errors":[],"messages":[],"result_info":{"page":1,"per_page":100,"count":0,"total_count":0,"total_pages":1}}
{"result":[{"id":"","name":"awmreunion.re","status":"active","paused":false,"type":"full","development_mode":0,"name_servers":["aida.ns.cloudflare.com","miguel.ns.cloudflare.com"],"original_name_servers":["ns1.digitalocean.com","ns3.digitalocean.com","ns2.digitalocean.com"],"original_registrar":null,"original_dnshost":null,"modified_on":"2021-08-29T15:00:15.706859Z","created_on":"2021-08-29T14:49:39.179627Z","activated_on":"2021-08-29T15:00:15.706859Z","meta":{"step":2,"wildcard_proxiable":false,"custom_certificate_quota":0,"page_rule_quota":3,"phishing_detected":false,"multiple_railguns_allowed":false},"owner":{"id":"","type":"user","email":"a*****.****@*****"},"account":{"id":"","name":"a*****.****@*****"},"permissions":["#access:edit","#access:read","#analytics:read","#app:edit","#auditlogs:read","#billing:edit","#billing:read","#cache_purge:edit","#dns_records:edit","#dns_records:read","#lb:edit","#lb:read","#legal:edit","#legal:read","#logs:edit","#logs:read","#member:edit","#member:read","#organization:edit","#organization:read","#ssl:edit","#ssl:read","#stream:edit","#stream:read","#subscription:edit","#subscription:read","#teams:edit","#teams:pii","#teams:read","#teams:report","#waf:edit","#waf:read","#webhooks:edit","#webhooks:read","#worker:edit","#worker:read","#zone:edit","#zone:read","#zone_settings:edit","#zone_settings:read"],"plan":{"id":"0feeeeeeeeeeeeeeeeeeeeeeeeeeeeee","name":"Free Website","price":0,"currency":"USD","frequency":"","is_subscribed":false,"can_subscribe":false,"legacy_id":"free","legacy_discount":false,"externally_managed":false}}],"result_info":{"page":1,"per_page":20,"total_pages":1,"count":1,"total_count":1},"success":true,"errors":[],"messages":[]}
{"result":[],"success":true,"errors":[],"messages":[],"result_info":{"page":1,"per_page":100,"count":0,"total_count":0,"total_pages":1}}
{"result":[{"id":"","name":"awmreunion.re","status":"active","paused":false,"type":"full","development_mode":0,"name_servers":["aida.ns.cloudflare.com","miguel.ns.cloudflare.com"],"original_name_servers":["ns1.digitalocean.com","ns3.digitalocean.com","ns2.digitalocean.com"],"original_registrar":null,"original_dnshost":null,"modified_on":"2021-08-29T15:00:15.706859Z","created_on":"2021-08-29T14:49:39.179627Z","activated_on":"2021-08-29T15:00:15.706859Z","meta":{"step":2,"wildcard_proxiable":false,"custom_certificate_quota":0,"page_rule_quota":3,"phishing_detected":false,"multiple_railguns_allowed":false},"owner":{"id":"","type":"user","email":"a*****.****@*****"},"account":{"id":"","name":"a*****.****@*****"},"permissions":["#access:edit","#access:read","#analytics:read","#app:edit","#auditlogs:read","#billing:edit","#billing:read","#cache_purge:edit","#dns_records:edit","#dns_records:read","#lb:edit","#lb:read","#legal:edit","#legal:read","#logs:edit","#logs:read","#member:edit","#member:read","#organization:edit","#organization:read","#ssl:edit","#ssl:read","#stream:edit","#stream:read","#subscription:edit","#subscription:read","#teams:edit","#teams:pii","#teams:read","#teams:report","#waf:edit","#waf:read","#webhooks:edit","#webhooks:read","#worker:edit","#worker:read","#zone:edit","#zone:read","#zone_settings:edit","#zone_settings:read"],"plan":{"id":"","name":"Free Website","price":0,"currency":"USD","frequency":"","is_subscribed":false,"can_subscribe":false,"legacy_id":"free","legacy_discount":false,"externally_managed":false}}],"result_info":{"page":1,"per_page":20,"total_pages":1,"count":1,"total_count":1},"success":true,"errors":[],"messages":[]}
{"result":[],"success":true,"errors":[],"messages":[],"result_info":{"page":1,"per_page":100,"count":0,"total_count":0,"total_pages":1}}
{"result":[{"id":"","name":"awmreunion.re","status":"active","paused":false,"type":"full","development_mode":0,"name_servers":["aida.ns.cloudflare.com","miguel.ns.cloudflare.com"],"original_name_servers":["ns1.digitalocean.com","ns3.digitalocean.com","ns2.digitalocean.com"],"original_registrar":null,"original_dnshost":null,"modified_on":"2021-08-29T15:00:15.706859Z","created_on":"2021-08-29T14:49:39.179627Z","activated_on":"2021-08-29T15:00:15.706859Z","meta":{"step":2,"wildcard_proxiable":false,"custom_certificate_quota":0,"page_rule_quota":3,"phishing_detected":false,"multiple_railguns_allowed":false},"owner":{"id":"","type":"user","email":"a*****.****@*****"},"account":{"id":"","name":"a*****.****@*****"},"permissions":["#access:edit","#access:read","#analytics:read","#app:edit","#auditlogs:read","#billing:edit","#billing:read","#cache_purge:edit","#dns_records:edit","#dns_records:read","#lb:edit","#lb:read","#legal:edit","#legal:read","#logs:edit","#logs:read","#member:edit","#member:read","#organization:edit","#organization:read","#ssl:edit","#ssl:read","#stream:edit","#stream:read","#subscription:edit","#subscription:read","#teams:edit","#teams:pii","#teams:read","#teams:report","#waf:edit","#waf:read","#webhooks:edit","#webhooks:read","#worker:edit","#worker:read","#zone:edit","#zone:read","#zone_settings:edit","#zone_settings:read"],"plan":{"id":"0feeeeeeeeeeeeeeeeeeeeeeeeeeeeee","name":"Free Website","price":0,"currency":"USD","frequency":"","is_subscribed":false,"can_subscribe":false,"legacy_id":"free","legacy_discount":false,"externally_managed":false}}],"result_info":{"page":1,"per_page":20,"total_pages":1,"count":1,"total_count":1},"success":true,"errors":[],"messages":[]}
{"result":[],"success":true,"errors":[],"messages":[],"result_info":{"page":1,"per_page":100,"count":0,"total_count":0,"total_pages":1}}
{"result":[{"id":"","name":"awmreunion.re","status":"active","paused":false,"type":"full","development_mode":0,"name_servers":["aida.ns.cloudflare.com","miguel.ns.cloudflare.com"],"original_name_servers":["ns1.digitalocean.com","ns3.digitalocean.com","ns2.digitalocean.com"],"original_registrar":null,"original_dnshost":null,"modified_on":"2021-08-29T15:00:15.706859Z","created_on":"2021-08-29T14:49:39.179627Z","activated_on":"2021-08-29T15:00:15.706859Z","meta":{"step":2,"wildcard_proxiable":false,"custom_certificate_quota":0,"page_rule_quota":3,"phishing_detected":false,"multiple_railguns_allowed":false},"owner":{"id":"","type":"user","email":"a*****.****@*****"},"account":{"id":"","name":"a*****.****@*****"},"permissions":["#access:edit","#access:read","#analytics:read","#app:edit","#auditlogs:read","#billing:edit","#billing:read","#cache_purge:edit","#dns_records:edit","#dns_records:read","#lb:edit","#lb:read","#legal:edit","#legal:read","#logs:edit","#logs:read","#member:edit","#member:read","#organization:edit","#organization:read","#ssl:edit","#ssl:read","#stream:edit","#stream:read","#subscription:edit","#subscription:read","#teams:edit","#teams:pii","#teams:read","#teams:report","#waf:edit","#waf:read","#webhooks:edit","#webhooks:read","#worker:edit","#worker:read","#zone:edit","#zone:read","#zone_settings:edit","#zone_settings:read"],"plan":{"id":"0feeeeeeeeeeeeeeeeeeeeeeeeeeeeee","name":"Free Website","price":0,"currency":"USD","frequency":"","is_subscribed":false,"can_subscribe":false,"legacy_id":"free","legacy_discount":false,"externally_managed":false}}],"result_info":{"page":1,"per_page":20,"total_pages":1,"count":1,"total_count":1},"success":true,"errors":[],"messages":[]}
{"result":[],"success":true,"errors":[],"messages":[],"result_info":{"page":1,"per_page":100,"count":0,"total_count":0,"total_pages":1}}
{"result":[{"id":"","name":"awmreunion.re","status":"active","paused":false,"type":"full","development_mode":0,"name_servers":["aida.ns.cloudflare.com","miguel.ns.cloudflare.com"],"original_name_servers":["ns1.digitalocean.com","ns3.digitalocean.com","ns2.digitalocean.com"],"original_registrar":null,"original_dnshost":null,"modified_on":"2021-08-29T15:00:15.706859Z","created_on":"2021-08-29T14:49:39.179627Z","activated_on":"2021-08-29T15:00:15.706859Z","meta":{"step":2,"wildcard_proxiable":false,"custom_certificate_quota":0,"page_rule_quota":3,"phishing_detected":false,"multiple_railguns_allowed":false},"owner":{"id":"","type":"user","email":"a*****.****@*****"},"account":{"id":"","name":"a*****.****@*****"},"permissions":["#access:edit","#access:read","#analytics:read","#app:edit","#auditlogs:read","#billing:edit","#billing:read","#cache_purge:edit","#dns_records:edit","#dns_records:read","#lb:edit","#lb:read","#legal:edit","#legal:read","#logs:edit","#logs:read","#member:edit","#member:read","#organization:edit","#organization:read","#ssl:edit","#ssl:read","#stream:edit","#stream:read","#subscription:edit","#subscription:read","#teams:edit","#teams:pii","#teams:read","#teams:report","#waf:edit","#waf:read","#webhooks:edit","#webhooks:read","#worker:edit","#worker:read","#zone:edit","#zone:read","#zone_settings:edit","#zone_settings:read"],"plan":{"id":"0feeeeeeeeeeeeeeeeeeeeeeeeeeeeee","name":"Free Website","price":0,"currency":"USD","frequency":"","is_subscribed":false,"can_subscribe":false,"legacy_id":"free","legacy_discount":false,"externally_managed":false}}],"result_info":{"page":1,"per_page":20,"total_pages":1,"count":1,"total_count":1},"success":true,"errors":[],"messages":[]}
{"result":[],"success":true,"errors":[],"messages":[],"result_info":{"page":1,"per_page":100,"count":0,"total_count":0,"total_pages":1}}
local_record: {'name': 'portail.portail.awmreunion.re', 'ttl': 3600, 'type': 'A', 'value': '206.189.37.195'}
local_record: {'name': 'portail.portail.awmreunion.re', 'ttl': 3600, 'type': 'AAAA', 'value': '2400:6180:0:d0::127d:6001'}
local_record: {'name': '_xmpp-client._tcp.portail.portail.awmreunion.re', 'ttl': 3600, 'type': 'SRV', 'value': '0 5 5222 portail.awmreunion.re.'}
local_record: {'name': '_xmpp-server._tcp.portail.portail.awmreunion.re', 'ttl': 3600, 'type': 'SRV', 'value': '0 5 5269 portail.awmreunion.re.'}
local_record: {'name': 'muc.portail.portail.awmreunion.re', 'ttl': 3600, 'type': 'CNAME', 'value': 'portail'}
local_record: {'name': 'pubsub.portail.portail.awmreunion.re', 'ttl': 3600, 'type': 'CNAME', 'value': 'portail'}
local_record: {'name': 'vjud.portail.portail.awmreunion.re', 'ttl': 3600, 'type': 'CNAME', 'value': 'portail'}
local_record: {'name': 'xmpp-upload.portail.portail.awmreunion.re', 'ttl': 3600, 'type': 'CNAME', 'value': 'portail'}
local_record: {'name': 'portail.portail.awmreunion.re', 'ttl': 3600, 'type': 'MX', 'value': '10 portail.awmreunion.re.'}
local_record: {'name': 'portail.portail.awmreunion.re', 'ttl': 3600, 'type': 'TXT', 'value': '"v=spf1 a mx -all"'}
local_record: {'name': 'mail._domainkey.portail.portail.awmreunion.re', 'ttl': 3600, 'type': 'TXT', 'value': '"v=DKIM1; h=sha256; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDyO84FWnkYBboZNsT9eDWfpXWYBnrKP09rhvq+sG4RI1OY2Rx/LVjUXwM8ZVrB7hqr3ejibTeIHBv0COhDudbxw7j4uftGunI8m/vMhktVWTSuq0hE4U9KsLpqZ0fMOL3o1v1KbzhN1/W4fUlgbJhYe4pduqG/lyNO917eLuUFSQIDAQAB"'}
local_record: {'name': '_dmarc.portail.portail.awmreunion.re', 'ttl': 3600, 'type': 'TXT', 'value': '"v=DMARC1; p=none"'}
local_record: {'name': '*.portail.portail.awmreunion.re', 'ttl': 3600, 'type': 'A', 'value': '206.189.37.195'}
local_record: {'name': '*.portail.portail.awmreunion.re', 'ttl': 3600, 'type': 'AAAA', 'value': '2400:6180:0:d0::127d:6001'}
pushed_record: {'action': 'create', 'type': 'A', 'name': 'portail.portail.awmreunion.re', 'content': '206.189.37.195'} → {"result":[{"id":"2c8ef36639687f7082f89976828c9ddb","name":"awmreunion.re","status":"active","paused":false,"type":"full","development_mode":0,"name_servers":["aida.ns.cloudflare.com","miguel.ns.cloudflare.com"],"original_name_servers":["ns1.digitalocean.com","ns3.digitalocean.com","ns2.digitalocean.com"],"original_registrar":null,"original_dnshost":null,"modified_on":"2021-08-29T15:00:15.706859Z","created_on":"2021-08-29T14:49:39.179627Z","activated_on":"2021-08-29T15:00:15.706859Z","meta":{"step":2,"wildcard_proxiable":false,"custom_certificate_quota":0,"page_rule_quota":3,"phishing_detected":false,"multiple_railguns_allowed":false},"owner":{"id":"","type":"user","email":"a*****.****@*****"},"account":{"id":"","name":"a*****.****@*****"},"permissions":["#access:edit","#access:read","#analytics:read","#app:edit","#auditlogs:read","#billing:edit","#billing:read","#cache_purge:edit","#dns_records:edit","#dns_records:read","#lb:edit","#lb:read","#legal:edit","#legal:read","#logs:edit","#logs:read","#member:edit","#member:read","#organization:edit","#organization:read","#ssl:edit","#ssl:read","#stream:edit","#stream:read","#subscription:edit","#subscription:read","#teams:edit","#teams:pii","#teams:read","#teams:report","#waf:edit","#waf:read","#webhooks:edit","#webhooks:read","#worker:edit","#worker:read","#zone:edit","#zone:read","#zone_settings:edit","#zone_settings:read"],"plan":{"id":"","name":"Free Website","price":0,"currency":"USD","frequency":"","is_subscribed":false,"can_subscribe":false,"legacy_id":"free","legacy_discount":false,"externally_managed":false}}],"result_info":{"page":1,"per_page":20,"total_pages":1,"count":1,"total_count":1},"success":true,"errors":[],"messages":[]}
{"result":{"id":"","zone_id":"","zone_name":"awmreunion.re","name":"portail.portail.awmreunion.re","type":"A","content":"206.189.37.195","proxiable":true,"proxied":false,"ttl":3600,"locked":false,"meta":{"auto_added":false,"managed_by_apps":false,"managed_by_argo_tunnel":false,"source":"primary"},"created_on":"2021-08-29T15:38:50.819698Z","modified_on":"2021-08-29T15:38:50.819698Z"},"success":true,"errors":[],"messages":[]}
results: True
pushed_record: {'action': 'create', 'type': 'AAAA', 'name': 'portail.portail.awmreunion.re', 'content': '2400:6180:0:d0::127d:6001'} → {"result":[{"id":"","name":"awmreunion.re","status":"active","paused":false,"type":"full","development_mode":0,"name_servers":["aida.ns.cloudflare.com","miguel.ns.cloudflare.com"],"original_name_servers":["ns1.digitalocean.com","ns3.digitalocean.com","ns2.digitalocean.com"],"original_registrar":null,"original_dnshost":null,"modified_on":"2021-08-29T15:00:15.706859Z","created_on":"2021-08-29T14:49:39.179627Z","activated_on":"2021-08-29T15:00:15.706859Z","meta":{"step":2,"wildcard_proxiable":false,"custom_certificate_quota":0,"page_rule_quota":3,"phishing_detected":false,"multiple_railguns_allowed":false},"owner":{"id":"","type":"user","email":"a*****.****@*****"},"account":{"id":"","name":"a*****.****@*****"},"permissions":["#access:edit","#access:read","#analytics:read","#app:edit","#auditlogs:read","#billing:edit","#billing:read","#cache_purge:edit","#dns_records:edit","#dns_records:read","#lb:edit","#lb:read","#legal:edit","#legal:read","#logs:edit","#logs:read","#member:edit","#member:read","#organization:edit","#organization:read","#ssl:edit","#ssl:read","#stream:edit","#stream:read","#subscription:edit","#subscription:read","#teams:edit","#teams:pii","#teams:read","#teams:report","#waf:edit","#waf:read","#webhooks:edit","#webhooks:read","#worker:edit","#worker:read","#zone:edit","#zone:read","#zone_settings:edit","#zone_settings:read"],"plan":{"id":"0feeeeeeeeeeeeeeeeeeeeeeeeeeeeee","name":"Free Website","price":0,"currency":"USD","frequency":"","is_subscribed":false,"can_subscribe":false,"legacy_id":"free","legacy_discount":false,"externally_managed":false}}],"result_info":{"page":1,"per_page":20,"total_pages":1,"count":1,"total_count":1},"success":true,"errors":[],"messages":[]}
{"result":{"id":"","zone_id":"","zone_name":"awmreunion.re","name":"portail.portail.awmreunion.re","type":"AAAA","content":"2400:6180:0:d0::127d:6001","proxiable":true,"proxied":false,"ttl":3600,"locked":false,"meta":{"auto_added":false,"managed_by_apps":false,"managed_by_argo_tunnel":false,"source":"primary"},"created_on":"2021-08-29T15:38:52.880201Z","modified_on":"2021-08-29T15:38:52.880201Z"},"success":true,"errors":[],"messages":[]}
results: True
pushed_record: {'action': 'create', 'type': 'SRV', 'name': '_xmpp-client._tcp.portail.portail.awmreunion.re', 'content': '0 5 5222 portail.awmreunion.re.'} → {"result":[{"id":"","name":"awmreunion.re","status":"active","paused":false,"type":"full","development_mode":0,"name_servers":["aida.ns.cloudflare.com","miguel.ns.cloudflare.com"],"original_name_servers":["ns1.digitalocean.com","ns3.digitalocean.com","ns2.digitalocean.com"],"original_registrar":null,"original_dnshost":null,"modified_on":"2021-08-29T15:00:15.706859Z","created_on":"2021-08-29T14:49:39.179627Z","activated_on":"2021-08-29T15:00:15.706859Z","meta":{"step":2,"wildcard_proxiable":false,"custom_certificate_quota":0,"page_rule_quota":3,"phishing_detected":false,"multiple_railguns_allowed":false},"owner":{"id":"","type":"user","email":"a*****.****@*****"},"account":{"id":"","name":"a*****.****@*****"},"permissions":["#access:edit","#access:read","#analytics:read","#app:edit","#auditlogs:read","#billing:edit","#billing:read","#cache_purge:edit","#dns_records:edit","#dns_records:read","#lb:edit","#lb:read","#legal:edit","#legal:read","#logs:edit","#logs:read","#member:edit","#member:read","#organization:edit","#organization:read","#ssl:edit","#ssl:read","#stream:edit","#stream:read","#subscription:edit","#subscription:read","#teams:edit","#teams:pii","#teams:read","#teams:report","#waf:edit","#waf:read","#webhooks:edit","#webhooks:read","#worker:edit","#worker:read","#zone:edit","#zone:read","#zone_settings:edit","#zone_settings:read"],"plan":{"id":"","name":"Free Website","price":0,"currency":"USD","frequency":"","is_subscribed":false,"can_subscribe":false,"legacy_id":"free","legacy_discount":false,"externally_managed":false}}],"result_info":{"page":1,"per_page":20,"total_pages":1,"count":1,"total_count":1},"success":true,"errors":[],"messages":[]}
{"result":null,"success":false,"errors":[{"code":1004,"message":"DNS Validation Error","error_chain":[{"code":9101,"message":"service is a required data field."}]}],"messages":[]}
Warning: unable to retrieve string to translate with key 'log_domain_registrar_push' for default locale 'locales/en.json' file (don't panic this is just a warning)
Info: The operation 'log_domain_registrar_push' could not be completed. Please share the full log of this operation using the command 'yunohost log share 20210829-153836-domain_registrar_push-portail.awmreunion.re' to get help
Traceback (most recent call last):
  File "/usr/bin/yunohost", line 72, in <module>
    parser=parser
  File "/usr/lib/moulinette/yunohost/__init__.py", line 25, in cli
    ret = moulinette.cli(args, output_as=output_as, timeout=timeout, top_parser=parser)
  File "/usr/lib/python3/dist-packages/moulinette/__init__.py", line 131, in cli
    args, output_as=output_as, timeout=timeout
  File "/usr/lib/python3/dist-packages/moulinette/interfaces/cli.py", line 495, in run
    ret = self.actionsmap.process(args, timeout=timeout)
  File "/usr/lib/python3/dist-packages/moulinette/actionsmap.py", line 604, in process
    return func(**arguments)
  File "/usr/lib/moulinette/yunohost/domain.py", line 556, in domain_registrar_push
    return yunohost.dns.domain_registrar_push(domain)
  File "/usr/lib/moulinette/yunohost/log.py", line 379, in func_wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/moulinette/yunohost/dns.py", line 576, in domain_registrar_push
    results = client.execute()
  File "/usr/lib/python3/dist-packages/lexicon/client.py", line 71, in execute
    return self.provider.create_record(record_type, name, content)
  File "/usr/lib/python3/dist-packages/lexicon/providers/cloudflare.py", line 53, in create_record
    '/zones/{0}/dns_records'.format(self.domain_id), data)
  File "/usr/lib/python3/dist-packages/lexicon/providers/base.py", line 90, in _post
    return self._request('POST', url, data=data, query_params=query_params)
  File "/usr/lib/python3/dist-packages/lexicon/providers/cloudflare.py", line 145, in _request
    r.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.cloudflare.com/client/v4/zones/***************/dns_records

Sorry, for truncating personal informations, please ask me if you need them... I can't push dns config and with this config only portail.portail.awmreunion.re was added in cloudflare instead of portail.awmreunion.re and all others dns records. In the last ouput there are also outputs of lexicon, to help you more. But i think the issue is in local dns record set by yunohost instance

Sorry for this very huge review but I really want the enhancement added to the next version. Thanks for reading !

EDIT (ljf) : i remove your email from the json output

zamentur · 2021-08-30T08:46:27Z

service is a required data field.

It seems the cloudflare API need a service key. It's probably an issue with the version of lexicon.

andretheolauret · 2021-08-30T12:58:18Z

service is a required data field.

It seems the cloudflare API need a service key. It's probably an issue with the version of lexicon.

It’s not relate to a servIce key because cloud flare SRV Records need an service entry like XMPP as service,

also there’s error in dns scheme because lexicon push portail.portail.awmreunion.re but the real domain should be just portail.awmreunion.re

https://www.cloudflare.com/fr-fr/learning/dns/dns-records/dns-srv-record/

andretheolauret · 2021-08-30T18:25:56Z

I tested on another yunohost instance with another domain and another provider (digitalocean) but same issue happen with SRV record..

…code snippet

… stuff to create/update/delete

Paco and others added 9 commits March 9, 2021 21:34

sample _get_domain_and_subdomains_settings()

9a8cbbd

First implementation of configurable dns conf generation

c111b9c

Add loading of domain settings. Generate defaults for missing entries…

1a4d02c

… (and backward-compability)

Merge branch 'enh-dns-autoconf' of https://github.com/MercierCorentin…

01fe61c

…/yunohost into enh-dns-autoconf

Integration of domain settings loading/generation with domains DNS en…

8dd5859

…tries generation

Saving domain settings in /etc/yunohost/domains.yml

cc3c073

Add TODOs for locales

fa5b319

Fix old_domains not assigned

f295dff

Check if DNS zone is owned by user

3b6599f

alexAubin marked this pull request as draft March 22, 2021 00:43

Paco and others added 17 commits March 22, 2021 01:57

Change API naming etc. Expect a new change to come!

fa818a4

Merge branch 'enh-dns-autoconf' of https://github.com/MercierCorentin…

4876860

…/yunohost into enh-dns-autoconf

XMPP configuration for subdomains (i.e. not owned zone dns)

b367505

Merge branch 'enh-dns-autoconf' of https://github.com/MercierCorentin…

223d260

…/yunohost into enh-dns-autoconf

Add yunohost domain config list command

503a5ed

Merge branch 'enh-dns-autoconf' of https://github.com/MercierCorentin…

1dcf9ab

…/yunohost into enh-dns-autoconf

Merge branch 'dev' of https://github.com/YunoHost/yunohost into enh-d…

4a92f97

…ns-autoconf

Domain settings: transform cli to be like app settings

afe6287

Merge branch 'enh-dns-autoconf' of https://github.com/MercierCorentin…

132085b

…/yunohost into enh-dns-autoconf

fix . set operation still not working.

e16f14f

Merge branch 'dev' of https://github.com/YunoHost/yunohost into enh-d…

ddeea56

…ns-autoconf

~ working push_config. Tested with Gandi. To be improved!

eeab7cd

Add providers parameter list

bb140b2

Add first version of set domain provider

51d6d19

Uncomment push_config function

5859028

fix: do not delete provider options when loading them

d4b4024

connect domain_push_config to the in-file provider options

914bd1f

zamentur requested changes May 24, 2021

View reviewed changes

Apply easy fixes from code review

e40f8fb

Co-authored-by: ljf (zamentur) <zamentur@users.noreply.github.com>

alexAubin added 4 commits August 28, 2021 17:58

domain remove: Improve file/folder cleanup code

808a69c

Semantics

5f338d3

Use moulinette helpers to read/write yaml

13f2cab

Misc cleanup

9f4ca2e

alexAubin added 11 commits August 28, 2021 19:10

domains.py: Attempt to clarify build_dns_zone?

756e604

More cleanup / simplify / homogenize

2f3467d

Cleanup get_dns_zone_from_domain utils

22aa1f2

Move dig() to utils.dns

8438efa

domains.py: Add cache for domain_list

0ec1516

Savagely split the dns/registrar stuff in a new dns.py

1eb0599

Misc fixes

7e048b8

domain settings: Only store the diff with respect to defaults, should…

951c669

… make possible migrations easier idk

Moar fikses

dded1cb

domain_push_config -> domain_registrar_push

5a93e06

Add logging to domain_registrar_push

4089c34

dns/lexicon: Tidying up the push function

1c2fff7

alexAubin mentioned this pull request Sep 1, 2021

Improve default app management on a domain YunoHost/issues#40

Closed

alexAubin added 4 commits September 1, 2021 17:13

registrars: remove unimplemented or unecessary stuff

01bc676

Add a small _assert_domain_exists to avoid always repeating the same …

d5b1eec

…code snippet

Various fixes after tests on OVH

e4783aa

Various changes to try to implement a proper dry-run + proper list of…

d2dea2e

… stuff to create/update/delete

alexAubin mentioned this pull request Sep 4, 2021

Configure automatically the DNS records using lexicon #1315

Merged

alexAubin merged commit 0da2f13 into YunoHost:dev Sep 19, 2021

4.3.x automation moved this from Review in progress to Done Sep 19, 2021

renne mentioned this pull request Jan 24, 2023

Support Lets Encrypt wildcard certificate YunoHost/issues#2089

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configure automatically the DNS records using lexicon #1194

Configure automatically the DNS records using lexicon #1194

ppom0 commented Mar 22, 2021 •

edited

ppom0 commented May 21, 2021

zamentur left a comment •

edited by alexAubin

zamentur May 24, 2021

zamentur May 24, 2021

zamentur May 24, 2021

zamentur May 24, 2021

zamentur commented Aug 28, 2021

andretheolauret commented Aug 29, 2021 •

edited by zamentur

zamentur commented Aug 30, 2021

andretheolauret commented Aug 30, 2021 •

edited

andretheolauret commented Aug 30, 2021

Configure automatically the DNS records using lexicon #1194

Configure automatically the DNS records using lexicon #1194

Conversation

ppom0 commented Mar 22, 2021 • edited

The problem

Solution

PR Status

How to test

ppom0 commented May 21, 2021

zamentur left a comment • edited by alexAubin

Choose a reason for hiding this comment

zamentur May 24, 2021

Choose a reason for hiding this comment

zamentur May 24, 2021

Choose a reason for hiding this comment

zamentur May 24, 2021

Choose a reason for hiding this comment

zamentur May 24, 2021

Choose a reason for hiding this comment

zamentur commented Aug 28, 2021

andretheolauret commented Aug 29, 2021 • edited by zamentur

zamentur commented Aug 30, 2021

andretheolauret commented Aug 30, 2021 • edited

andretheolauret commented Aug 30, 2021

ppom0 commented Mar 22, 2021 •

edited

zamentur left a comment •

edited by alexAubin

andretheolauret commented Aug 29, 2021 •

edited by zamentur

andretheolauret commented Aug 30, 2021 •

edited