Snapmirror-Resync fails on different Ontap-versions

[robertattenberger]

Summary

When I try to resync a broken-off snapmirror-relationsship between two Ontap-volumes on two different Ontap-clusters running on different Ontap-Versions the task fails.

The error message is: "Error patching SnapMirror: {''state'': ''snapmirrored''}: job reported error: Timeout error: Process still running, received {''job'': {''uuid'': ''xxxxxxxx-xxxx-xxxx-xxxxx-xxxxxxxxxxxx'', ''_links'': {''self'': {''href'': ''/api/cluster/jobs/xxxxxxxx-xxxx-xxxx-xxxxx-xxxxxxxxxxxx''}}}}.".

The source cluster is on Ontap 9.11.1P12.
The destination cluster is on Ontap 9.13.1P8

Ansible-module: netapp.ontap.na_ontap_snapmirror

The error occurs as well with the snapmirror-policy MirrorAndVault and MirrorLatest.

The error occurs about 60 seconds after the task started.

The error does not occour when the source is on 9.13.1P8 and the destination is on 9.11.1P12.###

Component Name

netapp.ontap.na_ontap_snapmirror

Ansible Version

$ ansible --version
ansible [core 2.15.3]
  config file = MYHOMEDIRECTORY/storage_ansible_bugfix/ansible.cfg
  configured module search path = ['MYHOMEDIRECTORY/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.11/site-packages/ansible
  ansible collection location = MYHOMEDIRECTORY/storage_ansible_bugfix/collections
  executable location = /usr/bin/ansible
  python version = 3.11.5 (main, Sep 22 2023, 15:34:29) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] (/usr/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True

ONTAP Collection Version

$ ansible-galaxy collection list

# MYHOMEDIRECTORY/storage_ansible_bugfix/collections/ansible_collections
Collection                    Version
----------------------------- -------
ansible.utils                 2.11.0
ansible.windows               2.1.0
awx.awx                       23.2.0
community.crypto              2.15.1
community.general             7.5.0
community.windows             2.0.0
infoblox.nios_modules         1.5.0
netapp.ontap                  22.9.0
netapp.storagegrid            21.11.1

# /usr/lib/python3.11/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
...
ansible.netcommon             5.1.2
ansible.posix                 1.5.4
ansible.utils                 2.10.3
ansible.windows               1.14.0
...
community.crypto              2.15.0
...
community.general             7.3.0
...
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.22.0
netapp.elementsw              21.7.0
netapp.ontap                  22.7.0
netapp.storagegrid            21.11.1
netapp.um_info                21.8.0
netapp_eseries.santricity     1.4.0
...

ONTAP Version

sourcecluster::> version
NetApp Release 9.11.1P12: Fri Sep 22 11:58:50 UTC 2023


destinationcluster::> version
NetApp Release 9.13.1P8: Fri Feb 23 14:13:29 UTC 2024

Playbook

---
- hosts: localhost
  gather_facts: false
  vars:
    source_svm: mysourcesvm
    source_vol: myvol
    destination_cluster: mydestcluster
    destination_svm: mydestsvm
    destination_vol: myvol_dest
  tasks:
    - name: Resync Snapmirror
      netapp.ontap.na_ontap_snapmirror:
        destination_endpoint:
          path: "{{ destination_svm }}:{{ destination_vol }}"
        hostname: "{{ hostvars[destination_cluster].cluster_host.netapp_fqdn }}"
        https: true
        password: "{{ hostvars[destination_cluster].password }}"
        relationship_state: active
        source_endpoint:
          path: "{{ source_svm }}:{{ source_vol }}"
        state: present
        transferring_time_out: 86400
        update: false
        use_rest: always
        username: "{{ hostvars[destination_cluster].username }}"
        validate_certs: false
        connection_type: ontap_ontap
...

Steps to Reproduce

mysourcecluster::> vol create -vserver mysourcesvm -volume myvol -size 120G -junction-path /myvol -aggregate mysourceaggr -state online
mydestcluster::> vol create -vserver mydestsvm -volume myvol_dest -type DP -size 150G -aggregate mydestaggr -state online
mydestcluster::> snapmirror create -source-path mysourcesvm:myvol -destination-path mydestsvm:myvol_dest -vserver mydestsvm -policy MirrorAndVault
mydestcluster::> snapmirror initialize -destination-path mydestsvm:myvol_dest
mydestcluster::> vol mount -vserver mydestsvm -volume myvol_dest -junction-path /myvol_dest
mydestcluster::> snapmirror break -destination-path mydestsvm:myvol_dest

ansible-playbook --ask-vault-pass playbooks/snapmirror_pb/resync.yml -vvv

Expected Results

I expected that the task finishes with status ok

Actual Results

The tasks finishes with an error and the following output. The snapmirror-resync is done but the playbook fails and subsequent tasks in the playbook are not executed.



ansible-playbook [core 2.15.3]
  config file = MYHOMEDIRECTORY/storage_ansible_bugfix/ansible.cfg
  configured module search path = ['MYHOMEDIRECTORY/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.11/site-packages/ansible
  ansible collection location = MYHOMEDIRECTORY/storage_ansible_bugfix/collections
  executable location = /usr/bin/ansible-playbook
  python version = 3.11.5 (main, Sep 22 2023, 15:34:29) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] (/usr/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True
Using MYHOMEDIRECTORY/storage_ansible_bugfix/ansible.cfg as config file
Vault password:
host_list declined parsing MYHOMEDIRECTORY/storage_ansible_bugfix/inventories/inventory.yml as it did not pass its verify_file() method
script declined parsing MYHOMEDIRECTORY/storage_ansible_bugfix/inventories/inventory.yml as it did not pass its verify_file() method
Parsed MYHOMEDIRECTORY/storage_ansible_bugfix/inventories/inventory.yml inventory source with yaml plugin
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: resync.yml ***********************************************************
1 plays in playbooks/snapmirror_pb/resync.yml

PLAY [localhost] ***************************************************************

TASK [Resync Snapmirror] *******************************************************
task path: MYHOMEDIRECTORY/storage_ansible_bugfix/playbooks/snapmirror_pb/resync.yml:11
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: MYLDAPID
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /tmp/ansible-$USER `"&& mkdir "` echo /tmp/ansible-$USER/ansible-tmp-1713201924.6106207-1065384-95121509992597 `" && echo ansible-tmp-1713201924.6106207-1065384-95121509992597="` echo /tmp/ansible-$USER/ansible-tmp-1713201924.6106207-1065384-95121509992597 `" ) && sleep 0'
Using module file MYHOMEDIRECTORY/storage_ansible_bugfix/collections/ansible_collections/netapp/ontap/plugins/modules/na_ontap_snapmirror.py
<127.0.0.1> PUT MYHOMEDIRECTORY/.ansible/tmp/ansible-local-10653802jzbmbze/tmpeog5vpv8 TO /tmp/ansible-MYLDADPID/ansible-tmp-1713201924.6106207-1065384-95121509992597/AnsiballZ_na_ontap_snapmirror.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /tmp/ansible-MYLDAPID/ansible-tmp-1713201924.6106207-1065384-95121509992597/ /tmp/ansible-MYLDAPID/ansible-tmp-1713201924.6106207-1065384-95121509992597/AnsiballZ_na_ontap_snapmirror.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3.11 /tmp/ansible-MYLDAPID/ansible-tmp-1713201924.6106207-1065384-95121509992597/AnsiballZ_na_ontap_snapmirror.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /tmp/ansible-MYLDAPPID/ansible-tmp-1713201924.6106207-1065384-95121509992597/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
NoneType: None
fatal: [localhost]: FAILED! => changed=false
  invocation:
    module_args:
      cert_filepath: null
      clean_up_failure: false
      connection_type: ontap_ontap
      create_destination: null
      destination_cluster: null
      destination_endpoint:
        cluster: null
        consistency_group_volumes: null
        ipspace: null
        path: mydestsvm:myvol_dest
        svm: null
      destination_path: null
      destination_volume: null
      destination_vserver: null
      feature_flags: null
      force_ontap_version: null
      hostname: mydestcluster.DOMAIN
      http_port: null
      https: true
      identity_preservation: null
      identity_preserve: null
      initialize: true
      key_filepath: null
      max_transfer_rate: null
      ontapi: null
      password: VALUE_SPECIFIED_IN_NO_LOG_PARAMETER
      peer_options: null
      policy: null
      relationship_info_only: false
      relationship_state: active
      relationship_type: null
      schedule: null
      source_cluster: null
      source_endpoint:
        cluster: null
        consistency_group_volumes: null
        ipspace: null
        path: mysourcesvm:myvol
        svm: null
      source_hostname: null
      source_password: null
      source_path: null
      source_snapshot: null
      source_username: null
      source_volume: null
      source_vserver: null
      state: present
      transferring_time_out: 86400
      update: false
      use_rest: always
      username: USERNAME
      validate_certs: false
      validate_source_path: true
  msg: 'Error patching SnapMirror: {''state'': ''snapmirrored''}: job reported error: Timeout error: Process still running, received {''job'': {''uuid'': ''xxxxxxxx-xxxx-xxxx-xxxxx-xxxxxxxxxxxx'', ''_links'': {''self'': {''href'': ''/api/cluster/jobs/xxxxxxxx-xxxx-xxxx-xxxxx-xxxxxxxxxxxx''}}}}.'

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0
   failed=1    skipped=0    rescued=0    ignored=0

[suhasbshekar]

https://discord.com/channels/855068651522490400/1062048885847117935/threads/1229302943568629770