DC synchronization problems

[narogon]

Could you check this problem I describe here:

http://lists.etherlab.org/pipermail/etherlab-users/2018/003281.html

Some improvements for the current code is change the order of (in lcec_main.c)

  // send process data
  rtapi_mutex_get(&master->mutex);
  

  // update application time
  master->app_time += master->app_time_period;
  ecrt_master_application_time(master->master, master->app_time);

  // sync ref clock to master
  if (master->sync_ref_cycles > 0) {
    if (master->sync_ref_cnt == 0) {
      master->sync_ref_cnt = master->sync_ref_cycles;
      ecrt_master_sync_reference_clock(master->master);
    }
    master->sync_ref_cnt--;
  }

  // sync slaves to ref clock
  ecrt_master_sync_slave_clocks(master->master);

  // send domain data
  ecrt_domain_queue(master->domain);
  ecrt_master_send(master->master);
  rtapi_mutex_give(&master->mutex);

Moving ecrt_domain_queue(master->domain); after rtapi_mutex_get(&master->mutex);.

By the way could you try to implement the b dc sync mode that Graeme Foot described here http://lists.etherlab.org/pipermail/etherlab-users/2016/003013.html???

Many thanks for your work
BR
Ignacio Rosales

[sittner]

Please could you try this:

https://github.com/sittner/linuxcnc-ethercat/tree/fix-dc-sync

I am not able to really test it at the moment, but I got good initial sync behavior with rt preempt and refClockSyncCycles set to 1.

Feedback is highly welcome since I also run into DC sync trouble while implementing AX6200 support.

I'll try to implement variant 'B' as soon as I go back to the AX6200 stuff (in two weeks).

Kind Regards
Sascha

[sittner]

OK, i've commited the first preview of sync-master-cycle-to-reference-clock to https://github.com/sittner/linuxcnc-ethercat/tree/fix-dc-sync. No testing yet. To check out you need to apply the included patch to linuxcnc and set refClockSyncCycles to -1 (or an other negative value). This feature only works with RT_PREEMPT.

[narogon]

I'm going to try it this afternoon.

Many thanks for your work. I will bring feedback as soon as possible.

BR
Nacho Rosales

[narogon]

Sascha I've testing your solution in a new machine. First of all many many thanks to program it and trying to solve this problem.

It works but doesn't solve the dc sync problem. I have not been able to test it in the machine that gets desynchronized and to SAFEOP state (the one with 37 servos) but I could try it in this new machine using the Cyclic position mode that is what better gives away the problem.

With the cyclic position mode you could hear how the increments of positions, that linuxcnc sends equals in case of a constant movement, don't arrive at same time making the servo change the speed noisily.

I installed rt-preemt and the PLL patch to linuxcnc in order to work with this feature using refSyncCycle -1.

With wireshark I could see where the problem is.

Using the follow network PC --> switch --> servo
--> Another PC with Wireshark running

In the case that twincat is the master the time between packets sent by the master is 1 ms +- 0,02 ms
Using lcec the deviation becomes + 0,8 ms

Could the b method solve this problem?

Any test or result you could need don´t hesitate to ask me.

BR
Nacho Rosales

[sittner]

I've tested the new stuff (fixed "sync refclock to master" mode with refSyncCycle>0 and new "sync master to refclock" mode with refSyncCycle < 0) on 3 different PCs with Delta, Stoeber and Beckhoff drives and got very nice results (latency in the 40us range).

What kind of PC and what NIC are you using? What about latency-test? Have you tried the isolcpus kernel parameter and disabled hyperthreading (if possible)?

I do not understand the use of a switch in an EtherCAT network. This will not work correctly. However the jitter of +- 0,8ms is not acceptable.

If you are using refSyncCycle = -1 (what actually means using method B), the lcec-master pins pll-out and pll-err will get updated. It may help to make a HAL scope session over this two pins to get an idea about the noise in this system. You could even try to tune the PLL PI controller by changing the values of the pll-i and pll-p HAL parameters.

I have pushed a recent commit that limits the PLL output to +/- 0.1% of the cycle time (prev was +/- 1%). Maybe that helps a little bit reducing the initial swing (however, this limit should not be hitten before).

Any feedback is welcome.
Regards
Sascha

[narogon]

Sascha it seems to work now for me. Tested over Hiwin D2 and many Beckhoff cards.

I will test it over Delta ASDA. I'm not pretty sure what was the problem with the first computer I tried this solution. Tomorrow I will try over a more complex system with 40 servos (the system where I found the problem first time) and bring here the feedback.

The use of a switch was only for testing purposes and to capture the network packets.

Could it be possible to install the PLL patch to a 2.8 (master branch of linuxcnc) version?? I found some problems patching it and also doing it manually I gets some compilation problems. Could you check it?

Many many thanks for your time and work.
Regards,
Nacho Rosales

[sittner]

No reported problems so far, so I close this issue.