Problems syncing imported data

[xj25vm]

I can't work out from the description if TBSync is supposed to do a two way sync? I've got it configured and connected to a Horde back end, and it has downloaded data from Horde. But it doesn't seem to be pushing anything back from Thunderbird to Horde. Is it supposed to be able to, or not? Thank you

Edit: checking the TBSync account manager, I can see against the calendar status: "status.js-error-in-calendarsync.sendLocalChanges". So it looks like at least it is trying to update data back to Horde

[jobisoft]

In general, yes. TbSync is not automatically sending your local changes as you edit them, but only if you hit the sync button in the TbSync account manager or set up auto sync (sync every x minutes). This is not ideal, but the current implementation. At some later point I will implement push services, but do not ask me when, I cannot tell.

Do you see your changes pushed to the horde server, if you hit the sync button?

[jobisoft]

Can you retry with the latest beta TbSync v0.6.8.3 (2017.12.28) ?

Can i have a debug log of when that error (status.js-error-in-calendarsync.sendLocalChanges) happens?

[xj25vm]

Sorry for the delay. Could we get this reopened, as the error is still there. I've retried with 6.9.1 and I get the same error. Please see attached debug log.

[jobisoft]

Please try again with beta7.

[xj25vm]

With beta7, I get a different error. It counts down 30s on "Waiting for acknowledgement of local changes" and then "Communication timeout". Please note that this is a large calendar, with at least several thousand entries. However, all of them are already identical locally and at the far end. I only added one locally, and tried to sync it back into the backend. I am on a local lan, not through a slow vpn, in case it makes a difference. Please see attached log.

[jobisoft]

please increase the timeout:

Go to thunderbird options -> Advanced -> Edit Config and search for tbsync. You will get en entry for

extensions.tbsync.eas.timeout

which defaults to 30000 (ms) which are 30s. Try to set it higher and see if it makes any difference.

[jobisoft]

something is very odd, the log shows, that he wants to sync 100 and more old entries (from 2005) to the server. can you please start from scratch? disconnect the account and reconnect?

[xj25vm]

Sorry - I just realised that my comment above was wrong. The local copy contains all the entries, going back to 2005 - which I've loaded from a text file export. The back-end is empty. Does that make more sense?

Edit: I've increased the timeout and tried again. There are 9881 entries in the local calendar, which it is trying to sync to the backend. In my case, "Waiting for acknowledgement of local changes" seems to need anywhere between 35s-40s - so a timeout of 60s/60000 seems to work reliably. I can see now it is syncing 50 at a time. I guess it needs the longer timeout because of the large calendar, and this is not a fast machine (laptop with a Celeron 1017U cpu and a regular hdd, on Linux).

Has Lightning moved to using SQLite for its data format, or is it still using that old Mork format, which was slow on large data sets?

[jobisoft]

I am just using the Lighning API, so I do not know how things work under the hood.

So you no longer have issues with beta7? I will later increase the default timeout from 30s to 90s.

Can we close this issue?

[xj25vm]

Maybe I've spoken too soon. It is still in the process of syncing the 9000 calendar entries. Every few hundred entries, I get either "UNKNOWN_NETWORK_ERROR" or "Sync failed. Server responds with status <Sync.Collections.Responses.Add[25].Status = 6>."

If I press "Syncronize this account" again, it continues to transfer another few hundred entries, before I see one of the two errors above again. Does it look like a different problem? Should I open a separate bug report?

[jobisoft]

Keeping this issue is fine.

You have hit two bugs.

1. If TbSync prepared an element to be send to the server, he marks it as done right away and does not wait for ack from the server. So if the server fails to accept (like response.add.status=6), tbSync still thinks it has synced that item and will not send it again. This will be fixed in the next version. So if you clear your remote calender, start from scratch (disconnect/connect) and re-import your data in TB and sync again, all those errors come back.

2. You are importing data, so you are not using the Thunderbird GUI itself to create the calendar items. That's why your data "looks" different from what I am used to and I may have to add more checks, if an element property is actually there. I need at least the last 100 lines of a full debug log of such an error, most important the javascript error and the item data being send.

[xj25vm]

Please find attached the log for "UNKNOWN_NETWORK_ERROR". I left only the last 1600 lines (it had 36000 lines). Please let me know if you need any other data.

tbsync_debug3.log

[jobisoft]

There is not much I can do about that error, tbSync was not able to connect to your server (using standard Thunderbird API).

I can only fix javascript errors.

Any chance you could send me your import file via email, so I can try to reproduce this here directly?I would need the entire file, because we do not know, which entry is the bad one...

[xj25vm]

That's fine - I've sent you the calendar data as an email attachment. Many thanks for your help so far.

[jobisoft]

This is taking very long...

Is there a true need to sync 10 years old calendar data to the server? I am thinking about adding a default limit of 1 year, which can be changed in the settings. What do you think?

[xj25vm]

I think many calendaring software have a similar default limit when syncing older calendar entries - so it would make sense to limit the import/sync to 1 year by default - as long as it can be changed in the settings. I guess it might be still worth persisting with testing this particular data set, to find out why the sync process breaks?

As a side note, I have tested a lot of calendar sharing solutions over the years - and all of them had the same two weaknesses - they couldn't cope with large data sets, and they couldn't cope if the data was slightly different from what they expected - be it a missing field, or the date format, or other things.

At least it is good that the current TBSync interface gives a clear indication of progress - i.e. how many entries have been processed already. I have encountered a lot of calendar import/sync interfaces which just froze for hours on end while importing or syncing large data sets - because the developer never considered that the import or sync might take more than a few minutes - and never implemented appropriate progress indication in the interface.

[jobisoft]

Please check 16.4.2011, Alan Harris- iPad tuition

It has a reminder AFTER the event started. How did that happen? Intentionally?

EDIT: This is not an import issue, the alarm in the importfile is also after

[jobisoft]

@#26 (comment) Yeah, corner cases are always a pain

[jobisoft]

item.startDate is undefined

Beautiful data set...

[jobisoft]

Please start from scratch* and retry with beta1

• disconnect account, remove all calendar entries on the server, reconnect, re-import, sync

[xj25vm]

item.startDate is undefined

Could I have the date for that appointment, please? I can't really think of any reason why that should be the case.

Please start from scratch* and retry with beta1

I will do and report back here

[jobisoft]

I did not identify the item in question, I just fixed the error. With beta1 I was able to sync the entire dataset to a Kopano server. I could not test with Horde myself.

[xj25vm]

Just an update - about 4000 records out of the 9000 have been synced, before starting to hit "UNKNOWN_NETWORK_ERROR" again. Restarting the sync manually seems to work. Is TBSync suppose to recover automatically after "UNKNOWN_NETWORK_ERROR" and continue?

[jobisoft]

If you get any other errors on your import, tell me :-)

[xj25vm]

Sure. After another 1200 items (I think), another similar error, on 2009-10-05. I truncated the Subject field again, and resumed the sync.

And then another one on 2009-10-09 - and probably there will be more :-)

[jobisoft]

How about that: I truncate the Subject (indicated by ... at the end) and add the full subject to the body, as

Too long Subject: <subject>

This would get it out of my feet without loosing data.

[xj25vm]

That would be an option. I think a more rigorous approach would be to continue with the sync and finish all the items which can be synced, and at the end to present a list of all the items which could not be synced because of incompatibility between Thunderbird and the back-end. This way the user can choose what to do with them. I think usually a sync software silently making changes on the data would be considered not really acceptable.

[jobisoft]

I really do not want to spend dev time on corner cases. Your suggestion req. a lot of changes to the sync process.

I am down to two options:

• hard fail like now, but with a specific error (event xy has a to long subject)
• silent truncation with backup in body

Since I agree with your statement about data integrity, I would go for option 1. Ok?

[xj25vm]

Would it be possible to hard-fail like now, but skip the affected items and continue with the others - or would that require a lot of changes to the code?

[jobisoft]

That is the exactly what would need refactoring :-) I will think about this ...

[xj25vm]

Hmm - yes, that is a tricky one. As things are now, it is quite a problem. It is perfectly possible for users to put in lots of text in the Subject field - as Thunderbird allows it. And then the sync stops completely - and without the additional debug output, it doesn't even tell users why or where the problem is.

[jobisoft]

Do not manually truncate your subjects, so that we still have something to test against. I will provide a solution over the weekend.

[xj25vm]

I have truncated about 3 of them - but I still have the original .ics file - so I could restart the import and sync from scratch if necessary.

[jobisoft]

Please try this version.

It should skip bad items and throw a final message at the end. Next time you sync, it should retry those failed items and throw the same message again.

[jobisoft]

If you get another error (network or something else that throws hard) before coming to the end, the information of the skipped items is lost, but they are still in the changelog (thus retried on next sync), so eventually you should see the list of failed items.

[xj25vm]

The first problem is that it doesn't seem to recognise the sync state left by the previous version of TBSync. After installing the version you suggested above, it attempts a sync, and then it shows "OK: Calendar" - although there should be another 6000 items to sync (I checked the count in the back-end). Do you want me to remove the local and remote calendars, re-import the .ics file and re-start from scratch - or try something else?

[jobisoft]

Yeah, there was a change to the changelog data format. Restart from scratch.

[jobisoft]

Did you restart the test yet?

I continued to work on the soft fail concept and I would consider it "done" now, but since I had to change a lot, I would like to ask you to test softfailtest2.

You will see, that your data set contains another "error": There are 234 tasks in your import data, which cannot by synced to a calendar folder. You will get the message at the very end. :-)

[jobisoft]

For the test with softfail2 you can set

extensions.tbsync.eas.maxitems = 50

again.

[xj25vm]

Thank you. I will re-run the sync over the next few days and report back

[xj25vm]

It has synced everything, except for 260 items. Any ideas what "wrong folder type" means?

By the way - I really like the verbose error message, full of useful details :-)

[jobisoft]

Wrong folder type means, you have tasks in a Calendar folder. For thunderbird, this is no problem, but I cannot sync mixed types to EAS (EAS has distinct calendar and tasks folders). What would be a better but still short error type/message?

[xj25vm]

Maybe "task item in calendar folder" and "calendar item in task folder" (if it happens the other way around)? I suppose the correct error message would be "wrong item type" - but I think this would be too vague - nobody would know what it really means.

It seems that this data set has been more challenging than I realised. I guess it helped to fix plenty of future bugs, though :-)

[jobisoft]

Yep, I will change those error messages. I will close this issue soon, it was indeed a beautiful dataset and i want to thank you very much for your testing. All changes have been migrated to beta4. May I put you on the beta testing list? The release of the v0.7 branch was a mess and I had to release lots of quick fixes. For the future, before releasing a new version to Mozilla, I would like to have some user feedback. If you are interested, just drop me an email: john.bieling@gmx.de

[xj25vm]

Thank you for taking the time to build fixes and release them.

Do add me to the beta testing list. I might not find the time every time to test TBSync releases - but I will try.

[xj25vm]

Just a quick suggestion before I forget - I wonder if it would be better to have a lower number for "extensions.tbsync.eas.maxitems" by default - maybe 10? It seems that on slower computers with larger calendars, the "waiting for acknowledgement of local changes" step takes quite a long time when it is set to 50 - and the Thunderbird interface is very sluggish (I wonder if that is because TB is still single threaded?)

[jobisoft]

Many server do not like such low maxitems, because the load is much higher, so I will not change the default, but you may of course set it as you like. I hope in the future, there will be better background thread support in Thunderbird.

[jobisoft]

Please update your beta version of TbSync to 0.7.6.8 or any later version available at the release page.

This version now includes an update notification, so I no longer have to contact all beta users, to update their TbSync versions:

https://raw.githubusercontent.com/jobisoft/TbSync/master/screenshots/update_notification.png

While on the beta release channel, I strongly suggest to update, if a newer version is available. Once a new stable version gets released, you automatically switch back to the stable release channel (you no longer receive update notifications for beta versions).