weed backup tool would miss syncing the updated fids

[hxiaodon]

Chris:
The weed backup tool now only remove the fids which the src do not have but dest have, and add the ones which src have but desc does not have at the backup place.
For those files which are updated many times, I think that the backup tool will do nothing for them.

If this is a bug, do you have any good suggestion to fix it? e.g . Add the file crc32 checksum value in fetchVolumeFileEntries() function(This need to scan the whole volume and get it from each needle),and for the fid keys in both src and dest index, if the corresponding file size and checksum value are both same, ignore syncing.

Hope for your suggestions.Thanks!

[chrislusf]

Could you help me to think of a case where current backup strategy would fail? If it does happen, we should switch to more strict way to backup.

[hxiaodon]

OK!
Suppose we have an online empty volume file(the max volume number for the volume service is 1), at the first time, the weed back tool running will sync nothing.

Then the weed client add file A,B,C into above volume. and at this time, the weed back tool running will sync A,B and C to the backup place correctly

Then the weed client add file D, delete file A ,and update file B, after this, the weed backup tool running will add file D, delete file A but do nothing for the updated file B

Does above case make sense?

[chrislusf]

Yes. We will need a new simpler algorithm.

[hxiaodon]

A simpler algorithm? What's you plan?
I could not think of a way to make the backup more simpler.
My draft idea has something to do with the remote compact version, if the backup side find that the remote volume's compact version is changed compared with the last one written in a local tag file after being successfully backed up, then scanning the whole volume to get checksum is necessary. Or we could just depend on the offset and size for those fids existed in local and remote, offset change means the file is updated

[chrislusf]

Can you write out your algorithm in psuedo code?

[hxiaodon]

@sure, I will provide the psuedo code(later this week), and please help to review～

func (v *Volume) Synchronize(volumeServer string) (err error) {

		//...
		lastRemoteCompactRevision:= getFromLocaVolume()
		//...
Retry:
		masterMap, _, compactRevision, err = fetchVolumeFileEntries(volumeServer, v.Id);
		var isRemoteVolumeCompactedSinceLastFetch bool
		if lastRemoteCompactRevision==compactRevision{
			isRemoteVolumeCompactedSinceLastFetch = false
		}else{
			updatedToLocalVolume(compactRevision)
			lastRemoteCompactRevision = compactRevision
			isRemoteVolumeCompactedSinceLastFetch = true
		}

		var crcMap map[fid]crcValue
		if isRemoteVolumeCompactedSinceLastFetch{
			//Fetch remote volume file's all needle checksum as a map with the latest remote compact version
			var compactVersion int
			crcMap,compactVersion= fetchVolumeFileCRCChecksums(volumeServer, v.Id)
			if compactVersion!=lastRemoteCompactRevision{
			          goto Retry
		}

		if err = v.trySynchronizing(volumeServer, masterMap, compactRevision,isRemoteVolumeCompactedSinceLastFetch,crcMap); err == nil {
			return
		}
	}
	return
}

func (v *Volume) trySynchronizing(volumeServer string, masterMap CompactMap, compactRevision uint16,isRemoteVolumeCompactedSinceLastFetch,crcMap) error {
//...

	//We just modify the masterMap Visit for the needle intersections 

	var localcrcMap map[fid]crcvalue
	if isRemoteVolumeCompactedSinceLastFetch{
		localcrcMap=getLocalCrcMap(v.vid)
	}
	if err := masterMap.Visit(func(needleValue NeedleValue) error {
		if needleValue.Key == 0 {
			return nil
		}
		if value, ok := slaveMap.Get(uint64(needleValue.Key)); ok {
			if isRemoteVolumeCompactedSinceLastFetch{
				//file offset is not changed at remote volume, file is not changed
				if value.Offset==needleValue.Offset{
					return nil
				}
			}else{
				if value.Size==needleValue.Size && localcrcMap[needleValue.Key]==crcMap[needleValue.Key]{
					return nil
				}
			}
		}
		delta = append(delta, needleValue)
		return nil
	}); err != nil {
		return fmt.Errorf("Add master entry: %v", err)
	}


//...

}

Above is the psuedo code, hope for your suggestions~

[hxiaodon]

fetchVolumeFileCRCChecksums function will get a map typed data with needle id as key and needle crc checksum as value. To reduce the network transferring cost, we could use protobuf's map feature and wrap this map and the compactVersion field into a struct which will be marshal into the http response body.

[chrislusf]

I think this crcMap would work mostly, but not sure whether it is water tight in edge cases.

I was thinking just treating the remote .dat file as a change log and just copy every bits of it.