Usage problems

[jaime-ez]

Hi, I'm using dat cli v12.0.0 and node v4.2.6 on a local ubuntu machine and a remote debian server. I executed:

• On the local machine:

• Then on the remote machine:

• All good. Then, while keeping the dat share command running on the local machine, on another bash tab I executed:

• Then on the remote machine:

Problem...the file is not updated to the latest version.

• Back on the local machine:

Bigger problem, the source folder got reverted to the previous state!

I can't figure out the problem...any help would be much appreciated, thanks!

[pfrazee]

A data revert makes me think this could be be related to #623

[joehand]

Thanks for the great bug report! I was able to get a similar but and feel like it may be related to #623 as @pfrazee says. We'll look into it, this bug is not good!

[pfrazee]

It's not good, but I think we know what's going on. @joehand in beaker I was able to fix it by only prioritizing the most recent version of content blocks, instead of prioritizing the whole content feed. Have yall tried that yet?

[jaime-ez]

Thanks for the quick reply, as you say it's related to #623, if look into hola.txt with nano I get those weird chars you mentioned:

[joehand]

This should be fixed in version 12.0.1. Please let us know if you see this issue again!

[jaime-ez]

Did some tests with version 12.0.1:

• dat sync works like a charm
• dat pull does not update file content on remote to the last version
• Files at the source are ok (no reversions)

Also there are some issues when it comes to looking for connections once the source has been sharing for more than 2 minutes aprox and the remote was offline... when you try to reconnect it takes a loong time (should I make a new issue for this?)

I'll be glad to help...where could I start?

[joehand]

dat pull does not update file content on remote to the last version

Hmm. So if you run dat sync on the download directory it works but dat pull doesn't work?

How did you install dat? One of the bug fixes was in hyperdrive, a dependency so we need to make sure that was updated.

Also there are some issues when it comes to looking for connections once the source has been sharing for more than 2 minutes aprox and the remote was offline... when you try to reconnect it takes a loong time (should I make a new issue for this?)

Ya, can you make a new issue for that one? If you add in the steps to reproduce that'd be great (ie. what you're running on the download side) and how long it takes to reconnect.

[joehand]

I wonder if mafintosh/discovery-swarm#6 is related to the connection issues.

[jaime-ez]

Hmm. So if you run dat sync on the download directory it works but dat pull doesn't work?

Exactly. I updated both on local and remote with npm install -g dat running on both machines node v4.2.6

Then using nvm I changed on both machines to node v6.9.1 and did npm install -g dat again and the issue persists.

Ya, can you make a new issue for that one? If you add in the steps to reproduce that'd be great (ie. what you're running on the download side) and how long it takes to reconnect.

Will do :)

[pfrazee]

You might also try uninstall -g before an install. I have no idea what npm's rules for updating package deps are, but that flow guarantees you get latest on all packages

…

On Tue, Feb 7, 2017 at 5:12 PM, jaime echegaray zipper < ***@***.***> wrote: Hmm. So if you run dat sync on the download directory it works but dat pull doesn't work? Exactly. I updated both on local and remote with npm install -g dat running on both machines node v4.2.6 Then using nvm I changed on both machines to node v6.9.1 and did npm install -g dat again and the issue persists. Ya, can you make a new issue for that one? If you add in the steps to reproduce that'd be great (ie. what you're running on the download side) and how long it takes to reconnect. Will do :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#626 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNhU7lLcBF6ywJRadFQXegIctDxmaH6ks5raPpTgaJpZM4L50GT> .

[joehand]

Ok. A few things to try:

1. uninstall then install as @pfrazee said above (to make sure you have latest deps)
2. Run dat pull --no-exit
3. Run dat pull --upload (if above doesn't work)

There are very few differences between dat sync and dat pull. Those two are the main differences that may be causing this.

• --no-exit should keep the process open to allow it to finish. Perhaps it is exiting before it is actually done.
• --upload will upload data to other peers. That really shouldn't fix it but maybe something very weird is happening?

(can't reproduce this bug anymore so can't test those locally, sorry)

[jaime-ez]

update:

• I made 2 VM on GCE with Debian8, and set a firewall rule to allow connections on port 3282. Used one as source machine the other as download
• then executed:

#!/bin/bash

#set locale
sudo sed -i 's|# es_CL.UTF-8 UTF-8|es_CL.UTF-8 UTF-8|g' /etc/locale.gen
sudo locale-gen

#update
sudo apt-get update && sudo apt-get -y upgrade

#git
sudo apt-get -y install git

#nvm
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash

# exit to load nvm
exit

# after relogin
nvm install v6.9.5

npm install dat -g

• Then I made the same exercise I described at the begining of this issue.
• This time neither dat pull nor dat sync synced the download folder.
• dat pull --no-exit and dat pull --upload made no difference
• dat doctor test passes successfully

The only hint I'm getting is that after running dat pull or dat sync on the download folder I get the Download Finished / Files updated to latest! message respectively and the Total size reported corresponds to the total file size of the modified source file. However, the file at destination remains unchanged...

[joehand]

This time neither dat pull nor dat sync synced the download folder.

Ok, at least it is consistent. Thanks for going through your steps. Just to make sure I understand to try to reproduce, you did:

1. dat share toshare - (with a hello.txt file in the dir)
2. dat clone <link> downloaded - Did this successfully download hello.txt?
3. echo "more text" >> toshare/hello.txt - What do you see in the sharing output here, did it show an updated file?
   4 dat pull downloaded

After the update, if you do a fresh dat clone <link> download-again to a new directory - do you get the updated file?

[jaime-ez]

1. dat share toshare - (with a hello.txt file in the dir)

Yes, the content of hello.txt is hola

2. dat clone downloaded - Did this successfully download hello.txt?

Yes, download successfull

3. echo "more text" >> toshare/hello.txt - What do you see in the sharing output here, did it show an updated file?

On another bash tab I executed echo "world" >> toshare/hola.txt. This is the updated sharing output:

It did show an updated file, the original was 5 B

4. dat pull downloaded

After pulling or cloning again the cat output should have been

hola
world

And as you see it says download finished and a file size of 11 B (which is the file size of the modified file, the original was only 5 B), but in both cases the file at the download folder is not updated.

[joehand]

Sweet, thank you so much for the great reporting and patience!

It seems we can assume this is a content write/replication issue (since the metadata seems to be replicating fine) and that it happens regardless of what happened previously on the download side. We'll let you know when we find something or have another release to test. Thanks!

[pfrazee]

I agree, really generous reporting.

@joehand I'm seeing signs of replication bugs too #635. We may need to do another big replication debug session.

[jaime-ez]

glad to help, I'll keep watching and testing whenever a new version is released..

[joehand]

(getting a bit closer)

We released hyperdrive 7.14.2 which should solve one bug related to this. Still chasing down a second one.

[joehand]

Okay @jaime-ez, I I think we figured this one out. See if you still experience this after reinstalling Dat (there are new hypercore + dat-node modules).

Thanks!

[jaime-ez]

Hi @joehand I did the exact same test and the problem persists :(
Running on node v6.9.5, dat v12.0.2, dat-node v1.3.7

[joehand]

Hi @joehand I did the exact same test and the problem persists :(

Apologies! We had a versioning confusion on hypercore. I will check today with a npm install (was using github copies which were on the right versions).

[jaime-ez]

Hi @joehand, I've done the same tests again and its working!

However, I added a couple more steps and there are still some issues. Same scenario as before: 2 machines, one running dat share on the source folder, the other with a destination folder after running dat clone.

1. When running dat sync at destination folder, if a make a change to the file at destination the file remains unaltered. That is the file at destination is no longer synced with the file at source. Is this how it is supposed to be?

2. If I make a change to the file at destination folder and then run dat pull the file remains unchanged again...

3. If I delete the file at the destination folder and then run dat pull the file is not retrieved:

4. If I go back to the source machine and modify the file at source the behavior described at 1, 2, 3 changes and it works as expected: the file is updated correctly at destination.

Note: all the file changes were done adding or deleting strings ath the end of the file

Cheers

[joehand]

Hi @joehand, I've done the same tests again and its working!

Sweet! Thanks for sticking with us through these replication bugs.

However, I added a couple more steps and there are still some issues. Same scenario as before: 2 machines, one running dat share on the source folder, the other with a destination folder after running dat clone.

Right now, once a file is downloaded dat marks it as downloaded and will never update it again until there is an update at the source. We weren't really sure how to handle this situation but we should definitely provide some way to get back in sync, similar to a git reset HEAD.

We will be changing how metadata is stored soon (in files instead of a database) that will make it easier to address this. In the meantime, you can delete the database (in .dat of the destination dir) and clone again:

rm -rf /home/jamie/folder/.dat
dat clone <link> /home/jamie/folder

This will reset the metadata, index the destination directory, and download anything missing.