Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OneDrive Windows mount: changing file names renders them in capital letters and duplicates #6682

Closed
ferferga opened this issue Jan 10, 2023 · 16 comments

Comments

@ferferga
Copy link
Contributor

ferferga commented Jan 10, 2023

The associated forum post URL from https://forum.rclone.org

https://forum.rclone.org/t/onedrive-windows-mount-changing-file-names-renders-them-in-capital-letters-and-duplicates/35179

What is the problem you are having with rclone?

Changing file names renders them in capital letters and duplicates

What is your rclone version (output from rclone version)

rclone v1.61.1
- os/version: Microsoft Windows 11 Pro 22H2 (64 bit)
- os/kernel: 10.0.22621.963 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: cmount

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Windows 11 Pro 22H2 64 bits 22621.963

Which cloud storage system are you using? (e.g. Google Drive)

OneDrive

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

Read the forum post associated for a better explanation of the issue

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

Read the forum post associated

Additional details

Although pointed in the forum post, I'll leave this commit found to contain the bug by @olefrost for ease of access: a947f29

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.
@ncw
Copy link
Member

ncw commented Jan 10, 2023

Could you write a short description of how to replicate the problem please?

@olefrost
Copy link
Contributor

olefrost commented Jan 10, 2023

Although pointed in the forum post, I'll leave this commit found to contain the bug by @olefrost for ease of access: a947f29

Just to clarify, the bug is present when building from this commit. I don't know (yet) if this is the commit causing the bug, or it is caused by an earlier commit.

Could you write a short description of how to replicate the problem please?

The simple replication is to make a simple mount something like this on Windows:

rclone mount ./testfolder W: --vfs-cache-mode full

and then open the Explorer in W: and click New/Folder - then you will see the new folder having the name "NEW FOLDER". Press F5 to refresh and you will see "New folder". The seemingly duplicate files are more difficult to reproduce, I don't have a bulletproof sequence yet, and guess the issues are related.

@ncw
Copy link
Member

ncw commented Jan 10, 2023

Is it possible to reproduce this issue just with CMD?

@olefrost
Copy link
Contributor

Tried, but not as simple as mkdir W:Test followed by dir W:

Perhaps you can recreate the sequence by replicating the actions performed to "Nueva carpeta"/"NUEVA CARPETA" in this snippet from the initial debug log:

"Nueva carpeta" means "New Folder".

2022/12/31 19:06:47 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:47 DEBUG : /: >Getattr: errc=0
2022/12/31 19:06:47 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:47 DEBUG : /: >Getattr: errc=0
2022/12/31 19:06:47 DEBUG : /: Opendir:
2022/12/31 19:06:47 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/12/31 19:06:47 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/12/31 19:06:47 DEBUG : /: >Opendir: errc=0, fh=0x6
2022/12/31 19:06:47 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:47 DEBUG : /: >Getattr: errc=0
2022/12/31 19:06:47 DEBUG : /: Readdir: ofst=0, fh=0x6
2022/12/31 19:06:47 DEBUG : /: >Readdir: items=29, errc=0
2022/12/31 19:06:47 DEBUG : /: Releasedir: fh=0x6
2022/12/31 19:06:47 DEBUG : /: >Releasedir: errc=0
2022/12/31 19:06:47 DEBUG : /test/Nueva carpeta: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:47 DEBUG : /test/Nueva carpeta: >Getattr: errc=-2
2022/12/31 19:06:47 DEBUG : /test: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:47 DEBUG : /test: >Getattr: errc=0
2022/12/31 19:06:47 DEBUG : /test/Nueva carpeta: Mkdir: mode=0770
2022/12/31 19:06:48 DEBUG : test: Added virtual directory entry vAddDir: "Nueva carpeta"
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: >Mkdir: errc=0
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: Opendir:
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: >OpenFile: fd=test/Nueva carpeta/ (r), err=<nil>
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: >Opendir: errc=0, fh=0x6
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: >Getattr: errc=0
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: Releasedir: fh=0x6
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: >Releasedir: errc=0
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: >Getattr: errc=0
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: >Getattr: errc=0
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: Opendir:
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: >OpenFile: fd=test/Nueva carpeta/ (r), err=<nil>
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: >Opendir: errc=0, fh=0x6
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: Releasedir: fh=0x6
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta: >Releasedir: errc=0
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta/desktop.ini: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:48 DEBUG : /test/Nueva carpeta/desktop.ini: >Getattr: errc=-2
2022/12/31 19:06:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:48 DEBUG : /: >Getattr: errc=0
2022/12/31 19:06:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:48 DEBUG : /: >Getattr: errc=0
2022/12/31 19:06:48 DEBUG : /: Opendir:
2022/12/31 19:06:48 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/12/31 19:06:48 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/12/31 19:06:48 DEBUG : /: >Opendir: errc=0, fh=0x6
2022/12/31 19:06:48 DEBUG : /: Releasedir: fh=0x6
2022/12/31 19:06:48 DEBUG : /: >Releasedir: errc=0
2022/12/31 19:06:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:48 DEBUG : /: >Getattr: errc=0
2022/12/31 19:06:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:48 DEBUG : /: >Getattr: errc=0
2022/12/31 19:06:48 DEBUG : /: Opendir:
2022/12/31 19:06:48 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/12/31 19:06:48 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/12/31 19:06:48 DEBUG : /: >Opendir: errc=0, fh=0x6
2022/12/31 19:06:48 DEBUG : /: Releasedir: fh=0x6
2022/12/31 19:06:48 DEBUG : /: >Releasedir: errc=0
2022/12/31 19:06:48 DEBUG : /test: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:48 DEBUG : /test: >Getattr: errc=0
2022/12/31 19:06:48 DEBUG : /test: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:48 DEBUG : /test: >Getattr: errc=0
2022/12/31 19:06:48 DEBUG : /test: Opendir:
2022/12/31 19:06:48 DEBUG : /test: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/12/31 19:06:48 DEBUG : /test: >OpenFile: fd=test/ (r), err=<nil>
2022/12/31 19:06:48 DEBUG : /test: >Opendir: errc=0, fh=0x6
2022/12/31 19:06:48 DEBUG : /test: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:48 DEBUG : /test: >Getattr: errc=0
2022/12/31 19:06:48 DEBUG : /test/NUEVA CARPETA: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:48 DEBUG : /test/NUEVA CARPETA: >Getattr: errc=0
2022/12/31 19:06:48 DEBUG : /test: Releasedir: fh=0x6
2022/12/31 19:06:48 DEBUG : /test: >Releasedir: errc=0
2022/12/31 19:06:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:48 DEBUG : /: >Getattr: errc=0
2022/12/31 19:06:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:48 DEBUG : /: >Getattr: errc=0
2022/12/31 19:06:48 DEBUG : /: Opendir:
2022/12/31 19:06:48 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/12/31 19:06:48 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/12/31 19:06:48 DEBUG : /: >Opendir: errc=0, fh=0x6
2022/12/31 19:06:48 DEBUG : /: Releasedir: fh=0x6
2022/12/31 19:06:48 DEBUG : /: >Releasedir: errc=0
2022/12/31 19:06:49 DEBUG : /TEST/NUEVA CARPETA: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:49 DEBUG : /TEST/NUEVA CARPETA: >Getattr: errc=0
2022/12/31 19:06:49 DEBUG : /TEST/NUEVA CARPETA: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/31 19:06:49 DEBUG : /TEST/NUEVA CARPETA: >Getattr: errc=0
2022/12/31 19:06:49 DEBUG : OneDrive root '': Checking for changes on remote
2022/12/31 19:06:49 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Imágenes/Fondos de Escritorio/1382492466739.jpg not removed, freed 0 bytes
2022/12/31 19:06:49 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Imágenes/Fondos de Escritorio/@iWallpaperPC (58).jpeg not removed, freed 0 bytes
2022/12/31 19:06:49 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Imágenes/Fondos de Escritorio/Warm Poly.png not removed, freed 0 bytes
2022/12/31 19:06:49 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Imágenes/Fondos de Escritorio/@iWallpaperPC (28).jpeg not removed, freed 0 bytes
2022/12/31 19:06:49 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Imágenes/Fondos de Escritorio/@iWallpaperPC (54).jpeg not removed, freed 0 bytes
2022/12/31 19:06:49 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Imágenes/Fondos de Escritorio/@iwallpaperPC (4).jpeg not removed, freed 0 bytes
2022/12/31 19:06:49 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Imágenes/Fondos de Escritorio/Cosmos05.jpg not removed, freed 0 bytes
2022/12/31 19:06:49 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Imágenes/Fondos de Escritorio/rolling-waves.jpg not removed, freed 0 bytes
2022/12/31 19:06:49 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item test/sample.pdf not removed, freed 0 bytes
2022/12/31 19:06:49 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item test/sample_hello.pdf not removed, freed 0 bytes
2022/12/31 19:06:49 INFO  : vfs cache: cleaned: objects 10 (was 10) in use 0, to upload 0, uploading 0, total size 12.689Mi (was 12.689Mi)
2022/12/31 19:06:50 DEBUG : : changeNotify: relativePath="root", type=0
2022/12/31 19:06:50 DEBUG : : invalidating directory cache
2022/12/31 19:06:50 DEBUG : : >changeNotify:
2022/12/31 19:06:50 DEBUG : : changeNotify: relativePath="test", type=0
2022/12/31 19:06:50 DEBUG : test: invalidating directory cache
2022/12/31 19:06:50 DEBUG : : >changeNotify:
2022/12/31 19:06:50 DEBUG : : changeNotify: relativePath="test/sample.pdf", type=1
2022/12/31 19:06:50 DEBUG : : >changeNotify:
2022/12/31 19:06:50 DEBUG : : changeNotify: relativePath="test/Nueva carpeta", type=0
2022/12/31 19:06:50 DEBUG : test/Nueva carpeta: invalidating directory cache
2022/12/31 19:06:50 DEBUG : : >changeNotify:

@olefrost
Copy link
Contributor

olefrost commented Jan 10, 2023

I have with the help of available betas narrowed the suspected changes down to the commits in this screenshot with some
good news below it:

image

There were unfortunately no betas to separate between my 2 primary suspects (marked in orange): a947f29 and 1b0128e and a947f29 is the last I can build without a lot of hassle.

Then I got the idea of reversing 1b0128e on top of a947f29 by changing go.mod to
github.com/winfsp/cgofuse v1.5.0
and that luckily builds (in my amd64 dev env) and doesn't seem to have the issue 😅

So the issue seems to come from the winfsp/cgofuse update from
github.com/winfsp/cgofuse v1.5.0
to
github.com/winfsp/cgofuse v1.5.1-0.20220421173602-ce7e5a65cac7
in 1b0128e

@ncw
Copy link
Member

ncw commented Jan 10, 2023

Great sleuthing :-)

Perhaps you can recreate the sequence by replicating the actions performed to "Nueva carpeta"/"NUEVA CARPETA" in this snippet from the initial debug log:

I don't see anything unusual in that log - I looked through it very carefully.


I reproduced this on my Windows VM. It did not reproduce with WinFSP 1.10.22006 however when I installed the latest version 1.12.22301 it does reproduce the problem:

>rclone version
rclone v1.62.0-DEV
- os/version: Microsoft Windows 10 Pro N 21H2 (64 bit)
- os/kernel: 10.0.19044.2251 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.19.1
- go/linking: static
- go/tags: cmount

I created a mount of a local directory

rclone mount c:\testfolder Q: --vfs-cache-mode full -vv

I then tried to create a new folder in explorer in Q: using the Explorer menu. This works but explorer shows it in UPPER CASE:

image

However we can see that it actually created it in lower case:

C:\Users\Dev>dir c:\testfolder
 Volume in drive C has no label.
 Volume Serial Number is 464C-ABA7

 Directory of c:\testfolder

10/01/2023  17:19    <DIR>          .
10/01/2023  17:19    <DIR>          ..
10/01/2023  17:07    <DIR>          dir
10/01/2023  17:08    <DIR>          New folder
10/01/2023  17:19    <DIR>          New folder (2)
               0 File(s)              0 bytes
               5 Dir(s)   5,175,824,384 bytes free

And the CMD view agrees.

C:\Users\Dev>dir Q:\
 Volume in drive Q is ? c  testfolder
 Volume Serial Number is AB51-09A8

 Directory of Q:\

10/01/2023  17:08    <DIR>          New folder
10/01/2023  17:19    <DIR>          New folder (2)
10/01/2023  17:07    <DIR>          dir
               0 File(s)              0 bytes
               3 Dir(s)   5,183,832,064 bytes free

I suspect this is because rclone calls host.SetCapCaseInsensitive to tell WinFSP that the underlying file system is case insensitive and that something has happened to that support.

host.SetCapCaseInsensitive(f.Features().CaseInsensitive)

If I comment that call out then Explorer then behaves sensibly.

Maybe rclone shouldn't be doing that - I don't know.

So I guess this is something to do with the case insensitive file handling, and it is likely to do with winfsp/cgofuse@9b9d107 (though I haven't bisected this yet).

@billziss-gh do you have any thoughts on this? Is it a cgofuse or WinFSP bug? Or should rclone just not call host.SetCapCaseInsensitive?

@billziss-gh
Copy link
Contributor

As you know the Windows file system is case-insensitive by default. This means that an app may open a path /Path/To/File with paths such as /PATH/TO/FILE and /path/to/file.

Regardless of the path used to open a file, it is sometimes desirable in Windows to find the "real" path of a file, i.e. the path of the file with the correct case. This is called the "normalized" path in Windows parlance and Windows provides a special file system operation to get it.

This file system operation is rather popular; lots of system components, filters, etc. inquire for the normalized path of a file that is being opened. For this reason this operation is built into the WinFsp driver. The driver follows a simple approach:

  • If the file system is case-sensitive then the normalized path is the same as the open path.
  • If the file system is case-insensitive then it may optionally report to the driver the normalized path of a file that is being opened. If it does so then the driver reports it as the normalized path, otherwise the driver reports the UPPERCASED version of the path as the normalized path. (The reason for UPPERCASING paths is that the normalized path of a file must always be the same regardless of the actual path used to open the file.)

Native WinFsp file systems always supported path normalization, but FUSE did not. I recently added this capability in FUSE and cgofuse via the Getpath operation.

Now it is unclear to me why you should see a difference in behavior between WinFsp 1.10 and 1.12. I may have screwed something up when implementing the Getpath mechanism.

@ncw
Copy link
Member

ncw commented Jan 11, 2023

As you know the Windows file system is case-insensitive by default. This means that an app may open a path /Path/To/File with paths such as /PATH/TO/FILE and /path/to/file.

Regardless of the path used to open a file, it is sometimes desirable in Windows to find the "real" path of a file, i.e. the path of the file with the correct case. This is called the "normalized" path in Windows parlance and Windows provides a special file system operation to get it.

This file system operation is rather popular; lots of system components, filters, etc. inquire for the normalized path of a file that is being opened. For this reason this operation is built into the WinFsp driver. The driver follows a simple approach:

  1. If the file system is case-sensitive then the normalized path is the same as the open path.
  2. If the file system is case-insensitive then it may optionally report to the driver the normalized path of a file that is being opened. If it does so then the driver reports it as the normalized path, otherwise the driver reports the UPPERCASED version of the path as the normalized path. (The reason for UPPERCASING paths is that the normalized path of a file must always be the same regardless of the actual path used to open the file.)

Native WinFsp file systems always supported path normalization, but FUSE did not. I recently added this capability in FUSE and cgofuse via the Getpath operation.

Now it is unclear to me why you should see a difference in behavior between WinFsp 1.10 and 1.12. I may have screwed something up when implementing the Getpath mechanism.

OK, so it looks like that rclone was taking option 1. with WinFSP 1.10 but now is taking option 2. after the FUSE/cgofuse Getpath changes. We didn't add any code to report the normalized path in rclone so I guess the driver is reporting the UPPERCASED path for us.

So it looks like to fix this, rclone should implement Getpath

I had a go with this.

I wasn't sure whether

  • returning / separated paths was OK or not
  • directories should end in / or not
  • I should return the full path or just the leaf

However all variants of that caused Explorer to give this new and exciting error message when I tried to create a directory.

image

I couldn't find any docs for Getpath - it isn't mentioned in https://winfsp.dev/

Any ideas @billziss-gh ?

@ferferga
Copy link
Contributor Author

@ncw Not sure if you work with this piece of info, since I mentioned it in the original forum post but it's not in the TLDR

v1.58.1 is the last stable version without the issue, while v1.59.0 is the first version with the issue, as reported here

@billziss-gh
Copy link
Contributor

@ncw

Assume that the actual path of a file is /Path/To/File. If an application attempts to open /PATH/TO/FILE you will get both an Open and Getpath for /PATH/TO/FILE. From Getpath now return the correct case for the path, i.e. /Path/To/File.

ncw added a commit that referenced this issue Jan 11, 2023
Before this fix, we told cgofuse/WinFSP that the backend was case
insensitive but didn't implement the Getpath backend function to
return the normalised case of a file.

Resently cgofuse started implementing case insensitive files properly
but since we hadn't implemented Getpath, the file names were taking
the default of all in UPPER CASE.

This patch implements Getpath for cgofuse which fixes the case
problems.

This problem came to light when we upgraded cgofuse and WinFSP (to
1.12) which had the code to implement Getpath.

Fixes #6682
@ncw
Copy link
Member

ncw commented Jan 11, 2023

Thanks @billziss-gh - I was missing the leading / off the paths I was returning from Getpath which was what was confusing windows.

It seems to work now and fixes the problem if you want to give it a try @ferferga

v1.62.0-beta.6674.6de91e367.fix-6682-cmount-case-insensitive on branch fix-6682-cmount-case-insensitive (uploaded in 15-30 mins)

@olefrost
Copy link
Contributor

Thanks @ferferga, very helpful. I used your version info to narrow down to commit 1b0128e in v1.59.0, which has then used by Nick to narrow down on the introduction of GetPath in winfsp/cgofuse@9b9d107

@billziss-gh
Copy link
Contributor

@ncw fantastic!

BTW, I have sent you some private emails recently and wondering if you have received them. If you did, no problem :)

@ferferga
Copy link
Contributor Author

@ncw Works great, and seems duplicated files are gone too!

@ncw ncw closed this as completed in 8c6ff1f Jan 11, 2023
@ncw
Copy link
Member

ncw commented Jan 11, 2023

Thank you for testing and for all your help everyone :-)

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.62

If we make a v1.61.2 I'll put it in that, but that isn't a definite!

@billziss-gh I have one email from 7 December I haven't replied to yet were there others? My INBOX is a trashfire as usual (11,000 unread messages). Although the problem is me, I'm sure, rather than my INBOX - I just like to externalise the blame on technology which can't answer back ;-)

@billziss-gh
Copy link
Contributor

@ncw no problem! There is nothing urgent, I just wanted to make sure that you did get them given that email can be an unreliable medium at times.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants