Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Supporting for Windows #47

Closed
lygstate opened this issue Oct 5, 2020 · 11 comments
Closed

Supporting for Windows #47

lygstate opened this issue Oct 5, 2020 · 11 comments
Labels
help wanted Extra attention is needed proposal proposal for new additions or changes

Comments

@lygstate
Copy link

lygstate commented Oct 5, 2020

thought-machine/please#1268
@advancedwebdeveloper
Copy link

CC @asw101 @arschles @devigned

@Tatskaari
Copy link
Contributor

Might want to update this with a bit more information...

Seems NTFS has two ways to set attributes. Extended attributes can be added to the MFT (master file table) however they have limited size. This approach is also supported by FAT I believe. Alternative data streams is only supported by NTFS but allows unlimited data to be stored alongside the main file.

http://ntfs.com/ntfs_basics.htm

I doubt please would run into the size limitation of extended attributes, we only need to store 64 byte.

@kuba-- kuba-- added enhancement New feature or request help wanted Extra attention is needed labels Oct 5, 2020
@advancedwebdeveloper
Copy link

advancedwebdeveloper commented Oct 6, 2020 •
edited

@AaLl86 , please explain about NTFS's capabilities/functionality
@craigbarkhouse , please share your vision
CC @brymat-msft

@advancedwebdeveloper
Copy link

@billziss-gh , could you tell whether your support of file systems (I see that you are dealing with few Golang packages, which has something to do with file systems) allow to improve QA part?

@billziss-gh
Copy link

billziss-gh commented Oct 6, 2020 •
edited

@advancedwebdeveloper

@billziss-gh , could you tell whether your support of file systems (I see that you are dealing with few Golang packages, which has something to do with file systems) allow to improve QA part?

I am not sure I understand what you mean by "improve QA part".

To answer the original question: Windows does have Extended Attribute (EA) support. NTFS and I believe some versions of FAT support them. There is however a caveat: there is no Win32 API to access them and the NT API that exists is not documented.

The NT API consists of the functions NtQueryEaFile and NtSetEaFile and you can find information about them by searching online. I have example implementations of xattr functions using the NT API in the following places:

  • In my port of fsx to Windows, which I use to test file systems.
  • In a simple POSIX layer that is included with one of the WinFsp file systems: winposix.c. Please note that this EA code is not regularly tested.

I also note that Cygwin has a Linux-like implementation of xattr that uses the same NT API. You may want to have a look there as well.

One final note: you will find out that NTFS likes to uppercase extended attribute names (so that myxattr will become MYXATTR upon retrieval). Some POSIX layers deal with this by forcing all extended attribute names to be lowercase.

@lygstate
Copy link
Author

lygstate commented Oct 6, 2020

@advancedwebdeveloper

@billziss-gh , could you tell whether your support of file systems (I see that you are dealing with few Golang packages, which has something to do with file systems) allow to improve QA part?

I am not sure I understand what you mean by "improve QA part".

To answer the original question: Windows does have Extended Attribute (EA) support. NTFS and I believe some versions of FAT support them. There is however a caveat: there is no Win32 API to access them and the NT API that exists is not documented.

The NT API consists of the functions NtQueryEaFile and NtSetEaFile and you can find information about them by searching online. I have example implementations of xattr functions using the NT API in the following places:

  • In my port of fsx to WIndows, which I use to test file systems.
  • In a simple POSIX layer that is included with one of the WinFsp file systems: winposix.c. Please note that this EA code is not regularly tested.

I also note that Cygwin has a Linux-like implementation of xattr that uses the same NT API. You may want to have a look there as well.

One final note: you will find out that NTFS likes to uppercase extended attribute names (so that myxattr will become MYXATTR upon retrieval). Some POSIX layers deal with this by forcing all extended attribute names to be lowercase.

I think uppercase can be resolved by things like base64 or hex encode

@AaLl86
Copy link

AaLl86 commented Oct 7, 2020 via email

@kuba-- kuba-- added proposal proposal for new additions or changes and removed enhancement New feature or request labels Dec 17, 2020
@advancedwebdeveloper
Copy link

So, any progress here?

gaul added a commit to gaul/fake-gcs-server that referenced this issue Feb 12, 2022
@gaul
Store object data natively and metadata in xattr
6c2c0ab 
The latter remains a JSON-encoded blob but is now stored in file
extended attributes.  This reduces memory usage and is much faster by
avoiding JSON-encoding large objects.  This enables a future commit to
avoid reading the entire object, particularly for range requests.
Note that this commit changes the on-disk format and is not compatible
with previous data sets.  Extended attributes have some caveats
including lack of tmpfs and Windows support.  References pkg/xattr#47.
References fsouza#669.  Fixes fsouza#671
@gaul gaul mentioned this issue Feb 12, 2022
Merged
gaul added a commit to gaul/fake-gcs-server that referenced this issue Feb 12, 2022
@gaul
Store object data natively and metadata in xattr
d92525c 
The latter remains a JSON-encoded blob but is now stored in file
extended attributes.  This reduces memory usage and is much faster by
avoiding JSON-encoding large objects.  This enables a future commit to
avoid reading the entire object, particularly for range requests.
Note that this commit changes the on-disk format and is not compatible
with previous data sets.  Extended attributes have some caveats
including lack of tmpfs and Windows support.  References pkg/xattr#47.
References fsouza#669.  Fixes fsouza#671.
@gaul gaul mentioned this issue Feb 12, 2022
Closed
gaul added a commit to gaul/fake-gcs-server that referenced this issue Feb 12, 2022
@gaul
Store object data natively and metadata in xattr
898a9cf 
The latter remains a JSON-encoded blob but is now stored in file
extended attributes.  This reduces memory usage and is much faster by
avoiding JSON-encoding large objects.  This enables a future commit to
avoid reading the entire object, particularly for range requests.
Note that this commit changes the on-disk format and is not compatible
with previous data sets.  Extended attributes have some caveats
including lack of tmpfs and Windows support.  References pkg/xattr#47.
References fsouza#669.  Fixes fsouza#671.
gaul added a commit to gaul/fake-gcs-server that referenced this issue Feb 13, 2022
@gaul
Store object data natively and metadata in xattr
0d13876 
The latter remains a JSON-encoded blob but is now stored in file
extended attributes, except on Windows where it is a separate file.
This reduces memory usage and is much faster by avoiding JSON-encoding
large objects.  This enables a future commit to avoid reading the
entire object, particularly for range requests.  Note that this commit
changes the on-disk format and is not compatible with previous data
sets.  Extended attributes have some caveats including lack of tmpfs
and Windows support.  References pkg/xattr#47.  References fsouza#669.
Fixes fsouza#671.
gaul added a commit to gaul/fake-gcs-server that referenced this issue Feb 13, 2022
@gaul
Store object data natively and metadata in xattr
74ea966 
The latter remains a JSON-encoded blob but is now stored in file
extended attributes, except on Windows where it is a separate file.
This reduces memory usage and is much faster by avoiding JSON-encoding
large objects.  This enables a future commit to avoid reading the
entire object, particularly for range requests.  Note that this commit
changes the on-disk format and is not compatible with previous data
sets.  Extended attributes have some caveats including lack of tmpfs
and Windows support.  References pkg/xattr#47.  References fsouza#669.
Fixes fsouza#671.
gaul added a commit to gaul/fake-gcs-server that referenced this issue Feb 13, 2022
@gaul
Store object data natively and metadata in xattr
cdf2acb 
The latter remains a JSON-encoded blob but is now stored in file
extended attributes, except on Windows where it is a separate file.
This reduces memory usage and is much faster by avoiding JSON-encoding
large objects.  This enables a future commit to avoid reading the
entire object, particularly for range requests.  Note that this commit
changes the on-disk format and is not compatible with previous data
sets.  Extended attributes have some caveats including lack of tmpfs
and Windows support.  References pkg/xattr#47.  References fsouza#669.
Fixes fsouza#671.
gaul added a commit to gaul/fake-gcs-server that referenced this issue Feb 13, 2022
@gaul
Store object data natively and metadata in xattr
e871bc3 
The latter remains a JSON-encoded blob but is now stored in file
extended attributes, except on Windows where it is a separate file.
This reduces memory usage and is much faster by avoiding JSON-encoding
large objects.  This enables a future commit to avoid reading the
entire object, particularly for range requests.  Note that this commit
changes the on-disk format and is not compatible with previous data
sets.  Extended attributes have some caveats including lack of tmpfs
and Windows support.  References pkg/xattr#47.  References fsouza#669.
Fixes fsouza#671.
gaul added a commit to gaul/fake-gcs-server that referenced this issue Feb 13, 2022
@gaul
Store object data natively and metadata in xattr
038aa52 
The latter remains a JSON-encoded blob but is now stored in file
extended attributes, except on Windows where it is a separate file.
This reduces memory usage and is much faster by avoiding JSON-encoding
large objects.  This enables a future commit to avoid reading the
entire object, particularly for range requests.  Note that this commit
changes the on-disk format and is not compatible with previous data
sets.  Extended attributes have some caveats including lack of tmpfs
and Windows support.  References pkg/xattr#47.  References fsouza#669.
Fixes fsouza#671.
gaul added a commit to gaul/fake-gcs-server that referenced this issue Feb 15, 2022
@gaul
Store object data natively and metadata in xattr
02c4ae1 
The latter remains a JSON-encoded blob but is now stored in file
extended attributes, except on Windows where it is a separate file.
This reduces memory usage and is much faster by avoiding JSON-encoding
large objects.  This enables a future commit to avoid reading the
entire object, particularly for range requests.  Note that this commit
changes the on-disk format and is not compatible with previous data
sets.  Extended attributes have some caveats including lack of tmpfs
and Windows support.  References pkg/xattr#47.  References fsouza#669.
Fixes fsouza#671.
gaul added a commit to gaul/fake-gcs-server that referenced this issue Feb 15, 2022
@gaul
Store object data natively and metadata in xattr
8ca54a1 
The latter remains a JSON-encoded blob but is now stored in file
extended attributes, except on Windows where it is a separate file.
This reduces memory usage and is much faster by avoiding JSON-encoding
large objects.  This enables a future commit to avoid reading the
entire object, particularly for range requests.  Note that this commit
changes the on-disk format and is not compatible with previous data
sets.  Extended attributes have some caveats including lack of tmpfs
and Windows support.  References pkg/xattr#47.  References fsouza#669.
Fixes fsouza#671.
gaul added a commit to gaul/fake-gcs-server that referenced this issue Feb 15, 2022
@gaul
Store object data natively and metadata in xattr
f2d545a 
The latter remains a JSON-encoded blob but is now stored in file
extended attributes, except on Windows where it is a separate file.
This reduces memory usage and is much faster by avoiding JSON-encoding
large objects.  This enables a future commit to avoid reading the
entire object, particularly for range requests.  Note that this commit
changes the on-disk format and is not compatible with previous data
sets.  Extended attributes have some caveats including lack of tmpfs
and Windows support.  References pkg/xattr#47.  References fsouza#669.
Fixes fsouza#671.
gaul added a commit to gaul/fake-gcs-server that referenced this issue Feb 15, 2022
@gaul
Store object data natively and metadata in xattr
0e3aff1 
The latter remains a JSON-encoded blob but is now stored in file
extended attributes, except on Windows where it is a separate file.
This reduces memory usage and is much faster by avoiding JSON-encoding
large objects.  This enables a future commit to avoid reading the
entire object, particularly for range requests.  Note that this commit
changes the on-disk format and is not compatible with previous data
sets.  Extended attributes have some caveats including lack of tmpfs
and Windows support.  References pkg/xattr#47.  References fsouza#669.
Fixes fsouza#671.
fsouza pushed a commit to fsouza/fake-gcs-server that referenced this issue Feb 15, 2022
@gaul
Store object data natively and metadata in xattr (#683)
0d523fe 
The latter remains a JSON-encoded blob but is now stored in file
extended attributes, except on Windows where it is a separate file.
This reduces memory usage and is much faster by avoiding JSON-encoding
large objects.  This enables a future commit to avoid reading the
entire object, particularly for range requests.  Note that this commit
changes the on-disk format and is not compatible with previous data
sets.  Extended attributes have some caveats including lack of tmpfs
and Windows support.  References pkg/xattr#47.  References #669.
Fixes #671.
@kuba--
Copy link
Member

kuba-- commented Feb 8, 2024 •
edited

Hey guys (@lygstate @Tatskaari @advancedwebdeveloper @AaLl86),
it's been a while (we have 2024 ;-))
Do you want to keep this issue open? Is anyone (at least) prototyping the feature?

@Tatskaari
Copy link
Contributor

We have no plans to support windows in Please at the moment, so have no need of this feature.

@kuba--
Copy link
Member

kuba-- commented Feb 9, 2024

Ok, I'm closing this one, so far.

@kuba-- kuba-- closed this as not planned Won't fix, can't repro, duplicate, stale Feb 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted Extra attention is needed proposal proposal for new additions or changes
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@lygstate @Tatskaari @kuba-- @advancedwebdeveloper @AaLl86 @billziss-gh