-
Notifications
You must be signed in to change notification settings - Fork 29.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Node occasionally gives multiple files/folders the same inode #12115
Comments
I suppose it could be either a Windows bug or quirk1 but it could also be caused by node.js converting libuv's 64 bits I expect it's the latter that is happening here. You should be able to verify that by instrumenting src/node_file.cc if you are so inclined. Grep for 'st_ino', it's used in only one location. 1 The inode number is derived from |
I had a little play with that Windows API and ran it directly against files that node reported with the same ino (Using C# DLL import) Example
Looks like you were right about the problem being in the double conversion. Here's a C++ snippet highlighting it. #include <iostream>
int main() {
long long a = 9851624185071827;
long long b = 9851624185071829;
std::cout.precision(100);
std::cout << a << std::endl; //9851624185071827
double da = a;
std::cout << da << std::endl; //9851624185071828
std::cout << b << std::endl; //9851624185071829
double db = b;
std::cout << db << std::endl; //9851624185071828
return 0;
} Closest possible fix I could think of is to change the |
Yes, that wouldn't be a solution. JS doesn't have a 64 bits integral type, only a 64 bits floating-point type. Values >= 253 and <= -253 cannot be represented exactly and get rounded up or down. We could detect such out-of-range values and present them as strings instead of numbers but it's an open question if a type change like that qualifies as a bug fix or a backwards-incompatible semver-major change. cc @mscdex since you worked on that code recently. |
There really is no good solution for this until JS gets 64-bit types. IIRC this isn't the only place where node hopes for the best by using a double for 64-bit integer values. |
Best thing to do might be make a native module that copies the core part that you need but returns a string? |
Alternatively we might think about having a second |
How would that work (having two different constructors at the same time)? Does someone opt-in to the new one with a flag passed to the
Define 'faithfully.' Is this an array with the upper and lower 32-bits? A string? An 8-byte Buffer? A new
I am all for getting rid of the |
Yes, a flag (not a fan) or a new set of functions on
Something like that. It’s not actually important to me which of your suggestions – they all make sense – but generally something that people could reimplement the current |
I still don't understand this, how they are changing/providing |
@mscdex Sorry, probably not expressing myself clearly. Basically, what I’d imagine is a 2nd API on Node’s side, not changing the current API’s behaviour. If somebody wanted, they could build the current API on top of that as a userland module – I’m not suggesting that they monkey-patch anything (or “change” |
Personally, I would prefer if Node fixed this properly going forward. I think passing the inode as a String would be appropriate since an inode is in essence an identifier. It happens to look like an integer, but it may as well be hex for all intents and purposes. I'm not sure if passing the inode as a String would have any performance implications? AFAIK it's a minimum of 24 bytes allocation compared to an integer? I wouldn't mind a Buffer either if that's lighter. Referring to @mscdex's comment, it would be terrific if we could do away with instantiating thousands of Date objects, when users really just need the timestamp. |
Just split the 64-bit word into a high and low 32-bit word and leave arithmetic to the user. |
Just another idea, the 64-bit inode integer returned by Windows could be remapped through a hash into the 53-bit address space supported by JS integers. This should give less collisions than the current folding. |
Here is a conversation from a few years ago about what to do with Windows inodes. It is interesting to see the same conversation about the issue happening all over again! |
It seems like a combination of the string solution @jorangreef advocates (and @piscisaureus advocated nearly 2 years ago) and the 64 bit high/low fields @kkoopa suggests is the way to go, to me. Or, rather, I don't see why we need to choose between the two. A string will allow Node to provide a consistent cross-platform representation at the cost of performance during comparisons. Switching an integer to a string would be a breaking change, though, so you'd need to account for that. The 64 high/low fields would allow you to do performant inode comparisons if you need them. |
I still find it odd no big-integer library is part of standard Node.js. It would have solved problems like this until JS got proper support for 64-bit integers. |
I suppose we could have done that like with
I agree with this, we could just return a The questions then become:
|
Actually, on second look we'd be adding a property to |
Why is there even discussion about big or small numbers? inode is a number only by accident (because it was an index into some internal array on early UNIX), in reality and semantically it's just an arbitrary identifier. Arithmetic on it is useless, so make it a string, object, whatever, it doesn't matter, just make sure it can be converted back and forth into the OS representation internally. |
FWIW comparing hex encoded strings is actually
Strings also make better keys than arrays. 😉 |
@deckar01 your benchmark is a bit flawed as it uses the |
Typed arrays are mutable though, so string is a better choice for immutable identifiers. (This is a kind of a design decision that should be considered and discussed before doing benchmarks.) Strings are also transparently encoded and decoded as JSON, unlike typed arrays, so programs that store inodes in JSON-encoded structures will continue doing so without issues. Finally, strings can be compared with |
I, personally, am totally fine with a string if that works well. |
given the lack of anything else more suitable, I'm good with string also |
I'd just have gone with low/high 32bit or 8Byte array. Btw just to mention, since we are at using strings anyway as it seems: ReFS uses 128Bit identifiers. The 64bit part is not unique there:
It seems that NtQueryInformationFile FILE_INTERNAL_INFORMATION structure and GetFileInformationByHandle(Ex) deliver the FileIDs, except the later one supports returning the 128Bit one in FILE_ID_INFO structure FILE_INTERNAL_INFORMATION |
I would venture a guess that this is because the |
@decal it's probably more because of the large number of config files, caches, and whatnot stored in there for the whole zoo of Windows applications. Create a lot of files in the same directory and some of their inodes are bound to be close. |
Off-topic comment deleted. Next offender gets a ban. (Added a note to the OP -@Fishrock123) |
conversions of 64-bit integers to a double (Number in Javascript) is necessarily lossy. Of course a double is represented by a 64-bit field, but it is then normalized for some values of the exponent part:
So basic unsigned 64-bit integer values represented in hex by
The best you can get is not preserving the full 64 bits, but only the sign bit, then force the next bit to 0 (avoids zeros/denormals/Nans,infinites where this bit is necessarily 1) and the 62 lowest bits (note that this transform will not preserve the apparent numeric value of this coerced "double" which will be very different from the value of the 63-bit integer (note also that the 63-bit is also not fully preserved the most negative value is also forbidden): you can only represent a 62-bit signed or unsigned integer this way (to convert that "strange" double to a computable and comparable value, you still need to use a BigInt): the unsigned 64-bit 0xFFFF_FFFF_FFFF_FFFF (-1 if signed) is not stored exactly, you loose 2 bits, you can only store 0x9FFF_FFFF_FFFF_FFFF (which would also be a negative double). With such 62-bit field, you can't perform any arithmetic (all you can do is to compare them, but even a double addition or substraction or multiplication or division, or just a negation, will not give you the correct result in this representation). So your inodes are necessarily limited to 62 bit: if the two highest bits of the 64-bit integer inode are significant, they will be lost. So you get the bug where two distinct files on the same volume (not hardlinked to the same store, so with possibly different contents and different security attributes) will apparently have the same inode in Javascript (the bug can be reproduced only on very large volumes which can hold many inodes but does not affect mosts users using harddisks or small RAID arrays: it occurs on very large cloud storages where there's a single virtual volume; it does not affect clients of these clouds that have partitioned their space in multiple virtual devices, but affects the very large volumes needed for very database storage of several petabytes). The only safe work around for now is to use a second Number to represent a 64-bit inode... or a BigInt. If inodes are represented as Javascript numbers with identical values, then max inodes cannot be more than MAX_SAFE_INTEGER=(2**53-1) per volume, i.e. you cannot have more than ~9 billions millions inodes per volume (i.e. per mountable filesystem device), and Javascript should be safe: this is already a very considerable space that only a few very large cloud providers like Google can offer in their big farms to their business customers (but something that will be useful for example to create a filesystem-based hashstore for RDF, with lot of extremely small files, typically about 1KB each, i.e. a filesystem total size slightly above 10^16 KB when counting the extra filesytem overhead for managing its own filename indexes, transaction logs and security attributes; with typical harddisks of 1TB each, such giant volume would require more than 100,000 harddisks, not counting redundant disks for RAID 5/6, but for Google that manages millions of customers and allows allocating several Gigabytes for each, such quantity is not demesurated, but Google would normally partition this space with one virtual volume per customer)! Even for Youtube, all videos don't need to be in the same volume and have to be in the storage space of one client So I have serious doubt that the 53 bit limit of inodes seen in Javascript is a real problem, except if you use javascript as a virtual connector to millions remote volumes, seen in Javascript as if it was only one: in that case, it's jsut best to split this huge space by creating multiple virtual volumes. The bug may be visible however if inodes returned to applications are not linear but are actually weak hashes (such as MD5) of effective remote inodes. Or if inodes are used not just to locate a file in the volume, but also to track other information such as a transaction id or version number, like in ReFS (not all nodes in a volume may be accessible simultaneously or with the same security context), in which case you need more bits to store that info (which may be encrypted along with the location in the inode exposed to clients, that must then use inode values exactly, without any loss). An inode number may even be an opaque 128-bit UUID (created randomly and associated to real inodes on servers, or with a strong hash like SHA), where most inode numbers on the same volume are invalid (inaccessible to the client) and the rest is distributed completely randomly over the 128-bit space: truncating this to 53 bit would necessarily reduce the usable space on the volume to just a tiny fraction of the volume before collisions in 53-bit values start occuring for very distinct 128-bit values. My opinion is that even Unix/Linux/POSIX will reform the datatype used for "inode numbers" to allow much longer pseudo-integer types (inode_t), and that it may as well be a long structure containing multiple words whose actual storage length/precision will depend on each device/volume, even if 64-bit will remain largely enough for any physical storage volumes like harddisks or RAID arrays or most clouds. So the BigInt representation in Javascript will still be a solution, even if Javascript is extended with native 64-bit integer support. |
@verdy-p Ref: #23821 |
This is identical to the solution of converting inode numbers to strings: strings are already hashed; but strings at least can detect collisions and ensure uniqueness. |
Unfortunately, BigInts are easily mutable, and not suitable as atomic identifiers. And they suffer from serilization locking problem (performance cost) when all we need is an atomic store, which is what strings offer natively (Javascript strings do not have to be displayable or even valid UTF-16, they are just immutable vectors of unsigned 16-bit code units; if we want to display these identifiers, we can still reencode them easily into other valid UTF-16 strings). Strings are also faster to compare (at least for equality: this is almost immediate by comparing only the references, if they are stored in a hashtable). But you may argue that we'll suffer from the storage cost of the hastable to make them atomic if we want to manage large collections of inodes: they would slow down all the rest of the API using strings. In terms of memory management and garbage collection, their cost will be equivalent to BigInts which also need to be allocated (the hashing of inode strings is not absolutely necessary as these strings are short enough to be compared directly, without comparing first their hash then looking up their actual location in the hashtable). |
BigInts are primitives; how are they mutable? |
Adding a string inode field will require some non trivial changes to our marshaling logic. ATM we pass the data as using a globaly mapped array of either double or BigInt: Lines 193 to 219 in 9827858
Adding another string value will also have a performance penalty. We could add make this an opt-in new option, something like |
As well some of the returned properties returned by fs.stat() may not be necessary, but costly to get from the underlying OS, and a script may just require a specific set of values, or just one, leaving others possibly unset in the returned Javascript object (also if these properties make no sense for the remote filesystem):
|
So I'm thinking iteratively.
What I'm suggesting is adding an independent
This is a nice server-minor change. It's opt-in so those who don't want it don't pay for it. As for unneeded fields, on POSIX we use stat syscall, so we get everything at the same price. |
Note that you speak about POSIX, not actual systems. Even if most OSes implement more or less the POSIX API, filling all the fields may be costly, and not enough to get file info for all filesystems. In the web context we can speak about virtual filesystems with web APIs that need security enforcement, and where getting all the infos at once may be costly or simply denied: under POSIX rules they would be set to pseudovalues like you do in your code by setting arbitrarily some fields arbitrarily to 0. |
Why is there's a theoretical discussion about future and past file systems, and request for |
Moderator's note (@Fishrock123): Off-topic comments have and will be deleted.
Node sometimes reports different files/folders to have identical ino values.
I can't reproduce this consistently and copying/reproducing a folder structure that contains dupes elsewhere doesn't replicate the issue.
I did encounter lots of duplicates under
C:\Users\%USER%\AppData
but it may be different for other peopleExample
Specific example I encountered
Test Script
Here's a hacky node script to loop through a directory and look for duplicate inodes.
Running it on most of my other folders didn't yield a result, until I ran it on
C:\Users\%USER%\AppData
where I encounted loads of duplicatesUsage:
node dupe.js [dir]
The text was updated successfully, but these errors were encountered: