Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move all references to stdfs:: into files.cpp #141

Merged
merged 31 commits into from
Aug 6, 2021

Conversation

BillyONeal
Copy link
Member

@BillyONeal BillyONeal commented Jul 29, 2021

This change makes us treat std::filesystem as if it were a system API, and making paths follow our "internals are UTF-8" conventions. This fixes numerous path encoding bugs where we forgot to say `vcpkg::u8path", and makes it easier to add a completely POSIX backend so that we can run on old Linuxes.

Adds a new type, vcpkg::Path, which is an analogue of std::filesystem::path, except which always is UTF-8 internally. This means we can't represent everything on the file system, but we don't consider creating file system objects with invalid unicode names reasonable behavior.

include/vcpkg/build.h Outdated Show resolved Hide resolved
@BillyONeal BillyONeal force-pushed the depath branch 4 times, most recently from f9e14ca to 7ceea10 Compare July 30, 2021 03:28
include/vcpkg/sourceparagraph.h Outdated Show resolved Hide resolved
src/vcpkg/base/downloads.cpp Outdated Show resolved Hide resolved
src/vcpkg/base/files.cpp Outdated Show resolved Hide resolved
src/vcpkg/base/files.cpp Outdated Show resolved Hide resolved
src/vcpkg/base/files.cpp Show resolved Hide resolved
src/vcpkg/postbuildlint.cpp Show resolved Hide resolved
src/vcpkg/postbuildlint.cpp Show resolved Hide resolved
src/vcpkg/registries.cpp Outdated Show resolved Hide resolved
src/vcpkg/sourceparagraph.cpp Show resolved Hide resolved
src/vcpkg/visualstudio.cpp Show resolved Hide resolved
BillyONeal and others added 18 commits July 30, 2021 13:49
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
# Conflicts:
#	include/vcpkg/base/downloads.h
#	src/vcpkg/base/downloads.cpp
@BillyONeal BillyONeal marked this pull request as ready for review August 5, 2021 19:21
Comment on lines +148 to +171
Ty unaligned_load(const void* pv) noexcept
{
static_assert(std::is_trivial<Ty>::value, "Unaligned loads require trivial types");
Ty tmp;
memcpy(&tmp, pv, sizeof(tmp));
return tmp;
}

bool is_drive_prefix(const char* const first) noexcept
{
// test if first points to a prefix of the form X:
// pre: first points to at least 2 char instances
// pre: Little endian
auto value = unaligned_load<unsigned short>(first);
value &= 0xFFDFu; // transform lowercase drive letters into uppercase ones
value -= (static_cast<unsigned short>(':') << CHAR_BIT) | 'A';
return value < 26;
}

bool has_drive_letter_prefix(const char* const first, const char* const last) noexcept
{
// test if [first, last) has a prefix of the form X:
return last - first >= 2 && is_drive_prefix(first);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems... extremely bit-bashy and overcomplex for vcpkg. Can we do something like:

return ((first[0] & 0xDF) - 'A' < 26) && (first[1] == ':')

the assembler is slightly different, but not like... that much different.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that this is commented as a copy-pasta from microsoft/STL I would prefer to leave it as close to that in form as possible?

Comment on lines 385 to 387
#if defined(__clang__)
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wunused-const-variable"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be using clang_diagnostic macros here; see vcpkg/base/pragmas.h

Suggested change
#if defined(__clang__)
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wunused-const-variable"
VCPKG_CLANG_DIAGNOSTIC(push)
VCPKG_CLANG_DIAGNOSTIC(ignored -Wunused-const-variable)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no VCPKG_CLANG_DIAGNOSTIC although I can add one

#endif // __clang__
constexpr char preferred_separator = VCPKG_PREFERED_SEPARATOR[0];
#if defined(__clang__)
#pragma clang diagnostic pop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

src/vcpkg/commands.civerifyversions.cpp Outdated Show resolved Hide resolved
@BillyONeal BillyONeal merged commit cb1e40d into microsoft:main Aug 6, 2021
@BillyONeal BillyONeal deleted the depath branch August 6, 2021 21:43
@wolfv
Copy link
Contributor

wolfv commented Aug 13, 2021

@BillyONeal
Copy link
Member Author

I think this change resulted in some include problem on our builds for conda-forge: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=363464&view=logs&jobId=58ac6aab-c4bc-5de2-2894-98e408cc8ec9&j=58ac6aab-c4bc-5de2-2894-98e408cc8ec9&t=933f325c-924e-533d-4d95-e93b5843ce8b

Would you have a second to check that?

The Windows build failures there are probably expected; now that we ship binary releases we only worry about building vcpkg itself with current (e.g. 2019) on Windows (which also means we don't have to workaround std::experimental::filesystem's many Many MANY bugs). The MacOS one is more concerning; I'm currently working on making binary releases possible for older versions of that too by ripping out the std::filesystem dependency, but I didn't expect intermediate broken there.

Is that attempting to use "macOS-10.14" Pipelines machines and that's the trigger or is something else going on?

@wolfv
Copy link
Contributor

wolfv commented Aug 13, 2021

Great, I can upgrade the conda-forge config to use VS2019 if that's fixing it.

On conda-forge we're using the 10.9 SDK for macOS, but a recent clang version.
These builds used to work before and if I looked correctly then you were using that symbol before as well, no? Could it be just a missing include?

@BillyONeal
Copy link
Member Author

Missing include is entirely possible

@ras0219-msft
Copy link
Contributor

Yes, this looks exactly like a missing include -- we aren't including <ios> in files that use std::io_errc::stream.

@ras0219
Copy link
Contributor

ras0219 commented Aug 15, 2021

Could you try #158 and confirm if it solves the problem for MacOS?

@BillyONeal
Copy link
Member Author

Could you try #158 and confirm if it solves the problem for MacOS?

Ping @wolfv

@wolfv
Copy link
Contributor

wolfv commented Aug 16, 2021

hey, sorry, I was off on the weekend :)

I applied the patch from #158 and that works on macOS! On Windows, I still have issues after switching to VS2019, although those seem to be in optional: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=364375&view=logs&jobId=a70f640f-cc53-5cd3-6cdc-236a1aa90802&j=a70f640f-cc53-5cd3-6cdc-236a1aa90802&t=f5d15007-a01c-5ad8-c9ce-4d519d3b275f

@BillyONeal
Copy link
Member Author

BillyONeal commented Aug 16, 2021

I was just making sure you were notified/ tagged since you weren't @'d before not being impatient :)

@wolfv
Copy link
Contributor

wolfv commented Aug 16, 2021

Thanks, no worries at all :)

@BillyONeal
Copy link
Member Author

@wolfv
Copy link
Contributor

wolfv commented Aug 16, 2021

you're right. sorry. I just pushed another set of modifications that should properly activate VS 2019. Fingers crossed!

@BillyONeal
Copy link
Member Author

No problem, I broke you after all :)

@wolfv
Copy link
Contributor

wolfv commented Aug 17, 2021

all green now! thx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
7 participants