Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Determine ABI stability timeline for C++20 #1814

Closed
StephanTLavavej opened this issue Apr 9, 2021 · 53 comments · Fixed by #2518
Closed

Determine ABI stability timeline for C++20 #1814

StephanTLavavej opened this issue Apr 9, 2021 · 53 comments · Fixed by #2518
Labels
fixed Something works now, yay! format C++20/23 format high priority Important!

Comments

@StephanTLavavej
Copy link
Member

StephanTLavavej commented Apr 9, 2021

We're in a novel situation: never before have we completed a Standard so rapidly. Adding to this concern, <format> is both ABI-sensitive and performance-sensitive, and WG21 is expected to mandate ABI-breaking changes soon.

(C++17 was different; we finished <charconv> in 2019 having completed all other features much earlier, and while it was performance-sensitive, we had time to extensively test and tune it, and its function-oriented nature means that it has no ABI concerns and can be revised in the future.)

We were initially planning to declare ABI stability when finishing C++20 (along with the addition of /std:c++20 as distinct from the always-experimental /std:c++latest), to align with the user expectation that non-experimental features supported for production use are also ABI stable. However, if this locks us into an implementation that can't keep up with retroactive changes to C++20, or freezes an implementation before it's had a chance for significant performance tuning, that will ultimately be harmful to users.

We should consider asking our bosses and boss-like entities for a change of plans: we will still declare C++20 to be feature-complete and ready for production use when we're done (as indicated by the feature-test macros), but we can announce that its ABI will potentially change for some defined period of time (3 months, 6 months, 1 year?) or number of releases (e.g. Dev17 17.x for some specific value of X). This will have no effect for /std:c++14 and /std:c++17 users, and we can expect that, at least initially, the population of /std:c++20 users will be the most eager upgraders for whom rebuilding the world is the least onerous.

Additionally, to mitigate mix-and-match problems, we could add a #pragma detect_mismatch to <format> right now, allowing us to revise it in the future and have the linker detect any OBJ/static LIB mixing.


Update: We arrived at a plan, see the comment below.

@StephanTLavavej StephanTLavavej added high priority Important! decision needed We need to choose something before working on this format C++20/23 format labels Apr 9, 2021
@sylveon
Copy link
Contributor

sylveon commented Apr 9, 2021

I personally think something like keeping the ABI unstable until the non-preview release of Dev17 would be a good idea.

Freezing it now will lead to major differences between standard libraries because the new changes to the standard library require ABI breakage, meaning some libraries and consumers might be reluctant to adopting C++20 because of inconsistency between compilers/standard library implementations, and this will ultimately be detrimental to the MSVC ecosystem.

@timsong-cpp
Copy link
Contributor

As @brevzin noted, there are three Ranges papers in flight with ABI implications as well (split, default construction, and join).

@jovibor
Copy link
Contributor

jovibor commented Apr 9, 2021

We should consider asking our bosses and boss-like entities for a change of plans: we will still declare C++20 to be feature-complete and ready for production use when we're done (as indicated by the feature-test macros), but we can announce that its ABI will potentially change for some defined period of time

Nothing but hoping for your bosses' good sanity😎

@hatcat
Copy link
Contributor

hatcat commented Apr 9, 2021

We're in a novel situation: never before have we completed a Standard so rapidly.

I have high hopes for C++23...

@miscco
Copy link
Contributor

miscco commented Apr 9, 2021

We're in a novel situation: never before have we completed a Standard so rapidly.

I have high hopes for C++23...

Will be finished in 22

@Lectem
Copy link

Lectem commented Apr 9, 2021

If this can help take a decision or convince anybody:
Out of the 5 last clients I worked with, 4 said they stopped using some of STL types due to either runtime performance or compile time performance, that could be fixed by changing ABI such as is done in vNext. I'm thinking mainly about debug iterators and regexp.
If std::format isn't fast enough (to compile or at runtime) compared to fmt::format, I'm afraid this will only keep fueling arguments against using the STL (and std library in general, whatever implementation it is).

While I am sympathetic to having some kind of ABI stability, I don't think "rushing" a release just to be able to put the "c++ latest compliant" stamp is a good idea if QoI improvements can't follow due to the same ABI stability.
An upgrade path must be kept, or some rupture planned to finally take this opportunity to break ABI, just as was done in the past.

@MikeGitb
Copy link

Personally, I'm not even that concerned about performance, exactly because there is fmt and the STL is rarely the best performing solution for a given problem anyway.

What I do care about though is correctness and portability. Effectively locking down the ABI before having any significant user contact, real-world testing and (ideally) a second implementation to compare against seems a very risky choice for something so fundamental - especially when people are still trying to "fix" the standard itself.

@jovibor
Copy link
Contributor

jovibor commented Apr 10, 2021

Personally, I'm not even that concerned about performance, exactly because there is fmt and the STL is rarely the best performing solution for a given problem anyway.

By this logic, there is no value in using STL at all, because every piece of it has more performant counterpart somewhere anyway. The only reason would be that it's available out-of-the box.

If the performance question is not about years of researching or some rocket science, but only delaying ABI locking by month or so I would definitely consider it as the way to go.

@MikeGitb
Copy link

By this logic, there is no value in using STL at all, because every piece of it has more performant counterpart somewhere anyway. The only reason would be that it's available out-of-the box.

That's pretty much the way it is. Yes. Except for

there is no value in using STL at all

Just because there is something faster somewhere doesn't mean the STL doesn't have value. Using 3rd party dependencies comes with it's own costs and even in c++ code, performance is not the only relevant metric.

But that gets off topic.

@vitaut
Copy link
Contributor

vitaut commented Apr 12, 2021

Personally, I'm not even that concerned about performance, exactly because there is fmt and the STL is rarely the best performing solution for a given problem anyway.

It's more complicated in case of fmt/std::format compared to, say, containers because here we have the formatter extension point. It would be really unfortunate if we ended up in the world with two sets of incompatible formatter specializations and resulting interoperability problems. I think it's worth taking time and addressing performance differences before ABI is frozen or at least making sure that they can be addressed in the future without ABI breakage.

@MikeGitb
Copy link

It's more complicated in case of fmt/std::format compared to, say, containers because here we have the formatter extension point. It would be really unfortunate if we ended up in the world with two sets of incompatible formatter specializations and resulting interoperability problems.

So far we have rarely been using a library provided operator<< for printing (except for printf debugging or quick prototyping), because most of the time we had our own format requirements anyway. In a similar fashoin, I'm not sure how much I'm going to use a 3rd party fmt/std::formatter specialization in a context, where performance matters. Also, performance aside, I expect fmt to stay ahead of std::format feature and compatibility wise for the forseable future, so we'll probably have to live with this duality for quite some time anyway.

Anyway, as I wrote previously: I'm also very much against locking down the ABI right away - just that performance isn't my primary reason for that.

@vitaut
Copy link
Contributor

vitaut commented Apr 13, 2021

far we have rarely been using a library provided operator<< for printing

You maybe, others have been using them extensively.

we'll probably have to live with this duality for quite some time anyway.

The important thing is the long term, not the next few years. In the long term I expect fmt::formatter to go away.

@MikeGitb
Copy link

MikeGitb commented Apr 13, 2021

You maybe, others have been using them extensively.

Not sure what you are trying to convince me of. I never said, 3rd party overloads of operator<< isn't bein used by others. I said we didn't use it. I also didn't say whatever performance such an ABI freeze would leave on the table isn't a problem for anyone. I said I am not concerned about the performance impact [EDIT: primarily].

The important thing is the long term, not the next few years.

Long term, the current msvc ABI doesn't matter anyway (assuming plans to introduce an new ABI in the Dev17 timeframe come to fruitition). Also, I'd speculate that implementation divergence or unfixable bugs that slipped through will be a much bigger obstacle to the adoption of std::formatter than x% performance difference.

Anyway. I think we agree that we'd prefer ABI not to be frozen right away. Arguing which reason (potential performance difference vs potential divergence/correctness issues or even the danger of preventing improvements to the standard) is "more important" feels silly.

@barcharcraz
Copy link
Member

I think std::format is a little less hamstrung by ABI than regex, but these sorts of things are extremely difficult to analyze correctly.

@TautvydasZilys
Copy link

TautvydasZilys commented Apr 19, 2021

This will have no effect for /std:c++14 and /std:c++17 users, and we can expect that, at least initially, the population of /std:c++20 users will be the most eager upgraders for whom rebuilding the world is the least onerous.

Does this mean that you will not be able to link /std:c++14 code with /std:c++20 code once it is properly released? Or is this about mixing /std:c++20 code compiled with VS 17.0-preview4 with /std:c++20 code compiled with VS 17.6?

@sylveon
Copy link
Contributor

sylveon commented Apr 19, 2021

The latter

@TheStormN
Copy link
Contributor

I'm concerned about the VS 2022 blog post saying: "If you want to upgrade to Visual Studio 2022 but are worried about compatibility, binary compatibility with the C++ runtime will make it painless."

Which basically means that they are not going to break the ABI of the current runtime for C++11/14/17, which means that some old bugs and performance issues aren't going to be addressed. Not sure if different runtime will be used for the C++20 mode(where older standard features are used), but if it is, then things are getting really confusing.

@StephanTLavavej Could you shed some light on these matters?

@sylveon
Copy link
Contributor

sylveon commented Apr 25, 2021

The ABI was not broken for VS 2022, as documented here: #1725 (reply in thread)

And no, C++20 doesn't use a different runtime, so you can link code compiled for C++20 to code compiled for older standards without issue.

@TheStormN
Copy link
Contributor

The ABI was not broken for VS 2022, as documented here: #1725 (reply in thread)

And no, C++20 doesn't use a different runtime, so you can link code compiled for C++20 to code compiled for older standards without issue.

I see, so unfortunate... Anyway, to be a bit more on the current topic, I also agree that the C++20 ABI should stay unstable a bit more time in order to properly fix issues from WG21.

@StephanTLavavej
Copy link
Member Author

I've set up a meeting with our bosses and boss-like entities to reach a decision here. Thanks to everyone who's expressed interest in this issue; it is very helpful to be able to point to the number of upvotes 😸

@cor3ntin
Copy link

cor3ntin commented Apr 29, 2021

@StephanTLavavej Great to hear!

FYI, the next plenary is on June 7, the ranges papers should be approved then.
I also would like to point out this issue - having a few more weeks to resolve it would be greatly appreciated LWG3547.

As a matter of personal opinion (not reflective of anyone in WG21), I think giving WG21 until the next plenary to put out the fires and address known defect would be a lovely compromise, I understand that we cannot keep doing that forever.

At the same time, I like to think that it's in the benefit of everyone to address known issues, while we still can.
The long time costs of not doing it would otherwise be paid for many decades by all of C++ developers, or back both WG21 and implementers in an uncomfortable corner.

Mistakes were made, we were blindsided by some issues, and frankly WG21's output also suffers from self-imposed deadlines.
Maybe we should have course corrected sooner, but times are challenging.

I apologize for whatever part I had to play in that lack of reactivity.

We can only try our best. And hope there aren't to many unknown problems that will pop up once C++20 is widely deployed.

Congrats on implementing the standard in record time in the midst of a pandemic,
and thanks for working with your organization on this issue.

Thanks!

@h-vetinari
Copy link

One other aspect that's novel and not mentioned explicitly in the OP, but came up in the reddit thread announcing STL's C++20 completeness (by, IIUC, the same author):

someone: Did open sourcing the library help in any way?

@StephanTLavavej: Yes, beyond our wildest dreams. Our amazing GitHub contributors were responsible for many large and difficult features, stressing both domain expertise and attention to detail. It took some time to migrate the STL to GitHub (an ongoing project) and it takes more time to perform detailed code reviews that communicate how to implement/maintain the STL (something that no book directly teaches; I learned it as an apprentice/minion), but ultimately it's saved a ton of time as our contributors have gone on to write way more (and better!) code than the maintainer team working in isolation ever could have in the old closed system. Without being open source, there's no way we'd have been complete now - probably it'd be calendar year 2022. [...]

It might be helpful for that meeting to point out how the community helped put the STL in a position where the option of declaring feature completeness & ABI stability is even on the table that early. It would be a wonderful way to acknowledge that contribution (on the level of the bosses and upwards) by listening when now many of the same people (not including myself...) are asking for STL's help in turn.

@StephanTLavavej
Copy link
Member Author

@h-vetinari An excellent point, I'll make sure to bring it up!

@vrubleg
Copy link

vrubleg commented May 1, 2021

image

@h-vetinari
Copy link

h-vetinari commented Oct 5, 2021

4 more defect reports approved for C++2021 🙃

@TheStormN
Copy link
Contributor

They all seem to be merged now, so finally we can go for stable full featured C++20(21) mode. :)

Guekka added a commit to Guekka/bethutil that referenced this issue Dec 4, 2021
CaseyCarter added a commit to CaseyCarter/STL that referenced this issue Dec 8, 2021
Our supported C++latest frontends (clang-cl and msvc) both have concepts support, so this appears redundant.

Fixes DevCom-1603100
Guekka added a commit to Guekka/bethutil that referenced this issue Dec 11, 2021
CaseyCarter added a commit to CaseyCarter/STL that referenced this issue Feb 4, 2022
* In `<yvals_core.h>`, don't require `_HAS_CXX23` to define `__cpp_lib_format` and `__cpp_lib_ranges`. Change the feature-test macro test consistently.

* Change test matrices for all tests that touch `<ranges>` and `<format>` to run in 20 mode instead of latest-only.

* Don't use the `span` constructor added by P1989 to C++23 in `P0896R4_views_split`, which now must run in c++20 mode where that constructor is unavailable.

Fixes microsoft#1814.
CaseyCarter added a commit that referenced this issue Feb 7, 2022
* In `<yvals_core.h>`, don't require `_HAS_CXX23` to define `__cpp_lib_format` and `__cpp_lib_ranges`. Change the feature-test macro test consistently.

* Change test matrices for all tests that touch `<ranges>` and `<format>` to run in 20 mode instead of latest-only.

* Don't use the `span` constructor added by P1989 to C++23 in `P0896R4_views_split`, which now must run in c++20 mode where that constructor is unavailable.

Fixes #1814.

* Implement `__msvc_int128.hpp` to complete WG21-P1522 and provide integer-class types `_Signed128` and `_Unsigned128` to be the distance / size types of `iota_view<64-bit integral>`.
@CaseyCarter CaseyCarter added the fixed Something works now, yay! label Feb 7, 2022
@CaseyCarter
Copy link
Member

FYI, I've closed this issue despite that there's still work to be done to backport <ranges> and <format> to VS 2019 16.11, because all of the work that will take place in this repository is now complete.

@SamVanheer
Copy link

Since this issue is closed, is #2492 the issue to keep an eye on for the backporting work?

DWVoid added a commit to NEWorldProject/klsxx that referenced this issue Mar 29, 2022
Change the detection method of vcpkg from toolchain file to env:VCPKG_ROOT
Add the ability to download vcpkg and dependent libraries automatically
Fixed unstable API warning for MSVC (see microsoft/STL#1814)
@GF-Huang
Copy link

GF-Huang commented May 3, 2022

Why I can't use std::format in VS2022? Namespace std has no member format!!!

image

image

@sylveon
Copy link
Contributor

sylveon commented May 3, 2022

std::format under /std:c++20 is available in VS 17.2. For now, 17.2 is in preview. It should release to the stable branch at some point.

@JonSuper
Copy link

Why I can't use std::format in VS2022? Namespace std has no member format!!!

image

image

需要 VS 2019 16.11.14 or VS 2022 17.2 above

@StephanTLavavej
Copy link
Member Author

@JonSuper It should be present in those versions. Are you sure you're looking at the properties for the configuration that you're actually building? I recommend looking at the build log and verifying whether /std:c++20 is on the command line.

You can also inspect the STL's macros to check its version and Standard mode. Include <version> and test _MSVC_STL_UPDATE - it should be exactly 202105L for VS 2019 16.11.14 or any later patch, and it should be 202203L or greater for VS 2022 17.2 or greater. Finally, after including <version>, you can inspect _HAS_CXX20 which will be defined to 1 if the STL believes that it's in C++20 mode (otherwise it will be 0).

Hopefully this should allow you to track down whether you're using an unexpected version or an unexpected compiler option.

@JonSuper
Copy link

JonSuper commented May 24, 2022

@JonSuper It should be present in those versions. Are you sure you're looking at the properties for the configuration that you're actually building? I recommend looking at the build log and verifying whether /std:c++20 is on the command line.

You can also inspect the STL's macros to check its version and Standard mode. Include <version> and test _MSVC_STL_UPDATE - it should be exactly 202105L for VS 2019 16.11.14 or any later patch, and it should be 202203L or greater for VS 2022 17.2 or greater. Finally, after including <version>, you can inspect _HAS_CXX20 which will be defined to 1 if the STL believes that it's in C++20 mode (otherwise it will be 0).

Hopefully this should allow you to track down whether you're using an unexpected version or an unexpected compiler option.

image
@StephanTLavavej
It will compiled fail while using VS 2022 17.2 with Visual Studio 2019 (v142) Toolset
the yvals_core.h here :

#if _HAS_CXX23 && defined(__cpp_lib_concepts) // TRANSITION, GH-395 and GH-1814
#define __cpp_lib_format 201907L
#endif // _HAS_CXX23 && defined(__cpp_lib_concepts)

but using VS 2019 16.11.15 with Visual Studio 2019 (v142) Toolset is OK

@StephanTLavavej
Copy link
Member Author

Ah, then the issue is that the VS 2022 IDE's selectable option of the VS 2019 v142 toolset hasn't been updated to the 16.11.14 or later bits. I believe this will happen at some point but I don't have an ETA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fixed Something works now, yay! format C++20/23 format high priority Important!
Projects
None yet
Development

Successfully merging a pull request may close this issue.