[AArch64] Add option -msve-streaming-vector-bits= . #144611
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is similar to -msve-vector-bits, but for streaming mode: it constrains the legal values of "vscale", allowing optimizations based on that constraint. This also fixes conversions between SVE vectors and fixed-width vectors in streaming functions with -msve-vector-bits and -msve-streaming-vector-bits. This currently doesn't touch the __ARM_FEATURE_SVE_BITS define or the arm_sve_vector_bits attribute.
efriedma-quic
requested review from
MacDue,
sdesmalen-arm,
kmclaughlin-arm and
MDevereau
llvmbot
added
clang
|
@llvm/pr-subscribers-clang @llvm/pr-subscribers-backend-arm Author: Eli Friedman (efriedma-quic) ChangesThis is similar to -msve-vector-bits, but for streaming mode: it constrains the legal values of "vscale", allowing optimizations based on that constraint. This also fixes conversions between SVE vectors and fixed-width vectors in streaming functions with -msve-vector-bits and This currently doesn't touch the __ARM_FEATURE_SVE_BITS define or the arm_sve_vector_bits attribute. Patch is 33.02 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/144611.diff 15 Files Affected:
diff --git a/clang/include/clang/AST/ASTContext.h b/clang/include/clang/AST/ASTContext.h
index 3abb49312255a..64d4c5547341e 100644
--- a/clang/include/clang/AST/ASTContext.h
+++ b/clang/include/clang/AST/ASTContext.h
@@ -2486,15 +2486,6 @@ class ASTContext : public RefCountedBase<ASTContext> {
/// types.
bool areCompatibleVectorTypes(QualType FirstVec, QualType SecondVec);
- /// Return true if the given types are an SVE builtin and a VectorType that
- /// is a fixed-length representation of the SVE builtin for a specific
- /// vector-length.
- bool areCompatibleSveTypes(QualType FirstType, QualType SecondType);
-
- /// Return true if the given vector types are lax-compatible SVE vector types,
- /// false otherwise.
- bool areLaxCompatibleSveTypes(QualType FirstType, QualType SecondType);
-
/// Return true if the given types are an RISC-V vector builtin type and a
/// VectorType that is a fixed-length representation of the RISC-V vector
/// builtin type for a specific vector-length.
diff --git a/clang/include/clang/Basic/LangOptions.def b/clang/include/clang/Basic/LangOptions.def
index 789761c1f3647..8054be1bb4e88 100644
--- a/clang/include/clang/Basic/LangOptions.def
+++ b/clang/include/clang/Basic/LangOptions.def
@@ -503,6 +503,9 @@ LANGOPT(OmitVTableRTTI, 1, 0,
LANGOPT(VScaleMin, 32, 0, "Minimum vscale value")
LANGOPT(VScaleMax, 32, 0, "Maximum vscale value")
+LANGOPT(VScaleStreamingMin, 32, 0, "Minimum streaming vscale value")
+LANGOPT(VScaleStreamingMax, 32, 0, "Maximum streaming vscale value")
+
ENUM_LANGOPT(ExtendIntArgs, ExtendArgsKind, 1, ExtendArgsKind::ExtendTo32,
"Controls how scalar integer arguments are extended in calls "
"to unprototyped and varargs functions")
diff --git a/clang/include/clang/Driver/Options.td b/clang/include/clang/Driver/Options.td
index 152df89118a6a..2e8d5b18483d7 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -5173,6 +5173,14 @@ def msve_vector_bits_EQ : Joined<["-"], "msve-vector-bits=">, Group<m_aarch64_Fe
Visibility<[ClangOption, FlangOption]>,
HelpText<"Specify the size in bits of an SVE vector register. Defaults to the"
" vector length agnostic value of \"scalable\". (AArch64 only)">;
+def msve_streaming_vector_bits_EQ
+ : Joined<["-"], "msve-streaming-vector-bits=">,
+ Group<m_aarch64_Features_Group>,
+ Visibility<[ClangOption, FlangOption]>,
+ HelpText<
+ "Specify the size in bits of an SVE vector register in streaming "
+ "mode. Defaults to the vector length agnostic value of "
+ "\"scalable\". (AArch64 only)">;
} // let Flags = [TargetSpecific]
def mvscale_min_EQ : Joined<["-"], "mvscale-min=">,
@@ -5184,6 +5192,17 @@ def mvscale_max_EQ : Joined<["-"], "mvscale-max=">,
HelpText<"Specify the vscale maximum. Defaults to the"
" vector length agnostic value of \"0\". (AArch64/RISC-V only)">,
MarshallingInfoInt<LangOpts<"VScaleMax">>;
+def mvscale_streaming_min_EQ
+ : Joined<["-"], "mvscale-streaming-min=">,
+ Visibility<[CC1Option, FC1Option]>,
+ HelpText<"Specify the vscale minimum. Defaults to \"1\". (AArch64 only)">,
+ MarshallingInfoInt<LangOpts<"VScaleStreamingMin">>;
+def mvscale_streaming_max_EQ
+ : Joined<["-"], "mvscale-streaming-max=">,
+ Visibility<[CC1Option, FC1Option]>,
+ HelpText<"Specify the vscale maximum. Defaults to the"
+ " vector length agnostic value of \"0\". (AArch64 only)">,
+ MarshallingInfoInt<LangOpts<"VScaleStreamingMax">>;
def msign_return_address_EQ : Joined<["-"], "msign-return-address=">,
Visibility<[ClangOption, CC1Option]>,
diff --git a/clang/include/clang/Sema/SemaARM.h b/clang/include/clang/Sema/SemaARM.h
index ce79e94ebdd9b..788a7abf5f9c1 100644
--- a/clang/include/clang/Sema/SemaARM.h
+++ b/clang/include/clang/Sema/SemaARM.h
@@ -82,6 +82,15 @@ class SemaARM : public SemaBase {
void handleInterruptSaveFPAttr(Decl *D, const ParsedAttr &AL);
void CheckSMEFunctionDefAttributes(const FunctionDecl *FD);
+
+ /// Return true if the given types are an SVE builtin and a VectorType that
+ /// is a fixed-length representation of the SVE builtin for a specific
+ /// vector-length.
+ bool areCompatibleSveTypes(QualType FirstType, QualType SecondType);
+
+ /// Return true if the given vector types are lax-compatible SVE vector types,
+ /// false otherwise.
+ bool areLaxCompatibleSveTypes(QualType FirstType, QualType SecondType);
};
SemaARM::ArmStreamingType getArmStreamingFnType(const FunctionDecl *FD);
diff --git a/clang/lib/AST/ASTContext.cpp b/clang/lib/AST/ASTContext.cpp
index 4d44f23c0f503..d785b5a5006cc 100644
--- a/clang/lib/AST/ASTContext.cpp
+++ b/clang/lib/AST/ASTContext.cpp
@@ -10443,87 +10443,6 @@ bool ASTContext::areCompatibleVectorTypes(QualType FirstVec,
return false;
}
-/// getSVETypeSize - Return SVE vector or predicate register size.
-static uint64_t getSVETypeSize(ASTContext &Context, const BuiltinType *Ty) {
- assert(Ty->isSveVLSBuiltinType() && "Invalid SVE Type");
- if (Ty->getKind() == BuiltinType::SveBool ||
- Ty->getKind() == BuiltinType::SveCount)
- return (Context.getLangOpts().VScaleMin * 128) / Context.getCharWidth();
- return Context.getLangOpts().VScaleMin * 128;
-}
-
-bool ASTContext::areCompatibleSveTypes(QualType FirstType,
- QualType SecondType) {
- auto IsValidCast = [this](QualType FirstType, QualType SecondType) {
- if (const auto *BT = FirstType->getAs<BuiltinType>()) {
- if (const auto *VT = SecondType->getAs<VectorType>()) {
- // Predicates have the same representation as uint8 so we also have to
- // check the kind to make these types incompatible.
- if (VT->getVectorKind() == VectorKind::SveFixedLengthPredicate)
- return BT->getKind() == BuiltinType::SveBool;
- else if (VT->getVectorKind() == VectorKind::SveFixedLengthData)
- return VT->getElementType().getCanonicalType() ==
- FirstType->getSveEltType(*this);
- else if (VT->getVectorKind() == VectorKind::Generic)
- return getTypeSize(SecondType) == getSVETypeSize(*this, BT) &&
- hasSameType(VT->getElementType(),
- getBuiltinVectorTypeInfo(BT).ElementType);
- }
- }
- return false;
- };
-
- return IsValidCast(FirstType, SecondType) ||
- IsValidCast(SecondType, FirstType);
-}
-
-bool ASTContext::areLaxCompatibleSveTypes(QualType FirstType,
- QualType SecondType) {
- auto IsLaxCompatible = [this](QualType FirstType, QualType SecondType) {
- const auto *BT = FirstType->getAs<BuiltinType>();
- if (!BT)
- return false;
-
- const auto *VecTy = SecondType->getAs<VectorType>();
- if (VecTy && (VecTy->getVectorKind() == VectorKind::SveFixedLengthData ||
- VecTy->getVectorKind() == VectorKind::Generic)) {
- const LangOptions::LaxVectorConversionKind LVCKind =
- getLangOpts().getLaxVectorConversions();
-
- // Can not convert between sve predicates and sve vectors because of
- // different size.
- if (BT->getKind() == BuiltinType::SveBool &&
- VecTy->getVectorKind() == VectorKind::SveFixedLengthData)
- return false;
-
- // If __ARM_FEATURE_SVE_BITS != N do not allow GNU vector lax conversion.
- // "Whenever __ARM_FEATURE_SVE_BITS==N, GNUT implicitly
- // converts to VLAT and VLAT implicitly converts to GNUT."
- // ACLE Spec Version 00bet6, 3.7.3.2. Behavior common to vectors and
- // predicates.
- if (VecTy->getVectorKind() == VectorKind::Generic &&
- getTypeSize(SecondType) != getSVETypeSize(*this, BT))
- return false;
-
- // If -flax-vector-conversions=all is specified, the types are
- // certainly compatible.
- if (LVCKind == LangOptions::LaxVectorConversionKind::All)
- return true;
-
- // If -flax-vector-conversions=integer is specified, the types are
- // compatible if the elements are integer types.
- if (LVCKind == LangOptions::LaxVectorConversionKind::Integer)
- return VecTy->getElementType().getCanonicalType()->isIntegerType() &&
- FirstType->getSveEltType(*this)->isIntegerType();
- }
-
- return false;
- };
-
- return IsLaxCompatible(FirstType, SecondType) ||
- IsLaxCompatible(SecondType, FirstType);
-}
-
/// getRVVTypeSize - Return RVV vector register size.
static uint64_t getRVVTypeSize(ASTContext &Context, const BuiltinType *Ty) {
assert(Ty->isRVVVLSBuiltinType() && "Invalid RVV Type");
diff --git a/clang/lib/Basic/Targets/AArch64.cpp b/clang/lib/Basic/Targets/AArch64.cpp
index e8abdf9aafd82..c5935390137b1 100644
--- a/clang/lib/Basic/Targets/AArch64.cpp
+++ b/clang/lib/Basic/Targets/AArch64.cpp
@@ -822,10 +822,16 @@ std::optional<std::pair<unsigned, unsigned>>
AArch64TargetInfo::getVScaleRange(const LangOptions &LangOpts,
bool IsArmStreamingFunction,
llvm::StringMap<bool> *FeatureMap) const {
- if (LangOpts.VScaleMin || LangOpts.VScaleMax)
+ if (!IsArmStreamingFunction && (LangOpts.VScaleMin || LangOpts.VScaleMax))
return std::pair<unsigned, unsigned>(
LangOpts.VScaleMin ? LangOpts.VScaleMin : 1, LangOpts.VScaleMax);
+ if (IsArmStreamingFunction &&
+ (LangOpts.VScaleStreamingMin || LangOpts.VScaleStreamingMax))
+ return std::pair<unsigned, unsigned>(
+ LangOpts.VScaleStreamingMin ? LangOpts.VScaleStreamingMin : 1,
+ LangOpts.VScaleStreamingMax);
+
if (hasFeature("sve") || (FeatureMap && (FeatureMap->lookup("sve"))))
return std::pair<unsigned, unsigned>(1, 16);
diff --git a/clang/lib/Driver/ToolChains/Clang.cpp b/clang/lib/Driver/ToolChains/Clang.cpp
index 1d11be1d82be8..24ec25601881f 100644
--- a/clang/lib/Driver/ToolChains/Clang.cpp
+++ b/clang/lib/Driver/ToolChains/Clang.cpp
@@ -1666,7 +1666,7 @@ void Clang::AddAArch64TargetArgs(const ArgList &Args,
}
// Handle -msve_vector_bits=<bits>
- if (Arg *A = Args.getLastArg(options::OPT_msve_vector_bits_EQ)) {
+ auto HandleVectorBits = [&](Arg *A, bool Streaming) {
StringRef Val = A->getValue();
const Driver &D = getToolChain().getDriver();
if (Val == "128" || Val == "256" || Val == "512" || Val == "1024" ||
@@ -1674,22 +1674,35 @@ void Clang::AddAArch64TargetArgs(const ArgList &Args,
Val == "1024+" || Val == "2048+") {
unsigned Bits = 0;
if (!Val.consume_back("+")) {
- bool Invalid = Val.getAsInteger(10, Bits); (void)Invalid;
+ bool Invalid = Val.getAsInteger(10, Bits);
+ (void)Invalid;
assert(!Invalid && "Failed to parse value");
+ StringRef VScaleMax =
+ Streaming ? "-mvscale-streaming-max=" : "-mvscale-max=";
CmdArgs.push_back(
- Args.MakeArgString("-mvscale-max=" + llvm::Twine(Bits / 128)));
+ Args.MakeArgString(VScaleMax + llvm::Twine(Bits / 128)));
}
- bool Invalid = Val.getAsInteger(10, Bits); (void)Invalid;
+ bool Invalid = Val.getAsInteger(10, Bits);
+ (void)Invalid;
assert(!Invalid && "Failed to parse value");
+
+ StringRef VScaleMin =
+ Streaming ? "-mvscale-streaming-min=" : "-mvscale-min=";
CmdArgs.push_back(
- Args.MakeArgString("-mvscale-min=" + llvm::Twine(Bits / 128)));
- // Silently drop requests for vector-length agnostic code as it's implied.
- } else if (Val != "scalable")
+ Args.MakeArgString(VScaleMin + llvm::Twine(Bits / 128)));
+ } else if (Val == "scalable") {
+ // Silently drop requests for vector-length agnostic code as it's implied.
+ } else {
// Handle the unsupported values passed to msve-vector-bits.
D.Diag(diag::err_drv_unsupported_option_argument)
<< A->getSpelling() << Val;
- }
+ }
+ };
+ if (Arg *A = Args.getLastArg(options::OPT_msve_vector_bits_EQ))
+ HandleVectorBits(A, /*Streaming*/ false);
+ if (Arg *A = Args.getLastArg(options::OPT_msve_streaming_vector_bits_EQ))
+ HandleVectorBits(A, /*Streaming*/ true);
AddAAPCSVolatileBitfieldArgs(Args, CmdArgs);
diff --git a/clang/lib/Frontend/CompilerInvocation.cpp b/clang/lib/Frontend/CompilerInvocation.cpp
index 2c02719121c73..b3334b1bc560a 100644
--- a/clang/lib/Frontend/CompilerInvocation.cpp
+++ b/clang/lib/Frontend/CompilerInvocation.cpp
@@ -4558,6 +4558,11 @@ bool CompilerInvocation::ParseLangArgs(LangOptions &Opts, ArgList &Args,
if (StringRef(A->getValue()).getAsInteger(10, VScaleMin) || VScaleMin == 0)
Diags.Report(diag::err_cc1_unbounded_vscale_min);
}
+ if (Arg *A = Args.getLastArg(options::OPT_mvscale_streaming_min_EQ)) {
+ unsigned VScaleMin;
+ if (StringRef(A->getValue()).getAsInteger(10, VScaleMin) || VScaleMin == 0)
+ Diags.Report(diag::err_cc1_unbounded_vscale_min);
+ }
if (const Arg *A = Args.getLastArg(OPT_frandomize_layout_seed_file_EQ)) {
std::ifstream SeedFile(A->getValue(0));
diff --git a/clang/lib/Sema/SemaARM.cpp b/clang/lib/Sema/SemaARM.cpp
index e992a1012fde0..ec892a5d82421 100644
--- a/clang/lib/Sema/SemaARM.cpp
+++ b/clang/lib/Sema/SemaARM.cpp
@@ -1409,4 +1409,101 @@ void SemaARM::CheckSMEFunctionDefAttributes(const FunctionDecl *FD) {
}
}
+/// getSVETypeSize - Return SVE vector or predicate register size.
+static uint64_t getSVETypeSize(ASTContext &Context, const BuiltinType *Ty,
+ bool IsStreaming) {
+ assert(Ty->isSveVLSBuiltinType() && "Invalid SVE Type");
+ uint64_t VScale = IsStreaming ? Context.getLangOpts().VScaleStreamingMin
+ : Context.getLangOpts().VScaleMin;
+ if (Ty->getKind() == BuiltinType::SveBool ||
+ Ty->getKind() == BuiltinType::SveCount)
+ return (VScale * 128) / Context.getCharWidth();
+ return VScale * 128;
+}
+
+bool SemaARM::areCompatibleSveTypes(QualType FirstType, QualType SecondType) {
+ bool IsStreaming = false;
+ if (const FunctionDecl *FD = SemaRef.getCurFunctionDecl(/*AllowLambda=*/true))
+ if (IsArmStreamingFunction(FD, /*IncludeLocallyStreaming=*/true))
+ IsStreaming = true;
+ auto IsValidCast = [&](QualType FirstType, QualType SecondType) {
+ if (const auto *BT = FirstType->getAs<BuiltinType>()) {
+ if (const auto *VT = SecondType->getAs<VectorType>()) {
+ // Predicates have the same representation as uint8 so we also have to
+ // check the kind to make these types incompatible.
+ ASTContext &Context = getASTContext();
+ if (VT->getVectorKind() == VectorKind::SveFixedLengthPredicate)
+ return BT->getKind() == BuiltinType::SveBool;
+ else if (VT->getVectorKind() == VectorKind::SveFixedLengthData)
+ return VT->getElementType().getCanonicalType() ==
+ FirstType->getSveEltType(Context);
+ else if (VT->getVectorKind() == VectorKind::Generic)
+ return Context.getTypeSize(SecondType) ==
+ getSVETypeSize(Context, BT, IsStreaming) &&
+ Context.hasSameType(
+ VT->getElementType(),
+ Context.getBuiltinVectorTypeInfo(BT).ElementType);
+ }
+ }
+ return false;
+ };
+
+ return IsValidCast(FirstType, SecondType) ||
+ IsValidCast(SecondType, FirstType);
+}
+
+bool SemaARM::areLaxCompatibleSveTypes(QualType FirstType,
+ QualType SecondType) {
+ bool IsStreaming = false;
+ if (const FunctionDecl *FD = SemaRef.getCurFunctionDecl(/*AllowLambda=*/true))
+ if (IsArmStreamingFunction(FD, /*IncludeLocallyStreaming=*/true))
+ IsStreaming = true;
+
+ auto IsLaxCompatible = [&](QualType FirstType, QualType SecondType) {
+ const auto *BT = FirstType->getAs<BuiltinType>();
+ if (!BT)
+ return false;
+
+ const auto *VecTy = SecondType->getAs<VectorType>();
+ if (VecTy && (VecTy->getVectorKind() == VectorKind::SveFixedLengthData ||
+ VecTy->getVectorKind() == VectorKind::Generic)) {
+ const LangOptions::LaxVectorConversionKind LVCKind =
+ getLangOpts().getLaxVectorConversions();
+ ASTContext &Context = getASTContext();
+
+ // Can not convert between sve predicates and sve vectors because of
+ // different size.
+ if (BT->getKind() == BuiltinType::SveBool &&
+ VecTy->getVectorKind() == VectorKind::SveFixedLengthData)
+ return false;
+
+ // If __ARM_FEATURE_SVE_BITS != N do not allow GNU vector lax conversion.
+ // "Whenever __ARM_FEATURE_SVE_BITS==N, GNUT implicitly
+ // converts to VLAT and VLAT implicitly converts to GNUT."
+ // ACLE Spec Version 00bet6, 3.7.3.2. Behavior common to vectors and
+ // predicates.
+ if (VecTy->getVectorKind() == VectorKind::Generic &&
+ Context.getTypeSize(SecondType) !=
+ getSVETypeSize(Context, BT, IsStreaming))
+ return false;
+
+ // If -flax-vector-conversions=all is specified, the types are
+ // certainly compatible.
+ if (LVCKind == LangOptions::LaxVectorConversionKind::All)
+ return true;
+
+ // If -flax-vector-conversions=integer is specified, the types are
+ // compatible if the elements are integer types.
+ if (LVCKind == LangOptions::LaxVectorConversionKind::Integer)
+ return VecTy->getElementType().getCanonicalType()->isIntegerType() &&
+ FirstType->getSveEltType(Context)->isIntegerType();
+ }
+
+ return false;
+ };
+
+ return IsLaxCompatible(FirstType, SecondType) ||
+ IsLaxCompatible(SecondType, FirstType);
+}
+
} // namespace clang
diff --git a/clang/lib/Sema/SemaChecking.cpp b/clang/lib/Sema/SemaChecking.cpp
index 8f8e1ceb7197e..bae4cc67da98f 100644
--- a/clang/lib/Sema/SemaChecking.cpp
+++ b/clang/lib/Sema/SemaChecking.cpp
@@ -12057,10 +12057,10 @@ void Sema::CheckImplicitConversion(Expr *E, QualType T, SourceLocation CC,
// Strip vector types.
if (isa<VectorType>(Source)) {
if (Target->isSveVLSBuiltinType() &&
- (Context.areCompatibleSveTypes(QualType(Target, 0),
- QualType(Source, 0)) ||
- Context.areLaxCompatibleSveTypes(QualType(Target, 0),
- QualType(Source, 0))))
+ (ARM().areCompatibleSveTypes(QualType(Target, 0),
+ QualType(Source, 0)) ||
+ ARM().areLaxCompatibleSveTypes(QualType(Target, 0),
+ QualType(Source, 0))))
return;
if (Target->isRVVVLSBuiltinType() &&
@@ -12120,10 +12120,10 @@ void Sema::CheckImplicitConversion(Expr *E, QualType T, SourceLocation CC,
const Type *OriginalTarget = Context.getCanonicalType(T).getTypePtr();
// Handle conversion from scalable to fixed when msve-vector-bits is
// specified
- if (Context.areCompatibleSveTypes(QualType(OriginalTarget, 0),
- QualType(Source, 0)) ||
- Context.areLaxCompatibleSveTypes(QualType(OriginalTarget, 0),
- QualType(Source, 0)))
+ if (ARM().areCompatibleSveTypes(QualType(OriginalTarget, 0),
+ QualType(Source, 0)) ||
+ ARM().areLaxCompatibleSveTypes(QualType(OriginalTarget, 0),
+ QualType(Source, 0)))
return;
// If the vector cast is cast between two vectors of the same size, it is
diff --git a/clang/lib/Sema/SemaExpr.cpp b/clang/lib/Sema/SemaExpr.cpp
index c7abbbd6993de..087db40573ef6 100644
--- a/clang/lib/Sema/SemaExpr.cpp
+++ b/clang/lib/Sema/SemaExpr.cpp
@@ -51,6 +51,7 @@
#include "clang/Sema/ParsedTemplate.h"
#include "clang/Sema/Scope.h"
#include "clang/Sema/ScopeInfo.h"
+#include "clang/Sema/SemaARM.h"
#include "clang/Sema/SemaCUDA.h"
#include "clang/Sema/SemaFixItUtils.h"
#include "clang/Sema/SemaHLSL.h"
@@ -9533,8 +9534,8 @@ AssignConvertType Sema::CheckAssignmentConstraints(QualType LHSType,
// Allow assignments between fixed-length and sizeless SVE vectors.
if ((LHSType->isSVESizelessBuiltinType() && RHSType->isVectorType()) ||
(LHSType->isVectorType() && RHSType->isSVESizelessBuiltinType()))
- if (Context.areCompatibleSveTypes(LHSType, RHSType) ||
- Context.a...
[truncated]
|
|
@llvm/pr-subscribers-backend-aarch64 Author: Eli Friedman (efriedma-quic) ChangesThis is similar to -msve-vector-bits, but for streaming mode: it constrains the legal values of "vscale", allowing optimizations based on that constraint. This also fixes conversions between SVE vectors and fixed-width vectors in streaming functions with -msve-vector-bits and This currently doesn't touch the __ARM_FEATURE_SVE_BITS define or the arm_sve_vector_bits attribute. Patch is 33.02 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/144611.diff 15 Files Affected:
diff --git a/clang/include/clang/AST/ASTContext.h b/clang/include/clang/AST/ASTContext.h
index 3abb49312255a..64d4c5547341e 100644
--- a/clang/include/clang/AST/ASTContext.h
+++ b/clang/include/clang/AST/ASTContext.h
@@ -2486,15 +2486,6 @@ class ASTContext : public RefCountedBase<ASTContext> {
/// types.
bool areCompatibleVectorTypes(QualType FirstVec, QualType SecondVec);
- /// Return true if the given types are an SVE builtin and a VectorType that
- /// is a fixed-length representation of the SVE builtin for a specific
- /// vector-length.
- bool areCompatibleSveTypes(QualType FirstType, QualType SecondType);
-
- /// Return true if the given vector types are lax-compatible SVE vector types,
- /// false otherwise.
- bool areLaxCompatibleSveTypes(QualType FirstType, QualType SecondType);
-
/// Return true if the given types are an RISC-V vector builtin type and a
/// VectorType that is a fixed-length representation of the RISC-V vector
/// builtin type for a specific vector-length.
diff --git a/clang/include/clang/Basic/LangOptions.def b/clang/include/clang/Basic/LangOptions.def
index 789761c1f3647..8054be1bb4e88 100644
--- a/clang/include/clang/Basic/LangOptions.def
+++ b/clang/include/clang/Basic/LangOptions.def
@@ -503,6 +503,9 @@ LANGOPT(OmitVTableRTTI, 1, 0,
LANGOPT(VScaleMin, 32, 0, "Minimum vscale value")
LANGOPT(VScaleMax, 32, 0, "Maximum vscale value")
+LANGOPT(VScaleStreamingMin, 32, 0, "Minimum streaming vscale value")
+LANGOPT(VScaleStreamingMax, 32, 0, "Maximum streaming vscale value")
+
ENUM_LANGOPT(ExtendIntArgs, ExtendArgsKind, 1, ExtendArgsKind::ExtendTo32,
"Controls how scalar integer arguments are extended in calls "
"to unprototyped and varargs functions")
diff --git a/clang/include/clang/Driver/Options.td b/clang/include/clang/Driver/Options.td
index 152df89118a6a..2e8d5b18483d7 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -5173,6 +5173,14 @@ def msve_vector_bits_EQ : Joined<["-"], "msve-vector-bits=">, Group<m_aarch64_Fe
Visibility<[ClangOption, FlangOption]>,
HelpText<"Specify the size in bits of an SVE vector register. Defaults to the"
" vector length agnostic value of \"scalable\". (AArch64 only)">;
+def msve_streaming_vector_bits_EQ
+ : Joined<["-"], "msve-streaming-vector-bits=">,
+ Group<m_aarch64_Features_Group>,
+ Visibility<[ClangOption, FlangOption]>,
+ HelpText<
+ "Specify the size in bits of an SVE vector register in streaming "
+ "mode. Defaults to the vector length agnostic value of "
+ "\"scalable\". (AArch64 only)">;
} // let Flags = [TargetSpecific]
def mvscale_min_EQ : Joined<["-"], "mvscale-min=">,
@@ -5184,6 +5192,17 @@ def mvscale_max_EQ : Joined<["-"], "mvscale-max=">,
HelpText<"Specify the vscale maximum. Defaults to the"
" vector length agnostic value of \"0\". (AArch64/RISC-V only)">,
MarshallingInfoInt<LangOpts<"VScaleMax">>;
+def mvscale_streaming_min_EQ
+ : Joined<["-"], "mvscale-streaming-min=">,
+ Visibility<[CC1Option, FC1Option]>,
+ HelpText<"Specify the vscale minimum. Defaults to \"1\". (AArch64 only)">,
+ MarshallingInfoInt<LangOpts<"VScaleStreamingMin">>;
+def mvscale_streaming_max_EQ
+ : Joined<["-"], "mvscale-streaming-max=">,
+ Visibility<[CC1Option, FC1Option]>,
+ HelpText<"Specify the vscale maximum. Defaults to the"
+ " vector length agnostic value of \"0\". (AArch64 only)">,
+ MarshallingInfoInt<LangOpts<"VScaleStreamingMax">>;
def msign_return_address_EQ : Joined<["-"], "msign-return-address=">,
Visibility<[ClangOption, CC1Option]>,
diff --git a/clang/include/clang/Sema/SemaARM.h b/clang/include/clang/Sema/SemaARM.h
index ce79e94ebdd9b..788a7abf5f9c1 100644
--- a/clang/include/clang/Sema/SemaARM.h
+++ b/clang/include/clang/Sema/SemaARM.h
@@ -82,6 +82,15 @@ class SemaARM : public SemaBase {
void handleInterruptSaveFPAttr(Decl *D, const ParsedAttr &AL);
void CheckSMEFunctionDefAttributes(const FunctionDecl *FD);
+
+ /// Return true if the given types are an SVE builtin and a VectorType that
+ /// is a fixed-length representation of the SVE builtin for a specific
+ /// vector-length.
+ bool areCompatibleSveTypes(QualType FirstType, QualType SecondType);
+
+ /// Return true if the given vector types are lax-compatible SVE vector types,
+ /// false otherwise.
+ bool areLaxCompatibleSveTypes(QualType FirstType, QualType SecondType);
};
SemaARM::ArmStreamingType getArmStreamingFnType(const FunctionDecl *FD);
diff --git a/clang/lib/AST/ASTContext.cpp b/clang/lib/AST/ASTContext.cpp
index 4d44f23c0f503..d785b5a5006cc 100644
--- a/clang/lib/AST/ASTContext.cpp
+++ b/clang/lib/AST/ASTContext.cpp
@@ -10443,87 +10443,6 @@ bool ASTContext::areCompatibleVectorTypes(QualType FirstVec,
return false;
}
-/// getSVETypeSize - Return SVE vector or predicate register size.
-static uint64_t getSVETypeSize(ASTContext &Context, const BuiltinType *Ty) {
- assert(Ty->isSveVLSBuiltinType() && "Invalid SVE Type");
- if (Ty->getKind() == BuiltinType::SveBool ||
- Ty->getKind() == BuiltinType::SveCount)
- return (Context.getLangOpts().VScaleMin * 128) / Context.getCharWidth();
- return Context.getLangOpts().VScaleMin * 128;
-}
-
-bool ASTContext::areCompatibleSveTypes(QualType FirstType,
- QualType SecondType) {
- auto IsValidCast = [this](QualType FirstType, QualType SecondType) {
- if (const auto *BT = FirstType->getAs<BuiltinType>()) {
- if (const auto *VT = SecondType->getAs<VectorType>()) {
- // Predicates have the same representation as uint8 so we also have to
- // check the kind to make these types incompatible.
- if (VT->getVectorKind() == VectorKind::SveFixedLengthPredicate)
- return BT->getKind() == BuiltinType::SveBool;
- else if (VT->getVectorKind() == VectorKind::SveFixedLengthData)
- return VT->getElementType().getCanonicalType() ==
- FirstType->getSveEltType(*this);
- else if (VT->getVectorKind() == VectorKind::Generic)
- return getTypeSize(SecondType) == getSVETypeSize(*this, BT) &&
- hasSameType(VT->getElementType(),
- getBuiltinVectorTypeInfo(BT).ElementType);
- }
- }
- return false;
- };
-
- return IsValidCast(FirstType, SecondType) ||
- IsValidCast(SecondType, FirstType);
-}
-
-bool ASTContext::areLaxCompatibleSveTypes(QualType FirstType,
- QualType SecondType) {
- auto IsLaxCompatible = [this](QualType FirstType, QualType SecondType) {
- const auto *BT = FirstType->getAs<BuiltinType>();
- if (!BT)
- return false;
-
- const auto *VecTy = SecondType->getAs<VectorType>();
- if (VecTy && (VecTy->getVectorKind() == VectorKind::SveFixedLengthData ||
- VecTy->getVectorKind() == VectorKind::Generic)) {
- const LangOptions::LaxVectorConversionKind LVCKind =
- getLangOpts().getLaxVectorConversions();
-
- // Can not convert between sve predicates and sve vectors because of
- // different size.
- if (BT->getKind() == BuiltinType::SveBool &&
- VecTy->getVectorKind() == VectorKind::SveFixedLengthData)
- return false;
-
- // If __ARM_FEATURE_SVE_BITS != N do not allow GNU vector lax conversion.
- // "Whenever __ARM_FEATURE_SVE_BITS==N, GNUT implicitly
- // converts to VLAT and VLAT implicitly converts to GNUT."
- // ACLE Spec Version 00bet6, 3.7.3.2. Behavior common to vectors and
- // predicates.
- if (VecTy->getVectorKind() == VectorKind::Generic &&
- getTypeSize(SecondType) != getSVETypeSize(*this, BT))
- return false;
-
- // If -flax-vector-conversions=all is specified, the types are
- // certainly compatible.
- if (LVCKind == LangOptions::LaxVectorConversionKind::All)
- return true;
-
- // If -flax-vector-conversions=integer is specified, the types are
- // compatible if the elements are integer types.
- if (LVCKind == LangOptions::LaxVectorConversionKind::Integer)
- return VecTy->getElementType().getCanonicalType()->isIntegerType() &&
- FirstType->getSveEltType(*this)->isIntegerType();
- }
-
- return false;
- };
-
- return IsLaxCompatible(FirstType, SecondType) ||
- IsLaxCompatible(SecondType, FirstType);
-}
-
/// getRVVTypeSize - Return RVV vector register size.
static uint64_t getRVVTypeSize(ASTContext &Context, const BuiltinType *Ty) {
assert(Ty->isRVVVLSBuiltinType() && "Invalid RVV Type");
diff --git a/clang/lib/Basic/Targets/AArch64.cpp b/clang/lib/Basic/Targets/AArch64.cpp
index e8abdf9aafd82..c5935390137b1 100644
--- a/clang/lib/Basic/Targets/AArch64.cpp
+++ b/clang/lib/Basic/Targets/AArch64.cpp
@@ -822,10 +822,16 @@ std::optional<std::pair<unsigned, unsigned>>
AArch64TargetInfo::getVScaleRange(const LangOptions &LangOpts,
bool IsArmStreamingFunction,
llvm::StringMap<bool> *FeatureMap) const {
- if (LangOpts.VScaleMin || LangOpts.VScaleMax)
+ if (!IsArmStreamingFunction && (LangOpts.VScaleMin || LangOpts.VScaleMax))
return std::pair<unsigned, unsigned>(
LangOpts.VScaleMin ? LangOpts.VScaleMin : 1, LangOpts.VScaleMax);
+ if (IsArmStreamingFunction &&
+ (LangOpts.VScaleStreamingMin || LangOpts.VScaleStreamingMax))
+ return std::pair<unsigned, unsigned>(
+ LangOpts.VScaleStreamingMin ? LangOpts.VScaleStreamingMin : 1,
+ LangOpts.VScaleStreamingMax);
+
if (hasFeature("sve") || (FeatureMap && (FeatureMap->lookup("sve"))))
return std::pair<unsigned, unsigned>(1, 16);
diff --git a/clang/lib/Driver/ToolChains/Clang.cpp b/clang/lib/Driver/ToolChains/Clang.cpp
index 1d11be1d82be8..24ec25601881f 100644
--- a/clang/lib/Driver/ToolChains/Clang.cpp
+++ b/clang/lib/Driver/ToolChains/Clang.cpp
@@ -1666,7 +1666,7 @@ void Clang::AddAArch64TargetArgs(const ArgList &Args,
}
// Handle -msve_vector_bits=<bits>
- if (Arg *A = Args.getLastArg(options::OPT_msve_vector_bits_EQ)) {
+ auto HandleVectorBits = [&](Arg *A, bool Streaming) {
StringRef Val = A->getValue();
const Driver &D = getToolChain().getDriver();
if (Val == "128" || Val == "256" || Val == "512" || Val == "1024" ||
@@ -1674,22 +1674,35 @@ void Clang::AddAArch64TargetArgs(const ArgList &Args,
Val == "1024+" || Val == "2048+") {
unsigned Bits = 0;
if (!Val.consume_back("+")) {
- bool Invalid = Val.getAsInteger(10, Bits); (void)Invalid;
+ bool Invalid = Val.getAsInteger(10, Bits);
+ (void)Invalid;
assert(!Invalid && "Failed to parse value");
+ StringRef VScaleMax =
+ Streaming ? "-mvscale-streaming-max=" : "-mvscale-max=";
CmdArgs.push_back(
- Args.MakeArgString("-mvscale-max=" + llvm::Twine(Bits / 128)));
+ Args.MakeArgString(VScaleMax + llvm::Twine(Bits / 128)));
}
- bool Invalid = Val.getAsInteger(10, Bits); (void)Invalid;
+ bool Invalid = Val.getAsInteger(10, Bits);
+ (void)Invalid;
assert(!Invalid && "Failed to parse value");
+
+ StringRef VScaleMin =
+ Streaming ? "-mvscale-streaming-min=" : "-mvscale-min=";
CmdArgs.push_back(
- Args.MakeArgString("-mvscale-min=" + llvm::Twine(Bits / 128)));
- // Silently drop requests for vector-length agnostic code as it's implied.
- } else if (Val != "scalable")
+ Args.MakeArgString(VScaleMin + llvm::Twine(Bits / 128)));
+ } else if (Val == "scalable") {
+ // Silently drop requests for vector-length agnostic code as it's implied.
+ } else {
// Handle the unsupported values passed to msve-vector-bits.
D.Diag(diag::err_drv_unsupported_option_argument)
<< A->getSpelling() << Val;
- }
+ }
+ };
+ if (Arg *A = Args.getLastArg(options::OPT_msve_vector_bits_EQ))
+ HandleVectorBits(A, /*Streaming*/ false);
+ if (Arg *A = Args.getLastArg(options::OPT_msve_streaming_vector_bits_EQ))
+ HandleVectorBits(A, /*Streaming*/ true);
AddAAPCSVolatileBitfieldArgs(Args, CmdArgs);
diff --git a/clang/lib/Frontend/CompilerInvocation.cpp b/clang/lib/Frontend/CompilerInvocation.cpp
index 2c02719121c73..b3334b1bc560a 100644
--- a/clang/lib/Frontend/CompilerInvocation.cpp
+++ b/clang/lib/Frontend/CompilerInvocation.cpp
@@ -4558,6 +4558,11 @@ bool CompilerInvocation::ParseLangArgs(LangOptions &Opts, ArgList &Args,
if (StringRef(A->getValue()).getAsInteger(10, VScaleMin) || VScaleMin == 0)
Diags.Report(diag::err_cc1_unbounded_vscale_min);
}
+ if (Arg *A = Args.getLastArg(options::OPT_mvscale_streaming_min_EQ)) {
+ unsigned VScaleMin;
+ if (StringRef(A->getValue()).getAsInteger(10, VScaleMin) || VScaleMin == 0)
+ Diags.Report(diag::err_cc1_unbounded_vscale_min);
+ }
if (const Arg *A = Args.getLastArg(OPT_frandomize_layout_seed_file_EQ)) {
std::ifstream SeedFile(A->getValue(0));
diff --git a/clang/lib/Sema/SemaARM.cpp b/clang/lib/Sema/SemaARM.cpp
index e992a1012fde0..ec892a5d82421 100644
--- a/clang/lib/Sema/SemaARM.cpp
+++ b/clang/lib/Sema/SemaARM.cpp
@@ -1409,4 +1409,101 @@ void SemaARM::CheckSMEFunctionDefAttributes(const FunctionDecl *FD) {
}
}
+/// getSVETypeSize - Return SVE vector or predicate register size.
+static uint64_t getSVETypeSize(ASTContext &Context, const BuiltinType *Ty,
+ bool IsStreaming) {
+ assert(Ty->isSveVLSBuiltinType() && "Invalid SVE Type");
+ uint64_t VScale = IsStreaming ? Context.getLangOpts().VScaleStreamingMin
+ : Context.getLangOpts().VScaleMin;
+ if (Ty->getKind() == BuiltinType::SveBool ||
+ Ty->getKind() == BuiltinType::SveCount)
+ return (VScale * 128) / Context.getCharWidth();
+ return VScale * 128;
+}
+
+bool SemaARM::areCompatibleSveTypes(QualType FirstType, QualType SecondType) {
+ bool IsStreaming = false;
+ if (const FunctionDecl *FD = SemaRef.getCurFunctionDecl(/*AllowLambda=*/true))
+ if (IsArmStreamingFunction(FD, /*IncludeLocallyStreaming=*/true))
+ IsStreaming = true;
+ auto IsValidCast = [&](QualType FirstType, QualType SecondType) {
+ if (const auto *BT = FirstType->getAs<BuiltinType>()) {
+ if (const auto *VT = SecondType->getAs<VectorType>()) {
+ // Predicates have the same representation as uint8 so we also have to
+ // check the kind to make these types incompatible.
+ ASTContext &Context = getASTContext();
+ if (VT->getVectorKind() == VectorKind::SveFixedLengthPredicate)
+ return BT->getKind() == BuiltinType::SveBool;
+ else if (VT->getVectorKind() == VectorKind::SveFixedLengthData)
+ return VT->getElementType().getCanonicalType() ==
+ FirstType->getSveEltType(Context);
+ else if (VT->getVectorKind() == VectorKind::Generic)
+ return Context.getTypeSize(SecondType) ==
+ getSVETypeSize(Context, BT, IsStreaming) &&
+ Context.hasSameType(
+ VT->getElementType(),
+ Context.getBuiltinVectorTypeInfo(BT).ElementType);
+ }
+ }
+ return false;
+ };
+
+ return IsValidCast(FirstType, SecondType) ||
+ IsValidCast(SecondType, FirstType);
+}
+
+bool SemaARM::areLaxCompatibleSveTypes(QualType FirstType,
+ QualType SecondType) {
+ bool IsStreaming = false;
+ if (const FunctionDecl *FD = SemaRef.getCurFunctionDecl(/*AllowLambda=*/true))
+ if (IsArmStreamingFunction(FD, /*IncludeLocallyStreaming=*/true))
+ IsStreaming = true;
+
+ auto IsLaxCompatible = [&](QualType FirstType, QualType SecondType) {
+ const auto *BT = FirstType->getAs<BuiltinType>();
+ if (!BT)
+ return false;
+
+ const auto *VecTy = SecondType->getAs<VectorType>();
+ if (VecTy && (VecTy->getVectorKind() == VectorKind::SveFixedLengthData ||
+ VecTy->getVectorKind() == VectorKind::Generic)) {
+ const LangOptions::LaxVectorConversionKind LVCKind =
+ getLangOpts().getLaxVectorConversions();
+ ASTContext &Context = getASTContext();
+
+ // Can not convert between sve predicates and sve vectors because of
+ // different size.
+ if (BT->getKind() == BuiltinType::SveBool &&
+ VecTy->getVectorKind() == VectorKind::SveFixedLengthData)
+ return false;
+
+ // If __ARM_FEATURE_SVE_BITS != N do not allow GNU vector lax conversion.
+ // "Whenever __ARM_FEATURE_SVE_BITS==N, GNUT implicitly
+ // converts to VLAT and VLAT implicitly converts to GNUT."
+ // ACLE Spec Version 00bet6, 3.7.3.2. Behavior common to vectors and
+ // predicates.
+ if (VecTy->getVectorKind() == VectorKind::Generic &&
+ Context.getTypeSize(SecondType) !=
+ getSVETypeSize(Context, BT, IsStreaming))
+ return false;
+
+ // If -flax-vector-conversions=all is specified, the types are
+ // certainly compatible.
+ if (LVCKind == LangOptions::LaxVectorConversionKind::All)
+ return true;
+
+ // If -flax-vector-conversions=integer is specified, the types are
+ // compatible if the elements are integer types.
+ if (LVCKind == LangOptions::LaxVectorConversionKind::Integer)
+ return VecTy->getElementType().getCanonicalType()->isIntegerType() &&
+ FirstType->getSveEltType(Context)->isIntegerType();
+ }
+
+ return false;
+ };
+
+ return IsLaxCompatible(FirstType, SecondType) ||
+ IsLaxCompatible(SecondType, FirstType);
+}
+
} // namespace clang
diff --git a/clang/lib/Sema/SemaChecking.cpp b/clang/lib/Sema/SemaChecking.cpp
index 8f8e1ceb7197e..bae4cc67da98f 100644
--- a/clang/lib/Sema/SemaChecking.cpp
+++ b/clang/lib/Sema/SemaChecking.cpp
@@ -12057,10 +12057,10 @@ void Sema::CheckImplicitConversion(Expr *E, QualType T, SourceLocation CC,
// Strip vector types.
if (isa<VectorType>(Source)) {
if (Target->isSveVLSBuiltinType() &&
- (Context.areCompatibleSveTypes(QualType(Target, 0),
- QualType(Source, 0)) ||
- Context.areLaxCompatibleSveTypes(QualType(Target, 0),
- QualType(Source, 0))))
+ (ARM().areCompatibleSveTypes(QualType(Target, 0),
+ QualType(Source, 0)) ||
+ ARM().areLaxCompatibleSveTypes(QualType(Target, 0),
+ QualType(Source, 0))))
return;
if (Target->isRVVVLSBuiltinType() &&
@@ -12120,10 +12120,10 @@ void Sema::CheckImplicitConversion(Expr *E, QualType T, SourceLocation CC,
const Type *OriginalTarget = Context.getCanonicalType(T).getTypePtr();
// Handle conversion from scalable to fixed when msve-vector-bits is
// specified
- if (Context.areCompatibleSveTypes(QualType(OriginalTarget, 0),
- QualType(Source, 0)) ||
- Context.areLaxCompatibleSveTypes(QualType(OriginalTarget, 0),
- QualType(Source, 0)))
+ if (ARM().areCompatibleSveTypes(QualType(OriginalTarget, 0),
+ QualType(Source, 0)) ||
+ ARM().areLaxCompatibleSveTypes(QualType(OriginalTarget, 0),
+ QualType(Source, 0)))
return;
// If the vector cast is cast between two vectors of the same size, it is
diff --git a/clang/lib/Sema/SemaExpr.cpp b/clang/lib/Sema/SemaExpr.cpp
index c7abbbd6993de..087db40573ef6 100644
--- a/clang/lib/Sema/SemaExpr.cpp
+++ b/clang/lib/Sema/SemaExpr.cpp
@@ -51,6 +51,7 @@
#include "clang/Sema/ParsedTemplate.h"
#include "clang/Sema/Scope.h"
#include "clang/Sema/ScopeInfo.h"
+#include "clang/Sema/SemaARM.h"
#include "clang/Sema/SemaCUDA.h"
#include "clang/Sema/SemaFixItUtils.h"
#include "clang/Sema/SemaHLSL.h"
@@ -9533,8 +9534,8 @@ AssignConvertType Sema::CheckAssignmentConstraints(QualType LHSType,
// Allow assignments between fixed-length and sizeless SVE vectors.
if ((LHSType->isSVESizelessBuiltinType() && RHSType->isVectorType()) ||
(LHSType->isVectorType() && RHSType->isSVESizelessBuiltinType()))
- if (Context.areCompatibleSveTypes(LHSType, RHSType) ||
- Context.a...
[truncated]
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is similar to -msve-vector-bits, but for streaming mode: it constrains the legal values of "vscale", allowing optimizations based on that constraint.
This also fixes conversions between SVE vectors and fixed-width vectors in streaming functions with -msve-vector-bits and
-msve-streaming-vector-bits.
This currently doesn't touch the __ARM_FEATURE_SVE_BITS define or the arm_sve_vector_bits attribute.