Slow AST builds in code involving large array (e.g. 1000000 elements) #1808
Comments
|
ADD: The performance issue do not occured in Linux. I don't know why. |
|
Clangd will only index the project if it contains a I assume that's not the case for you, so there should be no index to begin with.
How many source files are we talking about? And how many of them are open? Does the slowness persist if you restart vscode (using the workspace with many files), but only open the one file you're working on? |
Each source file is between 6-10kb (I use a fixed code template); there are about 50 files in a directory, and the entire workspace has about 250 files; even if I only leave one tab open, the lag issue still persists. The typical states during lag are "File is queued" and "Parsing Includes," especially "Parsing Includes." Even after restarting VSCode or clangd, the issue still persists. |
d0j1a1701
changed the title
|
This recording may give you a better understanding of my problem here. capture.mp4 |
|
The status bar showing "Parsing includes" as you're typing is expected, as clangd reparses the file in response to the edits you're making. I'm also not seeing anything out of the ordinary in the posted video, but maybe I just don't know what to look for. What operation is taking longer than expected, and at what timestamp in the video is it performed? One other thing I'll add is that 250 files should not be an issue, even if the project is indexed. I regularly use clangd with projects containing ~10,000 source files, and even with the project being indexed that doesn't really slow down the responsiveness of any editor operations. |
I've noticed that this is not an official windows clangd release. Looking around the build source, I suspect the slowness you are confronted with relates to the WinLibs threading model. (They're both from the same author, right?) However, I could not 100% conclude on this since I don't have a mingw-w64 installed nor enough knowledge on the windows threading model. One suggestion I could give is, why don't have a try with the official windows release? It is theoretically compatible with your msys / mingw terminal and should be a drop-in replacement for the (And please let us know if the laggy behavior persists or disappears. :) |
Yes! I'm using WinLibs. (MinGW-W64 x86_64-ucrt-mcf-seh) But the official |
One typical way is to have clangd extract the include paths from gcc using the |
I edited my Compiler fragment but nothing changed. config.yaml in If:
PathMatch: .*\.cpp
CompileFlags:
Add: [-xc++, -Wall, -Wextra, -Wno-unused-variable, -std=gnu++2b]
Compiler: g++
CompilationDatabase: None
Diagnostics:
ClangTidy:
Add: [performance*,bugprone*]
Remove: [bugprone-narrowing-conversions,bugprone-use-after-move]
UnusedIncludes: Strict
InlayHints:
Enabled: Yes
ParameterNames: Yes
DeducedTypes: Yes
Designators: No
---
If:
PathMatch: .*\.c
CompileFlags:
Add: [-xc, -Wall,-Wextra,-Wno-unused-variable,-std=C23]
Compiler: gcc
CompilationDatabase: None |
I might have overlooked something, but it seems that you're not running clangd with the --query-driver option. That being said, if your winlib gcc is located in Then make sure there are respective fragments in CompileFlags:
Add: ...
Compiler: D:\Software\Mingw-w64\bin\gcc.exe |
|
I specified the |
|
The problem still exists, it's just that this time it's stuck at "Parsing Includes, parsing main file". Every time I input a character, |
|
I believe it is not an issue with the |
This is, indeed, outrageously slow. For comparison, these actions usually take less than 10 ms on my Linux machine, even when I'm editing a medium-sized file with approx 1k ~ 2k lines. Let me setup a windows environment and see if I can reproduce it. @HighCommander4 Shall we transfer this issue to the server repo? |
|
So I can now reproduce the slowness. I've set up a windows dev with mingw-w64 gcc and clangd 17.0.2, and recovered the original code from the log above: #include <iostream>
#include <vector>
using namespace std;
/* d0j1a_1701 FastIO full ver. 4.2 */
//#define FIO //缓存模式 请勿在交互题开启
struct IO {
#ifdef FIO
const static int BUFSIZE = 1 << 20;
char buf[BUFSIZE], obuf[BUFSIZE], *p1, *p2, *pp;
inline char getchar() {
return (p1 == p2 && (p2 = (p1 = buf) + fread(buf, 1, BUFSIZE, stdin), p1 == p2) ? EOF : *p1++);
}
inline void putchar(char x) {
((pp - obuf == BUFSIZE && (fwrite(obuf, 1, BUFSIZE, stdout), pp = obuf)), *pp = x, pp++);
}
inline IO &flush() {
fwrite(obuf, 1, pp - obuf, stdout);
fflush(stdout);
return *this;
}
IO() {
p1 = buf, p2 = buf, pp = obuf;
}
~IO() {
flush();
}
#else
int (*getchar)() = &::getchar;
int (*putchar)(int) = &::putchar;
inline IO &flush() {
fflush(stdout);
return *this;
};
#endif
string sep = " ";
int k = 2;
template < typename Tp, typename std::enable_if < std::is_integral<Tp>::value || std::is_same<Tp, __int128_t>::value >::type * = nullptr >
inline int read(Tp &s) {
int f = 1;
char ch = getchar();
s = 0;
while(!isdigit(ch) && ch != EOF) f = (ch == '-' ? -1 : 1), ch = getchar();
while(isdigit(ch)) s = s * 10 + (ch ^ 48), ch = getchar();
s *= f;
return ch != EOF;
}
template<typename Tp, typename enable_if<is_floating_point<Tp>::value>::type * = nullptr>
inline int read(Tp &s) {
int f = 1;
char ch = getchar();
s = 0;
while(!isdigit(ch) && ch != EOF && ch != '.') f = (ch == '-' ? -1 : 1), ch = getchar();
while(isdigit(ch)) s = s * 10 + (ch ^ 48), ch = getchar();
if(ch == EOF) return false;
if(ch == '.') {
Tp eps = 0.1;
ch = getchar();
while(isdigit(ch)) s = s + (ch ^ 48) * eps, ch = getchar(), eps /= 10;
}
s *= f;
return ch != EOF;
}
inline int read(char &c) {
char ch = getchar();
c = EOF;
while(isspace(ch) && ch != EOF) ch = getchar();
if(ch != EOF) c = ch;
return c != EOF;
}
inline int read(char *c) {
char ch = getchar(), *s = c;
while(isspace(ch) && ch != EOF) ch = getchar();
while(!isspace(ch) && ch != EOF) *(c++) = ch, ch = getchar();
*c = '\\0';
return s != c;
}
inline int read(string &s) {
s.clear();
char ch = getchar();
while(isspace(ch) && ch != EOF) ch = getchar();
while(!isspace(ch) && ch != EOF) s += ch, ch = getchar();
return s.size() > 0;
}
inline int getline(char *c, const char &ed = '\\n') {
char ch = getchar(), *s = c;
while(ch != ed && ch != EOF) *(c++) = ch, ch = getchar();
*c = '\\0';
return s != c;
}
inline int getline(string &s, const char &ed = '\\n') {
s.clear();
char ch = getchar();
while(ch != ed && ch != EOF) s += ch, ch = getchar();
return s.size() > 0;
}
template<typename Tp = int>
inline Tp read() {
Tp x;
read(x);
return x;
}
template<typename Tp, typename... Ts>
int read(Tp &x, Ts &...val) {
return read(x) && read(val...);
}
template<typename Tp, typename enable_if<is_integral<Tp>::value>::type * = nullptr>
IO & write(Tp x) {
if(x < 0) putchar('-'), x = -x;
static char sta[20];
int top = 0;
do sta[top++] = x % 10 + '0', x /= 10;
while(x);
while(top)
putchar(sta[--top]);
return *this;
}
inline IO &write(const string &str) {
for(char ch : str) putchar(ch);
return *this;
}
inline IO &write(const char *str) {
while(*str != '\0') putchar(*(str++));
return *this;
}
inline IO &write(char *str) {
return write((const char *)str);
}
inline IO &write(const char &ch) {
return putchar(ch), *this;
}
template<typename Tp, typename enable_if<is_floating_point<Tp>::value>::type * = nullptr>inline IO & write(Tp x) {
if(x > 1e18 || x < -1e18) {
write("[Floating point overflow]");
throw;
}
if(x < 0) putchar('-'), x = -x;
const static long long pow10[] = {1, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000, 10000000000, 100000000000, 1000000000000, 10000000000000, 100000000000000, 1000000000000000, 10000000000000000, 100000000000000000, 100000000000000000, 100000000000000000};
const auto &n = pow10[k];
long long whole = (int)x;
double tmp = (x - whole) * n;
long long frac = tmp;
double diff = tmp - frac;
if(diff > 0.5) {
++frac;
if(frac >= n) frac = 0, ++whole;
} else if(diff == 0.5 && ((frac == 0U) || (frac & 1U))) ++frac;
write(whole);
if(k == 0U) {
diff = x - (double)whole;
if((!(diff < 0.5) || (diff > 0.5)) && (whole & 1))
++whole;
} else {
putchar('.');
static char sta[20];
int count = k, top = 0;
while(frac) {
sta[top++] = frac % 10 + '0';
frac /= 10, count--;
}
while(count--) putchar('0');
while(top) putchar(sta[--top]);
}
return *this;
}
template<typename Tp, typename... Ts>
inline IO &write(Tp x, Ts... val) {
write(x);
write(sep);
write(val...);
return *this;
}
template<typename... Ts>
inline IO &writeln(Ts... val) {
write(val...);
putchar('\n');
return *this;
}
inline IO &writeln(void) {
putchar('\n');
return *this;
}
template<typename Tp>
inline IO &writeWith(Tp x, const string &s = " ") {
write(x), write(s);
return *this;
}
inline IO &setsep(const string &s) {
return sep = s, *this;
}
inline IO &setprec(const int &K) {
return k = K, *this;
}
} io;
vector<pair<int, int> > edges[100010];
int n;
int main() {
io.read(n);
for(int i = 1, u, v, w; i <= n; i++)
io.read(u, v, w);
edges[0].^ // The code completion takes long to respond here!
system("pause");
return 0;
}I can clearly observe the slowness while typing something ordinary such as accessing a I[13:27:15.956] ASTWorker building file f:\clangd-17.0.2\clangd_17.0.2\bin\main.cpp version 106 with command
[F:\clangd-17.0.2\clangd_17.0.2\bin\]
"C:\\msys64\\mingw64\\bin\\g++.exe" --driver-mode=g++ -std=c++20 -isystem C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.1.0/../../../../include/c++/13.1.0 -isystem C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.1.0/../../../../include/c++/13.1.0/x86_64-w64-mingw32 -isystem C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.1.0/../../../../include/c++/13.1.0/backward -isystem C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.1.0/include -isystem C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.1.0/../../../../include -isystem C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.1.0/include-fixed --target=x86_64-w64-mingw32 "-resource-dir=F:\\clangd-17.0.2\\clangd_17.0.2\\lib\\clang\\17" -- "f:\\clangd-17.0.2\\clangd_17.0.2\\bin\\main.cpp"
I[13:27:15.958] --> textDocument/clangd.fileStatus
I[13:27:16.164] <-- textDocument/codeAction(463)
I[13:27:16.164] <-- $/cancelRequest
I[13:27:17.071] --> textDocument/publishDiagnostics
I[13:27:17.072] --> reply:textDocument/codeAction(459) 2127 ms, error: Task was cancelled.
[Error - 1:27:17 PM] Request textDocument/codeAction failed.
[object Object]
I[13:27:17.074] --> reply:textDocument/documentSymbol(460) 1755 ms
I[13:27:17.074] --> reply:textDocument/documentLink(461) 1650 ms
I[13:27:17.074] --> reply:textDocument/inlayHint(462) 1401 ms
I[13:27:17.074] --> reply:textDocument/codeAction(463) 910 ms
I[13:27:17.074] --> textDocument/clangd.fileStatus
I[13:27:17.236] <-- textDocument/semanticTokens/full/delta(464)
I[13:27:17.239] --> reply:textDocument/semanticTokens/full/delta(464) 2 ms
I[13:27:17.239] --> textDocument/clangd.fileStatus
I[13:27:17.331] <-- textDocument/codeAction(465)I've also performed the same edit action on my Linux machine, and the time cost looks much more reasonable. |
|
The sluggish responsiveness on code completion sheerly disappears if I reduce the array size to a smaller number, like 10. vector<pair<int, int> > edges[100010];to vector<pair<int, int> > edges[10];And why can't it be replicated on other platforms? |
|
Simplified code snippet: struct S {
int data;
void *ptr;
} s[1000000]; // Note the size here
int main() {
s[0].^
}One can observe the responsiveness latency cliff by adding one more 0 to the array size. I recalled that we had a similar bug report on the fixed array size, llvm/llvm-project#63562. I'm not sure if it is the same, though. Note that the same workaround mentioned in that issue applies too: use the standard prior to c++20. CompileFlags:
Add: [-std=c++17] |
Okay, I will use C++ 17 until the issue is resolved. I wish you smooth debugging and a prompt resolution to the problem. |
|
I remember that there was a similar bug in clangd before, where for extremely large struct static arrays like this, clangd would attempt to allocate all the memory at once, resulting in a significant memory usage. Perhaps it could be related to that? struct S {
int data;
void *ptr;
} s[1000000]; |
|
@zyn0217 I conducted the same test in Ubuntu LTS on WSL2 and found that this issue also exists in clangd on WSL2. The test code is the minimal reproducible code you provided. |
Maybe. I don't have an available nightly build for Windows at the time, so it's hard to tell if this has been resolved recently. (The commit that issue links to does not appear to be cherry-picked to the 17.0.x branch, so I assume we won't see the fix until clangd 18, which ought to be rolled out in the 2024H1.)
Unfortunately, I can reproduce this as well, using a recently-built version of clangd on Linux. ;) |
|
Okay. However, the memory issue I mentioned probably occurred between 2020 and 2021. |
Transferred. |
HighCommander4
changed the title
|
Problem still NOT resolved with clangd 18.1.3（LLVM-Mingw Project） #include <iostream>
using namespace std;
pair<int, int> arr[100000010];
int main() { return 0; } |
|
And the problem occured even if using "-std=c++17". |
|
I met the same problem. clangd will become very slow when I define a large array of This problem only occurs on I am using clangd 18.1.3 with ubuntu 22.04 |
|
In my case, clangd only consumes a lot of CPU but doesn't consume much memory.
|
I am an algorithmic competition participant, and I have a VSCode workspace where I have organized my code from various online judging systems into folders. The directory tree looks roughly like this:
Clearly, my problem files are independent of each other, and I don't need or want to include the contents of other files in one file. I only need to include system header files like
iostreamandvector.However, when my problem repository folder reaches a certain size, I noticed that the response time of
clangdbecomes significantly slower. Below is a part of theclangdlog where it can be observed that many operations take several thousand milliseconds.When I created a workspace with only one file, the performance of
clangdreturned to normal.I suspect that Clangd scans the entire folder (or even the workspace) of code every time it performs autocompletion (even though I haven't included them). How can I change this behavior?
The text was updated successfully, but these errors were encountered: