Add a global convenience package file #5916
Conversation
|
Thanks for your pull request, @wilzbach! Bugzilla referencesYour PR doesn't reference any Bugzilla issue. If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog. |
|
Even if you don't intend to documentation-build it yourself, could you do me a favor and add: to the top of it? will save me a small merge issue when i pull to adrdox again (I actually use a package file to create a landing page for the library http://dpldocs.info/experimental-docs/std.html ) |
|
I have reservations about this.
That said, it wouldn't be the first time we do something that has traditionally be shunned. We do need a fresh angle on it and a clear case that it steers clear of issues while reaping good advantages. For example, it would be a great showcase of fast compilation if time spent on importing unused imports were negligible. cc @WalterBright |
|
This is one of those changes where I see little/no reason to disallow it (and any argument against it can be made for disallowing I can see how this can be useful for prototyping or just figuring something out. E.g. yesterday I forgot how Also, it's useful to remember that it's the little conveniences which make people fall in love with languages, e.g. list comprehensions in Python. The only technical issue I could see from this is if two symbols in different modules have the same name, in which case they could add a |
wilzbach
force-pushed
the
global-convenience-package
branch
from
b80838e
to
38bef5a
Compare
Sure. Done. I also added a changelog entry.
On my machine it's ~1.5s, but at least for myself that's something I am more than willing to accept in exchange for the time saved typing all the imports.
The way I see Anyhow I wrote a script to check the status quo of the import loading time of each module:
(the script did nothing more than |
Well that problem already exists today if you e.g. import |
|
I'd like to have such option. In GCC version of C++, there's I use D for competitive programming solutions whenever I can, and I would like to have an option to just I'd like to reiterate that |
|
No opinion on this pull, but doesn't rdmd already enable this? |
|
@joakim-noah IIRC, rdmd hard-codes a bunch of "usual imports" from Phobos by default. I don't think it actually covers every Phobos module. But I could be wrong. |
|
Rdmd doesn't do this, but it has a https://github.com/dlang/tools/blob/master/rdmd.d#L823 For this most of the Phobos world is imported, but it's not really related to this PR as |
|
Cool, thanks for the info @GassaFM. @DmitryOlshansky is looking into the regex slowdown and @wilzbach into the others. I think we have a good chance to accelerate their compilation significantly. BTW I just timed the inclusion of |
As @andralex asked me a to look a bit at other languages and how they handle/support/recommend wildcard imports.
For completeness, for C++ there's this StackOverflow post on why
Imho the last point is the only valid concern which affects D too. I don't see this as a big concern as the Lastly, there's also the idea of putting this in sth. like |
|
@wilzbach, OK, I thought rdmd also enabled writing scripts without listing your imports, though I'd obviously never tried it, since it apparently doesn't. I don't think speed is that big a deal, but that affording this convenience now means we'll see a bunch of D code that simply uses |
Cool. Looking forward!
Hehe that's good! I wouldn't count on the DUB package being "usable" in the next few weeks - there just seem to be too many outstanding issues with it. |
It's not that crude... It parses and caches JSON data emitted by DMD, and emits editor-agnostic edit instructions for each option. It also recognizes typos and such. Here's the Emacs half: https://dump.thecybershadow.net/6adf07f0a11fde8ba11dedfa4309ecab/dfix.el I'm waiting until the Flycheck guys finally implement flycheck/flycheck#530 so that it could be built into d-mode or something. |
|
Of the concerns mentioned by other languages, the one that new symbols in the library may break the program is the most important. See also https://github.com/dlang/DIPs/blob/master/DIPs/DIP1007.md. One thing we could do from a naming perspective is to replace |
This would also give room for us to add various useful scripting functions to the module in the future. |
Just a quick status update: struct Message
{
Variant data;
}An example: import std.variant;semantic bar
semantic2 bar
semantic3 bar
dmd -v -c -o- bar.d -I.. 0.02s user 0.00s system 99% cpu 0.024 totaland now let's define a import std.variant;
Variant d;semantic bar
import std.algorithm.comparison (../std/algorithm/comparison.d)
import std.range.primitives (../std/range/primitives.d)
import std.algorithm.internal (../std/algorithm/internal.d)
semantic2 bar
semantic3 bar
import core.stdc.string (/usr/include/dlang/dmd/core/stdc/string.d)
import std.conv (../std/conv.d)
import std.ascii (../std/ascii.d)
import std.exception (../std/exception.d)
import std.format (../std/format.d)
import core.vararg (/usr/include/dlang/dmd/core/vararg.d)
import std.array (../std/array.d)
import std.algorithm.iteration (../std/algorithm/iteration.d)
import core.memory (/usr/include/dlang/dmd/core/memory.d)
import std.algorithm.searching (../std/algorithm/searching.d)
import std.bitmanip (../std/bitmanip.d)
import std.system (../std/system.d)
import core.internal.string (/usr/include/dlang/dmd/core/internal/string.d)
import core.bitop (/usr/include/dlang/dmd/core/bitop.d)
import std.utf (../std/utf.d)
import core.checkedint (/usr/include/dlang/dmd/core/checkedint.d)
import std.string (../std/string.d)
import std.uni (../std/uni.d)
import std.internal.unicode_tables (../std/internal/unicode_tables.d)
import std.range (../std/range/package.d)
import std.range.interfaces (../std/range/interfaces.d)
import std.algorithm.mutation (../std/algorithm/mutation.d)
dmd -v -c -o- bar.d -I.. 0.20s user 0.03s system 99% cpu 0.227 totalSo yeah having a |
|
@wilzbach great work on variant! I've investigated a bit more and found tuple to be the culprit, i.e. the cost of importing std.variant is almost 100% caused by the use of std.tuple. Look at this program: import std.typecons;
Tuple!(int, double) t;The variable definition adds a large compilation time. |
|
Further investigating the matter, it looks like most time is being spent in this function: // Generates named fields as follows:
// alias name_0 = Identity!(field[0]);
// alias name_1 = Identity!(field[1]);
// :
// NOTE: field[k] is an expression (which yields a symbol of a
// variable) and can't be aliased directly.
string injectNamedFields()
{
string decl = "";
foreach (i, name; staticMap!(extractName, fieldSpecs))
{
import std.format : format;
decl ~= format("alias _%s = Identity!(field[%s]);", i, i);
if (name.length != 0)
{
decl ~= format("alias %s = _%s;", name, i);
}
}
return decl;
}Replacing the function with |
Wow. Excellent catch! static foreach (i, name; staticMap!(extractName, fieldSpecs))
{{
import std.conv : to;
const iStr = i.to!string;
decl ~= "alias _" ~ iStr ~ " = Identity!(field[" ~ iStr ~"]);";
if (name.length != 0)
decl ~= "alias " ~ name ~ " = _"~ iStr ~";";
}}
CC @UplinkCoder |
wilzbach
force-pushed
the
global-convenience-package
branch
from
a85b49f
to
01e03a8
Compare
0.42s on my system with debug dmd - as mentioned
Ok. Even if I don't really like it, we should put it in Anyhow |
We have a much better tool for this: https://github.com/CyberShadow/DBuildStat
It doesn't look like it: |
|
Here is a random sample stacktrace of DMD parsing std.net.curl: I'm not sure that stack trace really should be that deep. Could it be doing some redundant work for some reason? Edit: Looking into this. I think we can improve our tooling for such situations as well. |
|
How'd you do that collapsable section? |
HTML |
|
GDB Python script which walks DMD's stack and prints arguments' locations: import gdb
import os.path
def get_loc(val):
if (val.type.code == gdb.TYPE_CODE_PTR and
val.type.target().name is not None and (
val.type.target().name.endswith("Statement") or
val.type.target().name.endswith("Expression"))):
return get_loc(val.referenced_value())
if val.type.name is None:
return None
if val.type.name.endswith("Statement"):
return val.cast(gdb.lookup_type("dmd.statement.Statement"))["loc"]
if val.type.name.endswith("Expression"):
return val.cast(gdb.lookup_type("dmd.expression.Expression"))["loc"]
return None
oldlocstr = ""
frame = gdb.newest_frame()
while frame:
block = frame.block()
while block:
if not block.is_global:
for symbol in block:
if symbol.is_argument:
loc = get_loc(symbol.value(frame))
if loc is not None:
try:
filename = loc["filename"].string("utf-8")
line = int(loc["linnum"])
char = int(loc["charnum"])
locstr = "{}({},{})".format(filename, line, char)
if locstr != oldlocstr:
oldlocstr = locstr
if os.path.exists(filename):
linestr = open(filename).readlines()[line-1]
locstr += ": " + linestr.rstrip('\n')
print(locstr)
except (gdb.MemoryError):
pass
block = block.superblock
frame = frame.older()Output: Looks like it's slow because pulling in (and actually using) std.regex is pulling in a lot of other stuff (std.uni, std.algorithm)... This script might come in useful in the future. |
Sorry, that was wrong. I decided to continue working and expanding on the Python script above and ended up with a sampling profiler for compilation time (i.e. it profiles which parts of your program take up most time being compiled). Here is the result for Click on the above image to get on-hover tooltips showing the contents of the source line in question, and clickable links to GitHub. Some of the longer chains:
Another:
Another:
|
|
On GitHub: https://github.com/CyberShadow/dmdprof |
wilzbach
force-pushed
the
global-convenience-package
branch
from
01e03a8
to
f70875c
Compare
Here's a patch that gets that down to 0.35... not sure if it's worthwhile: https://github.com/dlang/phobos/compare/master...CyberShadow:std-net-curl-async-ns?w=1 |
wilzbach
force-pushed
the
global-convenience-package
branch
from
f70875c
to
62174b5
Compare
I didn't know about this. I guess you have to write an article for the DBlog now ;-)
That's pretty amazing! How about integrating this into the Phobos Makefile (or maybe even DAutoTest)? Anyhow one of the bigger costs comes from importing the huge unicode_table:
I already figured that one out without your fancy graph and have a PR to fix that #5945 Now back to the PR - anything else blocking this PR? (I improved the changelog and added two examples to the |
Imho it is. Didn't we want to get deprecate these *Async functions anyhow? |
OK, PR sent: #6122
I think the main problem here was that top-level non-templated functions used types from imported modules ( |
|
Alrighty people, seeing that this has already spawned a lot of great work and there's still working going on (e.g. #6129 or #5945), I think we should move forward and give this a try in Also there's yet another rant on the NG:
... and this time we can address it by a simple merge ;-) |
| @@ -0,0 +1,38 @@ | |||
| `import std.experimental.scripting` as a global convenience import | |||
|
|
|||
| $(MREF std,experimental,scripting) allows to conveniently use all Phobos modules | |||
There was a problem hiding this comment.
Either
allows/gives convenient use of
or
allows us/one to conveniently use
| $(LINK2 $(ROOT)spec/module.html#renamed_imports, renamed imports) can be used | ||
| to uniquely select a specific symbol. | ||
|
|
||
| Importing entire Phobos costs less than half a second (varying on the system) and |
There was a problem hiding this comment.
Specify that you mean just importing and not using anything, as compile times will increase as soon as you instantiate a template from the module.
There was a problem hiding this comment.
I reworded to:
The baseline cost for
import std.experimental.scriptingis less than half a second (varying from system to system) and work is in progress to reduce this overhead even further.
Less potentially confusing?
std/experimental/scripting.d
Outdated
| @@ -0,0 +1,83 @@ | |||
| /++ | |||
| Phobos is the D standard library, built on top of druntime. | |||
There was a problem hiding this comment.
I originally wanted that on the package.d to give a high level overview introduction for generated docs. It doesn't make much sense in a scripting.d file though...
std/experimental/scripting.d
Outdated
|
|
||
| Convenience file that allows to import entire Phobos in one command. | ||
| +/ | ||
| module std.experimental.scripting; |
There was a problem hiding this comment.
Just wondering, scripting seems to be a context-keyword. I understand, naming is a hard part given that it'll mostly be used for scripting, but do you think it's possible to rename?
Let's say down the line, the import times get much sleeker and people start to use this by default in D, import std.scripting would look odd 😉
There was a problem hiding this comment.
Andrei suggested scripting. My personal favorite would be all.
Other opinions?
There was a problem hiding this comment.
Also my hope is that if this gets accepted / popular, we can do import std (as initially proposed)
There was a problem hiding this comment.
all is what I came up to my mind as well and looks appropriate. Other options in the order of sensibility can be: everything, completely, phobos, entirely, totally. Can't think of other synonyms of all. 😄
There was a problem hiding this comment.
What about std.experimental.allphobos? I.e., import all of Phobos. :-P
I don't think the name matters that much, since this is std.experimental. Once it becomes widely adopted, people will just change that to import std; anyway.
wilzbach
force-pushed
the
global-convenience-package
branch
from
62174b5
to
f8684bf
Compare
|
Let's give this a whirl! Thanks for the work. |
So after I have been thinking about this for quite a while, I realized that I am almost weekly in a case where
import stdwould make my life easier.Other reasons include:
For now I didn't add this file to the documentation build.
Also I didn't include
std.experimental.