Issue 9287 - Implement -stdin. #6880

quickfur · 2017-06-09T20:48:55Z

It's nice to be able to pipe source code into dmd via standard input, e.g., in code generation programs so that you don't have to create a temporary file just to be able to compile the code.

Based on the bug notes in the original bug, using - to mean stdin is probably a problematic idea, so I have chosen a different route, i.e., use -stdin to indicate that dmd should read source code from stdin. I think this is clearer, and less *nix-specific.

quickfur · 2017-06-09T20:50:57Z

Note that I had to hack together a stdio-based way of copying data from stdin into a buffer, because the existing File class in src/ddmd/root/file.d does not work with non-file handles: it uses stat to discover the total file length, which is not available with stdin. I also opted for using stdio so that I don't have to figure out how to write a Windows-specific version which I currently have no way to test.

UplinkCoder · 2017-06-09T23:33:22Z

src/ddmd/mars.d

+    size_t pos = 0;
+    size_t sz = bufIncrement;
+
+    ubyte* buf = cast(ubyte*).malloc(sz + 1); // +1 for sentinel


malloc is only meant for buffers which are freed later.
use xmalloc amd xrealloc instead
even better would be in walking the length of the input and the using allocmemory.

Went with xmalloc / xrealloc. Didn't really want to walk the input, since it could be large, if xrealloc already takes care of allocating a large-enough buffer for us.

UplinkCoder · 2017-06-09T23:35:30Z

@quickfur what is the specific usecase for this ?

wilzbach · 2017-06-09T23:42:01Z

@quickfur if you use "Fix " as a commit message, the bot will recognize it (more infos).

@UplinkCoder most tools/compilers allow this - this is nice for a bunch of use cases starting with showing compile errors in my favorite editor (most editors read the un-saved instance in-memory and thus can nicely pipe it to dmd) - see the issue for more details.

quickfur · 2017-06-10T00:02:12Z

Also, please advise if there is a better place to put copyFile than directly in mars.d. Somehow it feels a little out-of-place there.

WalterBright · 2017-06-10T01:37:58Z

Protocol: when creating a PR to fix Issue https://issues.dlang.org/show_bug.cgi?id=9287, add a link in bugzilla to the PR. I already did so for this.

WalterBright · 2017-06-10T01:38:46Z

As I commented in https://issues.dlang.org/show_bug.cgi?id=9287 please use - not -stdin. We shouldn't be reinventing common Linux conventions.

wilzbach · 2017-06-10T01:49:56Z

12.82% of diff hit (target 85.504%)
85.461% (-0.044%) compared to ebd995a

Sorry to bother you with this, but adding a test in the testsuite for this will highly increase chances of acceptance ;-)

quickfur · 2017-06-10T02:01:58Z

@WalterBright I did put the PR link in bugzilla, did you miss it?

As for '-' instead of '-stdin', I'll get to it later. Thanks for the feedback.

PS: where's the best place to put 'copyFile'?

quickfur · 2017-06-10T02:03:44Z

@wilzbach How would you test something involving standard input? I don't know if the current test suite runner is capable of piping code via stdin.

wilzbach · 2017-06-10T02:16:16Z

@wilzbach How would you test something involving standard input? I don't know if the current test suite runner is capable of piping code via stdin.

The test suite is Turing-complete (it allows the use of Bash), see e.g. this simple test:

#!/usr/bin/env bash

src=runnable${SEP}extra-files
dir=${RESULTS_DIR}${SEP}runnable
output_file=${dir}/test10567.sh.out

$DMD -m${MODEL} -I${src} -of${dir}${SEP}test10567a${OBJ} -c ${src}${SEP}test10567a.d || exit 1
$DMD -m${MODEL} -I${src} -of${dir}${SEP}test10567${EXE} ${src}${SEP}test10567.d ${dir}${SEP}test10567a${OBJ} || exit 1
${RESULTS_DIR}/runnable/test10567${EXE} || exit 1

rm ${dir}/{test10567a${OBJ},test10567${EXE}}

echo Success >${output_file}

Simply rewriting sth. like this to use cat should work ;-)
You can run an individual test with make test_results/<name of test>.out

quickfur · 2017-06-10T05:09:20Z

Thanks for the tip! Got a working test now.

quickfur · 2017-06-10T15:55:44Z

Hmm, the Jenkins errors appear unrelated to the changes in this PR. What's going on?

quickfur · 2017-06-10T16:07:51Z

Nevermind, it seems to be affecting other PRs too.

WalterBright · 2017-06-11T00:47:46Z

src/ddmd/mars.d

+    size_t sz = bufIncrement;
+
+    ubyte* buf = cast(ubyte*)mem.xmalloc(sz + 1); // +1 for sentinel
+    if (buf is null)


xmalloc() never returns null

WalterBright · 2017-06-11T00:49:21Z

src/ddmd/mars.d

+    assert(sz > pos);
+    size_t rlen;
+    pos += rlen = fread(buf + pos, 1, sz - pos, fp);
+    while (!ferror(fp) && !feof(fp))


This loop can be written so there is only one xmalloc call instead of two, and one fread instead of two.

WalterBright · 2017-06-11T00:53:59Z

src/ddmd/mars.d

+    {
+        if (pos == sz)
+        {
+            sz += bufIncrement;


Reallocating every 4K bytes makes for a very slow read. expression.d is 500K, so that makes for 125 copies and reallocations. I'd probably make it at least 128K.

WalterBright · 2017-06-11T00:54:39Z

src/ddmd/mars.d

+        assert(sz > pos);
+        pos += rlen = fread(buf + pos, 1, sz - pos, fp);
+    }
+    if (ferror(fp))


ferror is checking twice

WalterBright · 2017-06-11T00:55:33Z

src/ddmd/mars.d

+/**
+ * Copies data from a FILE* into a newly-allocated buffer.
+ *
+ * The buffer is always null-terminated.


Use Ddoc style comments, with Params: and Returns: sections.

WalterBright · 2017-06-11T00:58:00Z

src/ddmd/mars.d

@@ -1210,6 +1260,10 @@ Language changes listed by -transition=id:
    {
        files.push(cast(char*)global.main_d); // a dummy name, we never actually look up this file
    }
+    if (global.params.readStdin)
+    {
+        files.push(cast(char*)global.stdin_d); // a dummy name, we don't actually look it up


maybe put this before line 1134, then the test doesn't have to be done twice

WalterBright · 2017-06-11T01:03:12Z

src/ddmd/mars.d

@@ -1343,6 +1397,22 @@ Language changes listed by -transition=id:
            }
        }
    }
+    if (global.params.readStdin)


Seems like this code should go into root/file.d, as reading from stdin should be a low level thing, not a high level one.

WalterBright · 2017-06-11T01:09:55Z

I suggest that if root.File.read() simply checked for "-" as a filename in File.name, the logic could all be moved there and the the rest of the complicated logic can be removed from the main code.

andralex · 2017-06-11T05:17:13Z

great review (and idea) @WalterBright

dlang-bot · 2017-07-11T23:12:42Z

Thanks for your pull request, @quickfur!

Bugzilla references

Auto-close	Bugzilla	Description
✓	9287	DMD should read from stdin when an input file is "-"

quickfur · 2017-07-11T23:13:03Z

Ah, he's expecting the word "issue". Got it, thanks!

wilzbach · 2017-07-11T23:14:52Z

Ah, he's expecting the word "issue". Got it, thanks!

Exact regex-> https://github.com/dlang-bots/dlang-bot#nerdy-details
Though I don't know why the double posts happened - maybe yet another lag in the GH event API?

quickfur · 2017-07-21T00:48:24Z

ping @WalterBright @andralex

Is this PR satisfactory, or should a different approach be taken? Please advise, thanks!

andralex · 2017-07-21T02:27:55Z

It seems the PR cannot be made much simpler. I'm mildly in favor of the notion of accepting piped input, without being able to think of a solid application. OK with me. @WalterBright ?

MartinNowak · 2017-07-26T17:14:05Z

src/ddmd/mars.d

@@ -1033,6 +1033,8 @@ Language changes listed by -transition=id:
                    goto Lnoarg;
                }
            }
+            else if (p[1] == '\0')
+                files.push("__stdin");


Why not pushing __stdin.d to avoid ambiguities?

Good idea, done.

dlang-bot · 2017-07-26T20:08:35Z

Thanks for your pull request, @quickfur!

Bugzilla references

Auto-close	Bugzilla	Description
✓	9287	DMD should read from stdin when an input file is "-"

dlang-bot · 2017-07-31T21:05:52Z

Thanks for your pull request, @quickfur!

Bugzilla references

Auto-close	Bugzilla	Description
✓	9287	DMD should read from stdin when an input file is "-"

quickfur · 2017-07-31T21:06:50Z

Rebased, hopefully this fixes the travis error which seems unrelated.

@andralex What else needs to be done to get this PR merged?

andralex · 2017-08-01T14:15:24Z

@WalterBright 's call

quickfur · 2017-08-01T17:41:03Z

@andralex I was referring to your change requested review. @WalterBright already approved this PR.

wilzbach · 2018-02-03T10:37:03Z

So why do we should we add verbose changelog entries?
(reminder for the future)

Mention passing files from stdin as potential use case
thanks Seb, didn't know that.
lindt/docker-dmd#5

Of course, it's in the changelog, but at the end and rather hard to spot:

https://dlang.org/changelog/2.076.0.html

ibuclaw · 2018-02-03T11:01:02Z

Heh, I implemented this feature two months before this PR, and it seems to have the same __stdin name given to the input module.

D-Programming-GDC/gdc@dd81b9a

quickfur force-pushed the stdin branch 3 times, most recently from 85d13f5 to 34d8d78 Compare June 9, 2017 22:47

UplinkCoder reviewed Jun 9, 2017

View reviewed changes

quickfur force-pushed the stdin branch from 34d8d78 to 9611768 Compare June 9, 2017 23:53

quickfur force-pushed the stdin branch from f01b6ae to 2ba2511 Compare June 10, 2017 04:52

quickfur force-pushed the stdin branch from 4f70b2f to 6df4bbc Compare June 10, 2017 06:05

quickfur force-pushed the stdin branch from 6df4bbc to f64f2ca Compare June 10, 2017 20:20

WalterBright reviewed Jun 11, 2017

View reviewed changes

quickfur force-pushed the stdin branch from 11cef82 to 77fc7ba Compare July 11, 2017 23:12

dlang-bot added the Enhancement label Jul 11, 2017

WalterBright approved these changes Jul 21, 2017

View reviewed changes

MartinNowak reviewed Jul 26, 2017

View reviewed changes

quickfur force-pushed the stdin branch from 77fc7ba to 86817f0 Compare July 26, 2017 20:08

H. S. Teoh added 2 commits July 31, 2017 13:57

Fix issue 9287 - implement reading source from stdin.

7d18133

Add test case.

95f3315

quickfur force-pushed the stdin branch from 86817f0 to 95f3315 Compare July 31, 2017 21:05

andralex approved these changes Aug 1, 2017

View reviewed changes

andralex added the auto-merge label Aug 1, 2017

dlang-bot merged commit 28daf53 into dlang:master Aug 1, 2017

quickfur deleted the stdin branch August 1, 2017 21:01

andre2007 mentioned this pull request Nov 9, 2017

syntax check from stdin too, like Style check dlang-community/D-Scanner#529

Open

quickfur mentioned this pull request Dec 14, 2017

Fix issue18076: allow -run to work with - (stdin) #7435

Merged

wilzbach mentioned this pull request Dec 25, 2017

Avoid creation of a temporary probe file for newer compilers dlang/dub#1316

Open

wilzbach mentioned this pull request Jan 18, 2018

Implement -stdin for rdmd wilzbach/tools-test#176

Open

wilzbach mentioned this pull request Feb 3, 2018

Mention passing files from stdin as potential use case lindt/docker-dmd#5

Merged

wilzbach mentioned this pull request Feb 3, 2018

Require a verbose changelog entry for all enhancement bugs dlang/dlang-bot#178

Open

Issue 9287 - Implement -stdin. #6880

Issue 9287 - Implement -stdin. #6880

Conversation

quickfur commented Jun 9, 2017

quickfur commented Jun 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

UplinkCoder commented Jun 9, 2017

wilzbach commented Jun 9, 2017

quickfur commented Jun 10, 2017

WalterBright commented Jun 10, 2017

WalterBright commented Jun 10, 2017

wilzbach commented Jun 10, 2017

quickfur commented Jun 10, 2017

quickfur commented Jun 10, 2017

wilzbach commented Jun 10, 2017

quickfur commented Jun 10, 2017

quickfur commented Jun 10, 2017

quickfur commented Jun 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WalterBright commented Jun 11, 2017

andralex commented Jun 11, 2017

dlang-bot commented Jul 11, 2017

Bugzilla references

quickfur commented Jul 11, 2017

wilzbach commented Jul 11, 2017

quickfur commented Jul 21, 2017

andralex commented Jul 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dlang-bot commented Jul 26, 2017

Bugzilla references

dlang-bot commented Jul 31, 2017

Bugzilla references

quickfur commented Jul 31, 2017

andralex commented Aug 1, 2017

quickfur commented Aug 1, 2017

wilzbach commented Feb 3, 2018

ibuclaw commented Feb 3, 2018 • edited Loading

ibuclaw commented Feb 3, 2018 •

edited

Loading