Adding parallel processing of many files. #57

genuinelucifer · 2016-03-17T19:58:54Z

Currently it works fine if one file is passed. But if I pass more than one file then i get only extern(C): in the output.
I'm trying to find the mistake.
I would be thankful for any suggestion(s) regarding approach and/or the coding style (since it is my first PR to this project, I might have missed many coding styles standard for this projects)...

genuinelucifer · 2016-03-18T18:20:16Z

Most probably due to arguments (the way it is defined and used) it is getting a bit out of hand currently. Will try to do the necessary changes to make it work.

genuinelucifer · 2016-03-18T18:25:43Z

@jacob-carlborg Although I am unable to find exactly where arguments is defined. So, that I could make copies of it and pass to every thread, its own local copy. Is it in one of the libraries used?

jacob-carlborg · 2016-03-18T19:11:25Z

dstep/driver/Application.d

-        Compiler compiler;
+	    bool helpFlag;
+	Language language;
+	string[] argsToRestore;


Please use spaces (four) for indentation.

Sorry. Mixed up tabs and spaces. Will correct it.

jacob-carlborg · 2016-03-18T19:56:58Z

arguments is defined here: https://github.com/jacob-carlborg/dstack/blob/master/dstack/application/Application.d#L40-L48

jacob-carlborg · 2016-03-18T20:01:23Z

dstep/driver/Application.d

-            inputFiles ~= arguments.argument.input;
-            startConversion(inputFiles.first);
+	    int j = 0;
+	    if(remainingArgs.length >0)


Always put a space before the opening parenthesis in statements.

Sorry. Will do.

genuinelucifer · 2016-03-18T20:19:38Z

I'll look into dstack and try to figure out how to implement this.

genuinelucifer · 2016-03-19T20:58:35Z

@jacob-carlborg Please review. It is working fine for me now.

jacob-carlborg · 2016-03-20T21:15:24Z

Most things I commented on are not fixed as far as I can see.

jacob-carlborg · 2016-03-20T21:15:56Z

dstep/driver/Application.d

        if (arguments.argument.input.hasValue)
        {
-            inputFiles ~= arguments.argument.input;
-            startConversion(inputFiles.first);
+            int j = 0;


I don't thinks this is used.

Sorry, had used it in previous code, forgot to remove...

jacob-carlborg · 2016-03-20T21:27:11Z

I'm wondering if it's better that the -o flag specifies which directory to place all the resulting files in when there are multiple input files.

genuinelucifer · 2016-03-21T07:57:10Z

Sure seems better if -o tells the directory in case of multiple files.

genuinelucifer · 2016-03-21T11:17:25Z

I did the changes. Also, please tell me how much to keep the max value for Arguments class in mambo. For my usage, I kept it to 512 and worked. But, I don't know how to decide the exact amount to be kept in mambo.
.
Also, I tried to use parallel again. This time the problem is that, if I give 2 input files. The first output file is fine. But the second output file contains the combined output from 1st AND 2nd file. While using this version of task works as required. I'll try to come up with a solution to using parallel though if it is required.

genuinelucifer · 2016-03-21T11:21:41Z

And one of the buildbots is passing and other one is failing. I saw the output but can't figure out what actually is the error that is shown. Please help.

jacob-carlborg · 2016-03-21T14:12:00Z

And one of the buildbots is passing and other one is failing. I saw the output but can't figure out what actually is the error that is shown. Please help.

The error messages are not very good. But the issue is that DStep fails to produce a result in the case of features/objc/method.feature, for some reason.

jacob-carlborg · 2016-03-21T14:15:15Z

dstep/driver/Application.d

        }

        else
-            startConversion("");
+        {
+            auto conversionTask = task!startParsingFile(language, argsToRestore, "", false, arguments, compiler, compilerArgs);


I don't think it's necessary to use a task for this condition.

genuinelucifer · 2016-03-21T18:10:05Z

Again one build is passing, other one is not. The tests are failing on clang 3.7 and the best part is I have clang 3.7 only... How do I run these feature tests on my machine? Or what could be the probable cause for this failure?

jacob-carlborg · 2016-03-21T21:03:53Z

One building is running on Linux and the other one on OS X. The one on OS X is running additional tests, converting Objective-C to D. It's one of those that is failing.

You either need OS X to run those test or install GNUStep on Linux. I think you can run basically the same tests by running dub test, @ciechowoj recently added that capability. That might give you better error messages.

jacob-carlborg · 2016-03-21T21:08:20Z

Also, please tell me how much to keep the max value for Arguments class in mambo. For my usage, I kept it to 512 and worked. But, I don't know how to decide the exact amount to be kept in mambo

That's a good question. You can always use int.max and it will most likely be limited by something else 😃. Otherwise, what's the maximum number of files one could expect there to be in a C library.

ciechowoj · 2016-03-21T21:46:14Z

Again one build is passing, other one is not. The tests are failing on clang 3.7 and the best part is I have clang 3.7 only... How do I run these feature tests on my machine? Or what could be the probable cause for this failure?

If you are on linux try to install GNUStep and comment if OSX in features/step_definitions/common.rb.

You either need OS X to run those test or install GNUStep on Linux. I think you can run basically the same tests by running dub test, @ciechowoj recently added that capability. That might give you better error messages.

The unit tests seems to pass.

Testing with libclang version 3.6.0
Running unit tests
Running cucumber tests


int unitTest ()
{
    println("Running unit tests ");

    auto result = executeShell("dub test");

    if (result.status != 0)
        println(result.output);

    return result.status;
}

genuinelucifer · 2016-04-16T15:44:45Z

@jacob-carlborg I have rebased the PR. Please consider merging if it's acceptable now.

jacob-carlborg · 2016-04-16T15:47:43Z

👌

jacob-carlborg · 2016-04-16T15:53:02Z

Actually, a couple of minor things:

Could you please change the commit message. The feature is not parallel processing of input files, but rather multiple input files in the first place. I don't think "Enforcing" sounds good in this case
Update the changelog
Update the help text/usage information to somehow indicate that multiple input files are supported. Something like <input files...>

genuinelucifer · 2016-04-16T16:03:28Z

Is it okay now?

jacob-carlborg · 2016-04-16T16:06:21Z

With the "help text" i meant the help text at the bottom of Application.d that is printed when the --help flag is given. But it was good that you update the readme as well.

genuinelucifer · 2016-04-16T16:11:29Z

With the "help text" i meant the help text at the bottom of Application.d that is printed when the --help flag is given. But it was good that you update the readme as well.

I saw that after pushing the changes. Is it complete now? I changed some text in the readme as well.

jacob-carlborg · 2016-04-16T16:13:50Z

Looking good now, I'll wait with merging until the tests are complete, just to be sure.

jacob-carlborg · 2016-04-16T16:22:40Z

Thanks.

genuinelucifer · 2016-04-16T17:21:04Z

Although, i would take a couple of days break for a few other things. But, it would be great if you could point me to some other issue/feature that I could work on next.

jacob-carlborg · 2016-04-19T18:33:57Z

@genuinelucifer I think a natural extension to this PR would be to be able to pass a directory to DStep, which scans the directory for files to translate.

genuinelucifer · 2016-04-19T18:37:42Z

scans the directory for files to translate.

Ok. So a '-i <directory>` to check for all the '.h' files and then translate them? Or without a new flag. Simply check if inout file name is indeed a directory and if yes then translate all files in it. In this case if a file also exists with same name then that could be given priority. Which one would you prefer?

jacob-carlborg · 2016-04-19T18:42:00Z

Which one would you prefer?

The second approach. It's not possible to have a directory and a file with the same name.

genuinelucifer · 2016-04-19T18:45:31Z

Oops. I thought it might be possible in linux. Never tried.
On Apr 20, 2016 12:12 AM, "jacob-carlborg" notifications@github.com wrote:

Which one would you prefer?

The second approach. It's not possible to have a directory and a file with
the same name.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#57 (comment)

jacob-carlborg · 2016-04-19T18:47:22Z

An alternative approach could be to allow this: dstep foo/*, and for recursively scan a directory: dstep foo/**/*. The first should already be handled by the shell but I don't think the second is handled by the shell. Also, the shell does no globbing on Windows.

genuinelucifer · 2016-04-19T18:52:13Z

the shell does no globbing on Windows.

So, i should hard code the recursive search for both cases? Because i could
never make dstep work on Windows and hence i won't be able to test it. And
build bots are for mac and linux.
I guess it shouldn't be too hard. I'll try and submit a preliminary PR
probably within 1-2 days.

jacob-carlborg · 2016-04-19T18:57:14Z

Never mind the globbing, the shell already does that.

genuinelucifer · 2016-04-19T18:58:33Z

Oh great. :)

jacob-carlborg · 2016-04-19T19:00:25Z

But I still think it's convenient to be able to pass a whole directory to translate. That should work on Windows as well, if we ever get DStep to work on Windows.

genuinelucifer · 2016-04-19T19:02:36Z

Hmm. Right. Currently we only translate header files right? No source files.

jacob-carlborg · 2016-04-19T19:03:02Z

Yes, only header files.

jacob-carlborg · 2016-04-19T19:04:21Z

I think we can assume .h for all files. If the user wants to translate some other extension they can still use globbing or similar.

genuinelucifer · 2016-04-19T19:07:37Z

Oh hmm. Great.

jacob-carlborg · 2016-04-19T19:12:02Z

I noticed one thing that might be a problem with the current implementation of multiple files. If I have a file structure looking like this:

foo
├── bar
│   └── bar.h
└── foo
    └── foo.h

And run DStep like dstep foo/bar/bar.h foo/foo/foo.h -o bar then the resulting file structure will look like this:

bar
├── bar.d
└── foo.d

Looks like all files are placed in the top level of the target directory. That will cause problems if bar and foo in the source file structure have files with the same names. Then these files will overwrite each other in the target directory.

genuinelucifer · 2016-04-19T19:17:12Z

Oh hmm. But isn't it that the user should avoid? Translating files with same name at the same time? I mean even if we copy the directory structure then too it could be dificult (or undesired) in some situations. Probably a new flag for this?

jacob-carlborg · 2016-04-19T19:25:12Z

Oh hmm. But isn't it that the user should avoid? Translating files with same name at the same time?

I think it's a valid use case when using globbing and what I suggested as a new feature today, to handle a whole directory as input argument.

I mean even if we copy the directory structure then too it could be dificult (or undesired) in some situations.

I think it's undesired for a single input file. But for multiple input files, I'm not sure. Maybe a new flag for this.

jacob-carlborg reviewed Mar 18, 2016
View reviewed changes

genuinelucifer force-pushed the parallel branch from 26e55e5 to edf6779 Compare March 19, 2016 20:56

jacob-carlborg reviewed Mar 20, 2016
View reviewed changes

jacob-carlborg reviewed Mar 21, 2016
View reviewed changes

genuinelucifer force-pushed the parallel branch from 19c256a to b82d53f Compare April 16, 2016 16:03

Add support for processing of multiple input files

27aa465

genuinelucifer force-pushed the parallel branch from b82d53f to 27aa465 Compare April 16, 2016 16:10

jacob-carlborg merged commit dd3a5e8 into jacob-carlborg:master Apr 16, 2016

genuinelucifer deleted the parallel branch April 18, 2016 19:30

Adding parallel processing of many files. #57

Adding parallel processing of many files. #57

Conversation

genuinelucifer commented Mar 17, 2016

genuinelucifer commented Mar 18, 2016

genuinelucifer commented Mar 18, 2016

jacob-carlborg Mar 18, 2016

Choose a reason for hiding this comment

genuinelucifer Mar 18, 2016 via email

Choose a reason for hiding this comment

jacob-carlborg commented Mar 18, 2016

jacob-carlborg Mar 18, 2016

Choose a reason for hiding this comment

genuinelucifer Mar 18, 2016

Choose a reason for hiding this comment

genuinelucifer commented Mar 18, 2016

genuinelucifer commented Mar 19, 2016

jacob-carlborg commented Mar 20, 2016

jacob-carlborg Mar 20, 2016

Choose a reason for hiding this comment

genuinelucifer Mar 21, 2016

Choose a reason for hiding this comment

jacob-carlborg commented Mar 20, 2016

genuinelucifer commented Mar 21, 2016

genuinelucifer commented Mar 21, 2016

genuinelucifer commented Mar 21, 2016

jacob-carlborg commented Mar 21, 2016

jacob-carlborg Mar 21, 2016

Choose a reason for hiding this comment

genuinelucifer commented Mar 21, 2016

jacob-carlborg commented Mar 21, 2016

jacob-carlborg commented Mar 21, 2016

ciechowoj commented Mar 21, 2016

genuinelucifer commented Apr 16, 2016

jacob-carlborg commented Apr 16, 2016 • edited

jacob-carlborg commented Apr 16, 2016 • edited

genuinelucifer commented Apr 16, 2016 via email

jacob-carlborg commented Apr 16, 2016

genuinelucifer commented Apr 16, 2016 via email

jacob-carlborg commented Apr 16, 2016

jacob-carlborg commented Apr 16, 2016

genuinelucifer commented Apr 16, 2016 via email

jacob-carlborg commented Apr 19, 2016

genuinelucifer commented Apr 19, 2016 via email

jacob-carlborg commented Apr 19, 2016

genuinelucifer commented Apr 19, 2016

jacob-carlborg commented Apr 19, 2016

genuinelucifer commented Apr 19, 2016

jacob-carlborg commented Apr 19, 2016

genuinelucifer commented Apr 19, 2016 via email

jacob-carlborg commented Apr 19, 2016

genuinelucifer commented Apr 19, 2016 via email

jacob-carlborg commented Apr 19, 2016

jacob-carlborg commented Apr 19, 2016

genuinelucifer commented Apr 19, 2016 via email

jacob-carlborg commented Apr 19, 2016

genuinelucifer commented Apr 19, 2016 via email

jacob-carlborg commented Apr 19, 2016

jacob-carlborg commented Apr 16, 2016 •

edited

jacob-carlborg commented Apr 16, 2016 •

edited