New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchmark: add --format csv option #7961

Closed
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
7 participants
@adrian-nitu-92
Contributor

adrian-nitu-92 commented Aug 3, 2016

Checklist
  • make -j4 test (UNIX), or vcbuild test nosign (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines
Affected core subsystem(s)

benchmark

Description of change

Added the option of using --format csv when outputting data.

@AndreasMadsen

This comment has been minimized.

Member

AndreasMadsen commented Aug 3, 2016

I don't think we want to add extra arguments to each individual benchmark. That is a lot of extra complexity in parsing and output and it is not needed since benchmark/run.js or benchmark/scatter.js can be used to get the same result. Modifying only benchmark/run.js should be sufficient, I'm sorry if that wasn't clear.

@jasnell

This comment has been minimized.

Member

jasnell commented Aug 3, 2016

+1 to what @AndreasMadsen is saying here. I think there's an easier way to do this.

@adrian-nitu-92

This comment has been minimized.

Contributor

adrian-nitu-92 commented Aug 4, 2016

Fair point :), reduced the change set.
I added a secondary commit that fixes some typos and adds a set of missing {}.

@addaleax

View changes

benchmark/run.js Outdated
`, {
arrayArgs: ['set']
});
const benchmarks = cli.benchmarks();
const validFormats = ['csv', 'default'];

This comment has been minimized.

@addaleax

addaleax Aug 4, 2016

Member

Should this be 'simple'?

This comment has been minimized.

@adrian-nitu-92

adrian-nitu-92 Aug 4, 2016

Contributor

It could also be removed :))), but I wanted to add something to not let 'csv' alone :D. Anything should be good, it is an option I doubt would be used. Should I make the change?

This comment has been minimized.

@addaleax

addaleax Aug 4, 2016

Member

Huh? I meant, in the usage text you added --format as simple/csv, but this array doesn’t seem to match that. Sorry if that’s been unclear (or I’m just misunderstanding this in general?)

This comment has been minimized.

@adrian-nitu-92

adrian-nitu-92 Aug 4, 2016

Contributor

Oh, yea, my bad, thank you for spotting the inconsistency :D

@targos

View changes

benchmark/run.js Outdated
@@ -10,16 +10,26 @@ const cli = CLI(`usage: ./node run.js [options] [--] <category> ...
--filter pattern string to filter benchmark scripts
--set variable=value set benchmark variable (can be repeated)
--format simple/csv optional value that specifies the output format

This comment has been minimized.

@targos

targos Aug 4, 2016

Member

I'd write it as --format [simple|csv]

@AndreasMadsen

View changes

benchmark/run.js Outdated
if (benchmarks.length === 0) {
console.error('no benchmarks found');
process.exit(1);
}
if (cli.optional.format) {

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 4, 2016

Member

Use console.error and the return is not required (like the benchmark.length test above). Exit codes should only be non negative (code >= 0). You can also shorten it a bit.

const format = cli.optional.format || 'simple';
if (!['csv', 'simple'].includes(format)) {
  console.error('Invalid format detected');
  process.exit(1);
}

This comment has been minimized.

@targos

targos Aug 4, 2016

Member

Isn't it better practice to use process.exitCode = 1 and then just return?

This comment has been minimized.

@adrian-nitu-92

adrian-nitu-92 Aug 4, 2016

Contributor

I would prefer keeping the variable validFormats :D.

@AndreasMadsen

View changes

benchmark/run.js Outdated
for (const key of Object.keys(data.conf)) {
conf += ' ' + key + '=' + JSON.stringify(data.conf[key]);
conf += confseparator + key + '=' + JSON.stringify(data.conf[key]);

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 4, 2016

Member

This assumes that each benchmark has the same amount of parameters. You should do as in compare.js#L71 and just have conf being one field (remember to escape ").

@AndreasMadsen

View changes

benchmark/run.js Outdated
}
console.log(`${data.name}${conf}: ${data.rate}`);
console.log(`${data.name}${conf}${separator}${data.rate}`);

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 4, 2016

Member

Could you add data.time to the csv output and surround data.name with " just to be safe. I would suggest:

if (format === 'csv') {
  // Escape qoutes (") for correct csv formatting
  conf = conf.replace(/"/g, '""');
  console.log(`"${data.name}", "${conf}", ${data.rate}, ${data.time}`);
} else {
  console.log(`${data.name} ${conf}: ${data.rate}`);
}
@AndreasMadsen

This comment has been minimized.

Member

AndreasMadsen commented Aug 4, 2016

@adrian-nitu-92 Yes, this is much better. I've sugested some changes, you will also have to:

  • remove console.log(filename); in the csv case.
  • add csv header: console.log('"filename", "configuration", "rate", "time"'); should do.

edit: could you @intel.com email to your GitHub account. I don't know if we require the commits to link to your user account, but it would be nice.

@adrian-nitu-92

This comment has been minimized.

Contributor

adrian-nitu-92 commented Aug 4, 2016

Updated the PR with all changes, except removing the variable validFormats :)

@AndreasMadsen

View changes

benchmark/run.js Outdated
(function recursive(i) {
const filename = benchmarks[i];
const child = fork(path.resolve(__dirname, filename), cli.optional.set);
console.log();
console.log(filename);
if (format == 'csv') {
console.log('"filename","configuration","rate","time"');

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 4, 2016

Member

This is going to output a csv header for every file. A csv header should only appear once.

@AndreasMadsen

View changes

benchmark/run.js Outdated
return;
}
(function recursive(i) {
const filename = benchmarks[i];
const child = fork(path.resolve(__dirname, filename), cli.optional.set);
console.log();

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 4, 2016

Member

Empty lines should be included in csv.

This comment has been minimized.

@adrian-nitu-92

adrian-nitu-92 Aug 5, 2016

Contributor

Empty lines are already included in output regardless of format (simple/csv). Could you please elaborate?

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 5, 2016

Member

Sorry, misspelled. They should not be included in the csv output.

@AndreasMadsen

View changes

benchmark/run.js Outdated
// delete first space of the configuration
conf = conf.slice(1);
if (format == 'csv') {
conf = conf.replace(/"/g, '""');

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 4, 2016

Member

nit: you didn't like the comment :) Few people know that " is escaped as "", I think a comment would is useful.

@AndreasMadsen

View changes

benchmark/run.js Outdated
console.error('no benchmarks found');
process.exit(1);
console.error('No benchmarks found');
// node documentation suggests using this exit method when you need to still

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 4, 2016

Member

nit: you are absolutely right. But I don't think it is necessary to say it explicit in a comment.

@AndreasMadsen

View changes

benchmark/run.js Outdated
`, {
arrayArgs: ['set']
});
const benchmarks = cli.benchmarks();
const validFormats = ['csv', 'simple'];

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 4, 2016

Member

nit: maybe move the line down to where it is used.

@adrian-nitu-92

This comment has been minimized.

Contributor

adrian-nitu-92 commented Aug 5, 2016

Updated PR :)

@AndreasMadsen

This comment has been minimized.

Member

AndreasMadsen commented Aug 5, 2016

Great. Be sure to run make lint, I did see at least one lint error.

@AndreasMadsen

This comment has been minimized.

@AndreasMadsen

View changes

benchmark/run.js Outdated
return;
}
if (format == 'csv') {

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 6, 2016

Member

@Trott Does this really match our lint rules? The linter says it does. https://ci.nodejs.org/job/node-test-linter/3613/console

This comment has been minimized.

@Trott

Trott Aug 6, 2016

Member

Yes, as far as I can tell, that is consistent with our lint rules. (I would prefer this be === rather than == but that's not lint-enforced.)

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 6, 2016

Member

So would I. @adrian-nitu-92 could you fix this, I think there is another one too.

@AndreasMadsen

This comment has been minimized.

Member

AndreasMadsen commented Aug 6, 2016

/cc @nodejs/benchmarking @addaleax @mscdex who participated in the original issue. I need to run this to be sure (can't right now), but besides a "lint" nit this looks good.

Trott added a commit to Trott/io.js that referenced this pull request Aug 6, 2016

@Trott Trott referenced this pull request Aug 6, 2016

Closed

benchmark: favor === over == #8000

2 of 2 tasks complete
@adrian-nitu-92

This comment has been minimized.

Contributor

adrian-nitu-92 commented Aug 8, 2016

Ou, I get it now :D. Made the requested changes :)

@AndreasMadsen

View changes

benchmark/run.js Outdated
@@ -10,31 +10,54 @@ const cli = CLI(`usage: ./node run.js [options] [--] <category> ...
--filter pattern string to filter benchmark scripts
--set variable=value set benchmark variable (can be repeated)
--format [simple|csv] optional value that specifies the output format

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 8, 2016

Member

Could you align the descriptions. ``optional => optional

@AndreasMadsen

View changes

benchmark/run.js Outdated
}
if (format === 'csv') {
console.log('"filename","configuration","rate","time"');

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 8, 2016

Member

nit: the other files has a space in between , => ,``

It makes it slightly more readable

@AndreasMadsen

View changes

benchmark/run.js Outdated
if (format === 'csv') {
// Escape quotes (") for correct csv formatting
conf = conf.replace(/"/g, '""');
console.log(`"${data.name}","${conf}",${data.rate},${data.time}`);

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 8, 2016

Member

nit: the other files has a space in between , => ,``

@AndreasMadsen

This comment has been minimized.

Member

AndreasMadsen commented Aug 8, 2016

@adrian-nitu-92 awesome work. I have tried it out and it looks good.

As I'm not a very experienced node.js reviewer, thus I will give this 48 hours before merging, such other contributors have time to come with last remarks.

@jasnell

This comment has been minimized.

Member

jasnell commented Aug 8, 2016

LGTM

console.log(`${data.name}${conf}: ${data.rate}`);
// delete first space of the configuration
conf = conf.slice(1);

This comment has been minimized.

@adrian-nitu-92

adrian-nitu-92 Aug 9, 2016

Contributor

This line can be removed if I reformat 57&59. I like this version better because I don't think you should rely on spaces in strings you are concatenating.

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 9, 2016

Member

Agreed, just keep it.

Trott added a commit to Trott/io.js that referenced this pull request Aug 9, 2016

benchmark: favor === over ==
Refs: nodejs#7961 (comment)
PR-URL: nodejs#8000
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: Andreas Madsen <amwebdk@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>

cjihrig added a commit that referenced this pull request Aug 10, 2016

benchmark: favor === over ==
Refs: #7961 (comment)
PR-URL: #8000
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: Andreas Madsen <amwebdk@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
@AndreasMadsen

This comment has been minimized.

Member

AndreasMadsen commented Aug 11, 2016

LGTM

CI: https://ci.nodejs.org/job/node-test-pull-request/3621/

will merge, when ci is green.

@AndreasMadsen

View changes

benchmark/run.js Outdated
const validFormats = ['csv', 'simple'];
const format = cli.optional.format || 'simple';
if (!validFormats.includes(cli.optional.format)) {

This comment has been minimized.

@AndreasMadsen

AndreasMadsen Aug 11, 2016

Member

should be !validFormats.includes(format) otherwise it will print Invalid format detected when there are no --format argument.

@AndreasMadsen

This comment has been minimized.

Member

AndreasMadsen commented Aug 11, 2016

I found a bug, I will merge when that gets fixed.

adrian-nitu-92 added some commits Aug 3, 2016

benchmark: add --format csv option
Added the option of using --format csv when outputting data.

Signed-off-by: Adrian Nitu <adrian.nitu@intel.com>
benchmark: fix comment typos and code format
I noticed some typos and the lack of {} following an if.

Signed-off-by: Adrian Nitu <adrian.nitu@intel.com>
benchmark: update compare.js exit method
Node documentation recommends using process.exitCode = x and returning
as a way to exit.

Signed-off-by: Adrian Nitu <adrian.nitu@intel.com>
@adrian-nitu-92

This comment has been minimized.

Contributor

adrian-nitu-92 commented Aug 12, 2016

Updated the PR and the branch merging point ( pulled nodejs latest and rebased on top of it, no conflicts detected). Also ran the whole tests again just to make sure I don't have additional slip-ups :D

Thanks Andreas for your help :D

@AndreasMadsen

This comment has been minimized.

Member

AndreasMadsen commented Aug 12, 2016

@gibfahn gibfahn referenced this pull request Jun 15, 2017

Closed

Auditing for 6.11.1 #230

2 of 3 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment