Gives error regarding sort on OpenBSD #733

greduan · Apr 18, 2015

I am on OpenBSD 5.6, just installed it yesterday.

While one Linux I never had this error but now that I'm on OpenBSD I am having it. Which makes me believe that something in the script isn't POSIX-compliant.

This is the error:

$ nvm i 0.12 
sort: fields out of order
Version '0.12' not found - try `nvm ls-remote` to browse available versions.
$ nvm ls-remote
sort: fields out of order
    iojs-v1.0.0
    iojs-v1.0.1
    iojs-v1.0.2
    iojs-v1.0.3
    iojs-v1.0.4
    iojs-v1.1.0
    iojs-v1.2.0
    iojs-v1.3.0
    iojs-v1.4.1
    iojs-v1.4.2
    iojs-v1.4.3
    iojs-v1.5.0
    iojs-v1.5.1
    iojs-v1.6.0
    iojs-v1.6.1
    iojs-v1.6.2
    iojs-v1.6.3
    iojs-v1.6.4
    iojs-v1.7.1
$

And if I do nvm i iojs.v1.7.1 I get:

$ nvm i iojs-v1.7.1
sort: fields out of order

I imagine the problem is a GNU-ism being used or something, but I'm not sure.

You can search the OpenBSD man pages here, in case it's needed: http://www.openbsd.org/cgi-bin/man.cgi?manpath=OpenBSD-5.6

Interesting - what do you get if you run each of nvm_ls_remote >/dev/null 2>&1 and nvm_ls_remote_iojs >/dev/null 2>&1?

ljharb · Apr 18, 2015

Interesting - what do you get if you run each of nvm_ls_remote >/dev/null 2>&1 and nvm_ls_remote_iojs >/dev/null 2>&1?

@ljharb

$ . /home/greduan/.nvm/nvm.sh
$ nvm_ls_remote >/dev/null 2>&1
$ nvm_ls_remote                 
sort: fields out of order
N/A
$ nvm_ls_remote_iojs >/dev/null 2>&1
$ nvm_ls_remote_iojs                  
iojs-v1.0.0
iojs-v1.0.1
iojs-v1.0.2
iojs-v1.0.3
iojs-v1.0.4
iojs-v1.1.0
iojs-v1.2.0
iojs-v1.3.0
iojs-v1.4.1
iojs-v1.4.2
iojs-v1.4.3
iojs-v1.5.0
iojs-v1.5.1
iojs-v1.6.0
iojs-v1.6.1
iojs-v1.6.2
iojs-v1.6.3
iojs-v1.6.4
iojs-v1.7.1
$

greduan · Apr 18, 2015

@ljharb

$ . /home/greduan/.nvm/nvm.sh
$ nvm_ls_remote >/dev/null 2>&1
$ nvm_ls_remote                 
sort: fields out of order
N/A
$ nvm_ls_remote_iojs >/dev/null 2>&1
$ nvm_ls_remote_iojs                  
iojs-v1.0.0
iojs-v1.0.1
iojs-v1.0.2
iojs-v1.0.3
iojs-v1.0.4
iojs-v1.1.0
iojs-v1.2.0
iojs-v1.3.0
iojs-v1.4.1
iojs-v1.4.2
iojs-v1.4.3
iojs-v1.5.0
iojs-v1.5.1
iojs-v1.6.0
iojs-v1.6.1
iojs-v1.6.2
iojs-v1.6.3
iojs-v1.6.4
iojs-v1.7.1
$

Thanks! It's interesting that the error output seems to be on stdin, not stderr. If you run echo v1.2.3 | sort -t. -u -k 1.2,1n -k 2,2n -k 3,3n do you get the same error?

ljharb · Apr 19, 2015

Thanks! It's interesting that the error output seems to be on stdin, not stderr. If you run echo v1.2.3 | sort -t. -u -k 1.2,1n -k 2,2n -k 3,3n do you get the same error?

@ljharb Yes.

greduan · Apr 19, 2015

@ljharb Yes.

Now I just have to verify that t, u, and k are all POSIX flags :-) All google gives me on the error is https://mail-index.netbsd.org/netbsd-bugs/2004/05/14/0008.html and http://gnats.netbsd.org/25572 but the latter looks like it's a BSD bug that's since been fixed (in 2006). I'll keep digging.

ljharb · Apr 19, 2015

Now I just have to verify that t, u, and k are all POSIX flags :-) All google gives me on the error is https://mail-index.netbsd.org/netbsd-bugs/2004/05/14/0008.html and http://gnats.netbsd.org/25572 but the latter looks like it's a BSD bug that's since been fixed (in 2006). I'll keep digging.

Check this, the OpenBSD man page for sort: http://www.openbsd.org/cgi-bin/man.cgi?query=sort&apropos=0&sec=0&arch=default&manpath=OpenBSD-5.6

greduan · Apr 19, 2015

Check this, the OpenBSD man page for sort: http://www.openbsd.org/cgi-bin/man.cgi?query=sort&apropos=0&sec=0&arch=default&manpath=OpenBSD-5.6

All the flags we're using are listed there

ljharb · Apr 19, 2015

All the flags we're using are listed there

Perhaps they work differently?

greduan · Apr 19, 2015

Perhaps they work differently?

Any progress on this? Also I'm not sure if this same error is present in FreeBSD, so I think it's a good idea to make it clear it's on OpenBSD with an OpenBSD tag, not a FreeBSD tag. :)

greduan · Apr 20, 2015

Any progress on this? Also I'm not sure if this same error is present in FreeBSD, so I think it's a good idea to make it clear it's on OpenBSD with an OpenBSD tag, not a FreeBSD tag. :)

I don't really have a separate tag for those :-) I'll rename the label tho for clarity.

Just to be sure there's not two things going on here - nvm_download -L -s $NVM_NODEJS_ORG_MIRROR/ -o - | \egrep -o 'v[0-9]+\.[0-9]+\.[0-9]+' does print out the unsorted node versions?

ljharb · Apr 20, 2015

I don't really have a separate tag for those :-) I'll rename the label tho for clarity.

Just to be sure there's not two things going on here - nvm_download -L -s $NVM_NODEJS_ORG_MIRROR/ -o - | \egrep -o 'v[0-9]+\.[0-9]+\.[0-9]+' does print out the unsorted node versions?

@ljharb Yes it is unsorted. I'll paste the output just in case it's useful: http://sprunge.us/IRhY

greduan · Apr 20, 2015

@ljharb Yes it is unsorted. I'll paste the output just in case it's useful: http://sprunge.us/IRhY

Just a suggestion, but perhaps it'd fix the problem to first find unique version numbers according to uniq and then sorting those with sort? It's Unixier that way, and it may just solve the problem. 😉

greduan · Apr 20, 2015

Just a suggestion, but perhaps it'd fix the problem to first find unique version numbers according to uniq and then sorting those with sort? It's Unixier that way, and it may just solve the problem. 😉

I could do that, but it doesn't seem like the issue is with the -u flag, it sounds like it's with the -k flag. If you'd like to try changing nvm.sh from | sort -t. -u -k 1.2,1n -k 2,2n -k 3,3n to | uniq | sort -t. -k 1.2,1n -k 2,2n -k 3,3n and see if that fixes it tho, I'm open to it

ljharb · Apr 20, 2015

I could do that, but it doesn't seem like the issue is with the -u flag, it sounds like it's with the -k flag. If you'd like to try changing nvm.sh from | sort -t. -u -k 1.2,1n -k 2,2n -k 3,3n to | uniq | sort -t. -k 1.2,1n -k 2,2n -k 3,3n and see if that fixes it tho, I'm open to it

Same error, sadly. Though it's still Unixier. :P

I'm thinking it's probably caused by how out of order what it has to sort is. The list goes from 10 to 12, then jumps to 5 and continues until 9, then goes to 1, briefly to 10, then continues from 1 until 6.

The reason io.js doesn't get that is because the io.js version numbers are not jumping all over the place like that. If my theory is correct.

If GNU's sort does sort this out correctly, perhaps it's necessary to report this as a bug to the OpenBSD devs? I'll look into how that's done so I can report it. :)

greduan · Apr 20, 2015

Same error, sadly. Though it's still Unixier. :P

I'm thinking it's probably caused by how out of order what it has to sort is. The list goes from 10 to 12, then jumps to 5 and continues until 9, then goes to 1, briefly to 10, then continues from 1 until 6.

The reason io.js doesn't get that is because the io.js version numbers are not jumping all over the place like that. If my theory is correct.

If GNU's sort does sort this out correctly, perhaps it's necessary to report this as a bug to the OpenBSD devs? I'll look into how that's done so I can report it. :)

This definitely works across sh, dash, bash, ksh, and zsh, and this is the first time I've ever seen this error reported - it very well might be an OpenBSD bug.

As for "unixier", I agree and had it that way initially, but for performance we try to minimize how many external calls and pipes are used :-) Also, fwiw, uniq requires the values to already be sorted - so it'd be sort … | uniq, and the -u sort flag just removes that extra step.

ljharb · Apr 20, 2015

This definitely works across sh, dash, bash, ksh, and zsh, and this is the first time I've ever seen this error reported - it very well might be an OpenBSD bug.

As for "unixier", I agree and had it that way initially, but for performance we try to minimize how many external calls and pipes are used :-) Also, fwiw, uniq requires the values to already be sorted - so it'd be sort … | uniq, and the -u sort flag just removes that extra step.

I just tried with bash but it makes no difference, I'm using ksh that's why I wanted to make sure.

This here is the source code for sort in OpenBSD. I'll see if I can find the error and find the relating piece of code to save the OpenBSD devs a bit of time. Then I'll look into reporting this.

Now I am using version 5.6-stable, not -current and in a couple weeks 5.7-stable will land. Perhaps they've already fixed this issue and I just don't have the patched version.

greduan · Apr 20, 2015

I just tried with bash but it makes no difference, I'm using ksh that's why I wanted to make sure.

This here is the source code for sort in OpenBSD. I'll see if I can find the error and find the relating piece of code to save the OpenBSD devs a bit of time. Then I'll look into reporting this.

Now I am using version 5.6-stable, not -current and in a couple weeks 5.7-stable will land. Perhaps they've already fixed this issue and I just don't have the patched version.

Let's hope that's the case! :-)

ljharb · Apr 20, 2015

Let's hope that's the case! :-)

Don't know how much this helps you all:

$ echo v1.2.3 | sort -t. -u -k 1.2,1n -k 2,2n -k 3,3n 
v1.2.3
$ uname -a
OpenBSD napalm 5.7 GENERIC.MP#938 amd64
$

That's from OpenBSD -current amd64 running a snapshot from Apr 19

sort underwent tremendous rewrite/changes in current.

@greduan those changes might not be in 5.7 unfortunately as if I recall correctly most sort changes started after 5.7 freeze was lifted.

mulander · Apr 23, 2015

Don't know how much this helps you all:

$ echo v1.2.3 | sort -t. -u -k 1.2,1n -k 2,2n -k 3,3n 
v1.2.3
$ uname -a
OpenBSD napalm 5.7 GENERIC.MP#938 amd64
$

That's from OpenBSD -current amd64 running a snapshot from Apr 19

sort underwent tremendous rewrite/changes in current.

@greduan those changes might not be in 5.7 unfortunately as if I recall correctly most sort changes started after 5.7 freeze was lifted.

Dang it, yeah I noticed there were pretty recent changes to the sort stuff, though I wasn't sure if it was in the freeze. Well I guess I'll go -current some time soon.

Thanks @mulander :)

greduan · Apr 23, 2015

Dang it, yeah I noticed there were pretty recent changes to the sort stuff, though I wasn't sure if it was in the freeze. Well I guess I'll go -current some time soon.

Thanks @mulander :)

Thanks @mulander for clearing this up! Looks like it's a bug in OpenBSD's sort implementation prior to ~~v5.7~~ v5.8.

ljharb · Apr 23, 2015

Thanks @mulander for clearing this up! Looks like it's a bug in OpenBSD's sort implementation prior to ~~v5.7~~ v5.8.

Thanks for your support @ljharb :)

greduan · Apr 23, 2015

Thanks for your support @ljharb :)

@ljharb it would be prior to v5.8 if my memory serves well. v5.7 will still contain the unchanged sort (or with less changes) so I can't confirm that v5.7 will work. OpenBSD -current is past what was released as v5.7.

Just leaving this info here in case someone stumbles upon the bug report post release.

mulander · Apr 23, 2015

@ljharb it would be prior to v5.8 if my memory serves well. v5.7 will still contain the unchanged sort (or with less changes) so I can't confirm that v5.7 will work. OpenBSD -current is past what was released as v5.7.

Just leaving this info here in case someone stumbles upon the bug report post release.

Out of curiosity, are you guys using this to build versions of node/io with any success? There are quite a few patches required to get various version running on OpenBSD.

qbit · Apr 24, 2015

Out of curiosity, are you guys using this to build versions of node/io with any success? There are quite a few patches required to get various version running on OpenBSD.

@qbit I was trying to download binaries but I couldn't do that.

greduan · Apr 24, 2015

@qbit I was trying to download binaries but I couldn't do that.

I just updated to 5.7, I can confirm the bug is still there. Although it seems -current doesn't have the problem, I'll be updating to -current soon.

greduan · May 1, 2015

I just updated to 5.7, I can confirm the bug is still there. Although it seems -current doesn't have the problem, I'll be updating to -current soon.

ljharb added OS: FreeBSD / OpenBSD bugs labels Apr 19, 2015

ljharb closed this Apr 23, 2015

qbit referenced this issue Jan 4, 2016
Closed
unix: uv_exepath should try to return a absolute path on OpenBSD #652

creationix/nvm

Join GitHub today

Gives error regarding sort on OpenBSD #733

Comments

greduan commented Apr 18, 2015

This comment has been minimized.

ljharb commented Apr 18, 2015

This comment has been minimized.

greduan commented Apr 18, 2015

This comment has been minimized.

ljharb commented Apr 19, 2015

This comment has been minimized.

greduan commented Apr 19, 2015

This comment has been minimized.

ljharb commented Apr 19, 2015

ljharb added OS: FreeBSD / OpenBSD bugs labels Apr 19, 2015

This comment has been minimized.

greduan commented Apr 19, 2015

This comment has been minimized.

ljharb commented Apr 19, 2015

This comment has been minimized.

greduan commented Apr 19, 2015

This comment has been minimized.

greduan commented Apr 20, 2015

This comment has been minimized.

ljharb commented Apr 20, 2015

This comment has been minimized.

greduan commented Apr 20, 2015

This comment has been minimized.

greduan commented Apr 20, 2015

This comment has been minimized.

ljharb commented Apr 20, 2015

This comment has been minimized.

greduan commented Apr 20, 2015

This comment has been minimized.

ljharb commented Apr 20, 2015

This comment has been minimized.

greduan commented Apr 20, 2015

This comment has been minimized.

ljharb commented Apr 20, 2015

This comment has been minimized.

mulander commented Apr 23, 2015

This comment has been minimized.

greduan commented Apr 23, 2015

This comment has been minimized.

ljharb commented Apr 23, 2015

ljharb closed this Apr 23, 2015

This comment has been minimized.

greduan commented Apr 23, 2015

This comment has been minimized.

mulander commented Apr 23, 2015

This comment has been minimized.

qbit commented Apr 24, 2015

This comment has been minimized.

greduan commented Apr 24, 2015

This comment has been minimized.

greduan commented May 1, 2015

qbit referenced this issue Jan 4, 2016