New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

package.json format for scripts and binaries #17

Open
juanibiapina opened this Issue Jul 4, 2014 · 40 comments

Comments

Projects
None yet
10 participants
@juanibiapina

juanibiapina commented Jul 4, 2014

It seems to me you based your package.json on the npm format: https://www.npmjs.org/doc/package.json.html

But by reading the documentation, the 'scripts' member is used for lifecycle management, not binaries. The 'bin' member is for listing executables that should go to the path.

Did you depart from this format on purpose?

Examples:
https://github.com/sstephenson/bats/blob/master/package.json#L7
https://github.com/progrium/gh-release/blob/master/package.json#L7

I'm planning to add this feature to basher, and it would be really sad if basher and bpkg were incompatible.

@stephenmathieson

This comment has been minimized.

Show comment
Hide comment
@stephenmathieson

stephenmathieson Jul 4, 2014

Member

npm and bpkg are already completely incompatible. npm is for JavaScript, not bash.

On Jul 4, 2014, at 9:24 AM, Juan Ibiapina notifications@github.com wrote:

It seems to me you based your package.json on the npm format: https://www.npmjs.org/doc/package.json.html

But by reading the documentation, the 'scripts' member is used for lifecycle management, not binaries. The 'bin' member is for listing executables that should go to the path.

Did you depart from this format on purpose?

Examples:
https://github.com/sstephenson/bats/blob/master/package.json#L7
https://github.com/progrium/gh-release/blob/master/package.json#L7

I'm planning to add this feature to basher, and it would be really sad if both package managers were incompatible.


Reply to this email directly or view it on GitHub.

Member

stephenmathieson commented Jul 4, 2014

npm and bpkg are already completely incompatible. npm is for JavaScript, not bash.

On Jul 4, 2014, at 9:24 AM, Juan Ibiapina notifications@github.com wrote:

It seems to me you based your package.json on the npm format: https://www.npmjs.org/doc/package.json.html

But by reading the documentation, the 'scripts' member is used for lifecycle management, not binaries. The 'bin' member is for listing executables that should go to the path.

Did you depart from this format on purpose?

Examples:
https://github.com/sstephenson/bats/blob/master/package.json#L7
https://github.com/progrium/gh-release/blob/master/package.json#L7

I'm planning to add this feature to basher, and it would be really sad if both package managers were incompatible.


Reply to this email directly or view it on GitHub.

@juanibiapina

This comment has been minimized.

Show comment
Hide comment
@juanibiapina

juanibiapina Jul 4, 2014

I edited the comment to make it clear I'm not talking about npm when I say "both package managers". It's about bpkg and basher.

Regardless, my expectation of using 'package.json' is that it at least keeps most of the known and documented logic in place, even though is not compatible with npm itself. If the intent is to diverge from the original, than the choice of the name 'package' and the format 'json' are both misleading and confusing for users.

juanibiapina commented Jul 4, 2014

I edited the comment to make it clear I'm not talking about npm when I say "both package managers". It's about bpkg and basher.

Regardless, my expectation of using 'package.json' is that it at least keeps most of the known and documented logic in place, even though is not compatible with npm itself. If the intent is to diverge from the original, than the choice of the name 'package' and the format 'json' are both misleading and confusing for users.

@jwerle

This comment has been minimized.

Show comment
Hide comment
@jwerle

jwerle Jul 4, 2014

Member

The format used is similar to component and clibs. Considering a lot of its
users are also owners/contributors it made sense to maintain the same
format. It may be early enough to change the format but a PITA
On Jul 4, 2014 10:01 AM, "Juan Ibiapina" notifications@github.com wrote:

I edited the comment to make it clear I'm not talking about npm when I say
"both package managers". It's about bpkg and basher.

Regardless, my expectation of using 'package.json' is that it at least
keeps most of the known and documented logic in place, even though is not
compatible with npm itself. If the intent is to diverge from the original,
than the choice of the name 'package' and the format 'json' are both
misleading and confusing for users.


Reply to this email directly or view it on GitHub
#17 (comment).

Member

jwerle commented Jul 4, 2014

The format used is similar to component and clibs. Considering a lot of its
users are also owners/contributors it made sense to maintain the same
format. It may be early enough to change the format but a PITA
On Jul 4, 2014 10:01 AM, "Juan Ibiapina" notifications@github.com wrote:

I edited the comment to make it clear I'm not talking about npm when I say
"both package managers". It's about bpkg and basher.

Regardless, my expectation of using 'package.json' is that it at least
keeps most of the known and documented logic in place, even though is not
compatible with npm itself. If the intent is to diverge from the original,
than the choice of the name 'package' and the format 'json' are both
misleading and confusing for users.


Reply to this email directly or view it on GitHub
#17 (comment).

@stephenmathieson

This comment has been minimized.

Show comment
Hide comment
@stephenmathieson

stephenmathieson Jul 4, 2014

Member

Ah, I didn't know what basher is. That makes more sense ;)

I think there was talk of using "bin" for executables already in order to drop the need for an install script.

On Jul 4, 2014, at 10:01 AM, Juan Ibiapina notifications@github.com wrote:

I edited the comment to make it clear I'm not talking about npm when I say "both package managers". It's about bpkg and basher.

Regardless, my expectation of using 'package.json' is that it at least keeps most of the known and documented logic in place, even though is not compatible with npm itself. If the intent is to diverge from the original, than the choice of the name 'package' and the format 'json' are both misleading and confusing for users.


Reply to this email directly or view it on GitHub.

Member

stephenmathieson commented Jul 4, 2014

Ah, I didn't know what basher is. That makes more sense ;)

I think there was talk of using "bin" for executables already in order to drop the need for an install script.

On Jul 4, 2014, at 10:01 AM, Juan Ibiapina notifications@github.com wrote:

I edited the comment to make it clear I'm not talking about npm when I say "both package managers". It's about bpkg and basher.

Regardless, my expectation of using 'package.json' is that it at least keeps most of the known and documented logic in place, even though is not compatible with npm itself. If the intent is to diverge from the original, than the choice of the name 'package' and the format 'json' are both misleading and confusing for users.


Reply to this email directly or view it on GitHub.

@juanibiapina

This comment has been minimized.

Show comment
Hide comment
@juanibiapina

juanibiapina Jul 4, 2014

@jwerle component uses component.json, which might differ from package.json. I couldn't find any examples of scripts or bin in clibs, although I looked only for a couple of minutes.

@stephenmathieson I like using bins instead of install scripts. That was one of the reasons I revived basher instead of simply using bpkg.

juanibiapina commented Jul 4, 2014

@jwerle component uses component.json, which might differ from package.json. I couldn't find any examples of scripts or bin in clibs, although I looked only for a couple of minutes.

@stephenmathieson I like using bins instead of install scripts. That was one of the reasons I revived basher instead of simply using bpkg.

@jwerle

This comment has been minimized.

Show comment
Hide comment
@jwerle

jwerle Jul 4, 2014

Member

I think bin would be so much easier too.
I'm sorry, I should have noted clibs uses src, not scripts.
I think it is early enough to make changes

On Fri, Jul 4, 2014 at 10:35 AM, Juan Ibiapina notifications@github.com
wrote:

@jwerle https://github.com/jwerle component uses component.json, which
might differ from package.json. I couldn't find any examples of scripts or
bin in clibs, although I looked only for a couple of minutes.

@stephenmathieson https://github.com/stephenmathieson I like using bins
instead of install scripts. That was one of the reasons I revived basher
instead of simply using bpkg.


Reply to this email directly or view it on GitHub
#17 (comment).

Member

jwerle commented Jul 4, 2014

I think bin would be so much easier too.
I'm sorry, I should have noted clibs uses src, not scripts.
I think it is early enough to make changes

On Fri, Jul 4, 2014 at 10:35 AM, Juan Ibiapina notifications@github.com
wrote:

@jwerle https://github.com/jwerle component uses component.json, which
might differ from package.json. I couldn't find any examples of scripts or
bin in clibs, although I looked only for a couple of minutes.

@stephenmathieson https://github.com/stephenmathieson I like using bins
instead of install scripts. That was one of the reasons I revived basher
instead of simply using bpkg.


Reply to this email directly or view it on GitHub
#17 (comment).

@stephenmathieson

This comment has been minimized.

Show comment
Hide comment
@stephenmathieson

stephenmathieson Jul 4, 2014

Member

fwiw, if there wasn't already a small community of people using clib, I'd move from package.json to clib.yml. I've personally run into issues using clib to manage C deps for native node modules. also, json is ugly and hard to read :/

On Jul 4, 2014, at 10:45 AM, Joseph Werle notifications@github.com wrote:

I think bin would be so much easier too.
I'm sorry, I should have noted clibs uses src, not scripts.
I think it is early enough to make changes

On Fri, Jul 4, 2014 at 10:35 AM, Juan Ibiapina notifications@github.com
wrote:

@jwerle https://github.com/jwerle component uses component.json, which
might differ from package.json. I couldn't find any examples of scripts or
bin in clibs, although I looked only for a couple of minutes.

@stephenmathieson https://github.com/stephenmathieson I like using bins
instead of install scripts. That was one of the reasons I revived basher
instead of simply using bpkg.


Reply to this email directly or view it on GitHub
#17 (comment).


Reply to this email directly or view it on GitHub.

Member

stephenmathieson commented Jul 4, 2014

fwiw, if there wasn't already a small community of people using clib, I'd move from package.json to clib.yml. I've personally run into issues using clib to manage C deps for native node modules. also, json is ugly and hard to read :/

On Jul 4, 2014, at 10:45 AM, Joseph Werle notifications@github.com wrote:

I think bin would be so much easier too.
I'm sorry, I should have noted clibs uses src, not scripts.
I think it is early enough to make changes

On Fri, Jul 4, 2014 at 10:35 AM, Juan Ibiapina notifications@github.com
wrote:

@jwerle https://github.com/jwerle component uses component.json, which
might differ from package.json. I couldn't find any examples of scripts or
bin in clibs, although I looked only for a couple of minutes.

@stephenmathieson https://github.com/stephenmathieson I like using bins
instead of install scripts. That was one of the reasons I revived basher
instead of simply using bpkg.


Reply to this email directly or view it on GitHub
#17 (comment).


Reply to this email directly or view it on GitHub.

@juanibiapina

This comment has been minimized.

Show comment
Hide comment
@juanibiapina

juanibiapina Jul 4, 2014

On the same thought, would you think bpkg could use a different format too? For basher I'm considering a package.bash file. I haven't completely decided, but I'll get to it as soon as I have some time.

I love the json parser, and it all looks beautiful, but a native bash format would be both simpler to implement and less confusing. Possible format (making this up on the spot):

NAME="name"
VERSION="1.2.3"
DESCRIPTION="Best package ever"
GLOBAL=true
BIN=bin/exec:otherexec

Let me know what you think.

juanibiapina commented Jul 4, 2014

On the same thought, would you think bpkg could use a different format too? For basher I'm considering a package.bash file. I haven't completely decided, but I'll get to it as soon as I have some time.

I love the json parser, and it all looks beautiful, but a native bash format would be both simpler to implement and less confusing. Possible format (making this up on the spot):

NAME="name"
VERSION="1.2.3"
DESCRIPTION="Best package ever"
GLOBAL=true
BIN=bin/exec:otherexec

Let me know what you think.

@stephenmathieson

This comment has been minimized.

Show comment
Hide comment
@stephenmathieson

stephenmathieson Jul 4, 2014

Member

oh wow - that makes a ton of sense. then we'd just source the package
file, right?

On Fri, Jul 4, 2014 at 12:48 PM, Juan Ibiapina notifications@github.com
wrote:

On the same thought, would you think bpkg could use a different format
too? For basher I'm considering a package.bash file. I haven't completely
decided, but I'll get to it as soon as I have some time.

I love the json parser, and it all looks beautiful, but a native bash
format would be both simpler to implement and less confusing. Possible
format (making this up on the spot):

NAME="name"
VERSION="1.2.3"
DESCRIPTION="Best package ever"
GLOBAL=true
BIN=bin/exec:otherexec

Let me know what you think.


Reply to this email directly or view it on GitHub
#17 (comment).

Member

stephenmathieson commented Jul 4, 2014

oh wow - that makes a ton of sense. then we'd just source the package
file, right?

On Fri, Jul 4, 2014 at 12:48 PM, Juan Ibiapina notifications@github.com
wrote:

On the same thought, would you think bpkg could use a different format
too? For basher I'm considering a package.bash file. I haven't completely
decided, but I'll get to it as soon as I have some time.

I love the json parser, and it all looks beautiful, but a native bash
format would be both simpler to implement and less confusing. Possible
format (making this up on the spot):

NAME="name"
VERSION="1.2.3"
DESCRIPTION="Best package ever"
GLOBAL=true
BIN=bin/exec:otherexec

Let me know what you think.


Reply to this email directly or view it on GitHub
#17 (comment).

@juanibiapina

This comment has been minimized.

Show comment
Hide comment
@juanibiapina

juanibiapina Jul 4, 2014

Indeed, but I haven't tested it yet.

juanibiapina commented Jul 4, 2014

Indeed, but I haven't tested it yet.

@jwerle

This comment has been minimized.

Show comment
Hide comment
@jwerle

jwerle Jul 4, 2014

Member

Yeah, that looks much better!

On Fri, Jul 4, 2014 at 12:53 PM, Juan Ibiapina notifications@github.com
wrote:

Indeed, but I haven't tested it yet.


Reply to this email directly or view it on GitHub
#17 (comment).

Member

jwerle commented Jul 4, 2014

Yeah, that looks much better!

On Fri, Jul 4, 2014 at 12:53 PM, Juan Ibiapina notifications@github.com
wrote:

Indeed, but I haven't tested it yet.


Reply to this email directly or view it on GitHub
#17 (comment).

@juanibiapina

This comment has been minimized.

Show comment
Hide comment
@juanibiapina

juanibiapina Jul 4, 2014

Might be a slight security problem though.

http://wiki.bash-hackers.org/howto/conffile

juanibiapina commented Jul 4, 2014

Might be a slight security problem though.

http://wiki.bash-hackers.org/howto/conffile

@juanibiapina

This comment has been minimized.

Show comment
Hide comment
@juanibiapina

juanibiapina Jul 5, 2014

My initial version seems to work well: https://github.com/juanibiapina/basher/blob/master/libexec/basher-install#L25

I'm only using the BIN variable, with a list of binaries separated by ":". Not using NAME or other things for now.

Let me know what you think. If you decide to keep package.json, in whatever format, I might be able to make basher support both package.json and package.sh.

juanibiapina commented Jul 5, 2014

My initial version seems to work well: https://github.com/juanibiapina/basher/blob/master/libexec/basher-install#L25

I'm only using the BIN variable, with a list of binaries separated by ":". Not using NAME or other things for now.

Let me know what you think. If you decide to keep package.json, in whatever format, I might be able to make basher support both package.json and package.sh.

@juanibiapina

This comment has been minimized.

Show comment
Hide comment
@juanibiapina

juanibiapina commented Jul 10, 2014

hi

@jwerle

This comment has been minimized.

Show comment
Hide comment
@jwerle

jwerle Jul 10, 2014

Member

I like the idea of a bash script. How do you feel about an ini file? There is a ini parser written in bash as well here

Member

jwerle commented Jul 10, 2014

I like the idea of a bash script. How do you feel about an ini file? There is a ini parser written in bash as well here

@juanibiapina

This comment has been minimized.

Show comment
Hide comment
@juanibiapina

juanibiapina Jul 10, 2014

I think it's simple enough and much safer. The syntax is also very similar to bash.

I don't know how to choose here.

juanibiapina commented Jul 10, 2014

I think it's simple enough and much safer. The syntax is also very similar to bash.

I don't know how to choose here.

@jwerle

This comment has been minimized.

Show comment
Hide comment
@jwerle

jwerle Jul 10, 2014

Member

I'm all for being safer! It is also MUCH EASIER than json! :D

On Thu, Jul 10, 2014 at 2:24 PM, Juan Ibiapina notifications@github.com
wrote:

I think it's simple enough and much safer. The syntax is also very similar
to bash.

I don't know how to choose here.


Reply to this email directly or view it on GitHub
#17 (comment).

Member

jwerle commented Jul 10, 2014

I'm all for being safer! It is also MUCH EASIER than json! :D

On Thu, Jul 10, 2014 at 2:24 PM, Juan Ibiapina notifications@github.com
wrote:

I think it's simple enough and much safer. The syntax is also very similar
to bash.

I don't know how to choose here.


Reply to this email directly or view it on GitHub
#17 (comment).

@juanibiapina

This comment has been minimized.

Show comment
Hide comment
@juanibiapina

juanibiapina Jul 10, 2014

On the other hand, the ini parser is almost 300 lines with many features we might not need (or are you planning or using the sections?), and no direct support for arrays.

This is interesting: https://github.com/rudimeier/bash_ini_parser/blob/master/read_ini.sh#L271

We might be able to use a simple bash format, but just parse the lines with "=" and eval with the sanitized value.

Or we could even drop eval. Use an if statement to select either BIN= or NAME= and assign to the correct variable. The list of attributes is fixed anyway.

Just throwing ideas here. The ini parser is fine xD

juanibiapina commented Jul 10, 2014

On the other hand, the ini parser is almost 300 lines with many features we might not need (or are you planning or using the sections?), and no direct support for arrays.

This is interesting: https://github.com/rudimeier/bash_ini_parser/blob/master/read_ini.sh#L271

We might be able to use a simple bash format, but just parse the lines with "=" and eval with the sanitized value.

Or we could even drop eval. Use an if statement to select either BIN= or NAME= and assign to the correct variable. The list of attributes is fixed anyway.

Just throwing ideas here. The ini parser is fine xD

@alganet

This comment has been minimized.

Show comment
Hide comment
@alganet

alganet Oct 13, 2014

What about using a function?

package () {
    name    "myapplication"
    version "0.1.1"
    global   true
}

No need for manual parsing as well, information could be retrieved simply calling the package function:

# Define valid properties
name    () ( pkg_name="$1" )
version () ( pkg_version="$1" )
global  () ( pkg_is_global="$1" )

# Load package information
. package.sh && package

echo "$pkg_name $pkg_version $pkg_is_global"

Security problems can be be solved in two ways:

  • Running the package.sh in another restricted bash instance (empty PATH, -r option).
  • Shallow parsing the file to look for problems. I already have a very minimalistic parser (portable, sed, line based, support for double quotes) that could be potentially reused if anyone is interested.

alganet commented Oct 13, 2014

What about using a function?

package () {
    name    "myapplication"
    version "0.1.1"
    global   true
}

No need for manual parsing as well, information could be retrieved simply calling the package function:

# Define valid properties
name    () ( pkg_name="$1" )
version () ( pkg_version="$1" )
global  () ( pkg_is_global="$1" )

# Load package information
. package.sh && package

echo "$pkg_name $pkg_version $pkg_is_global"

Security problems can be be solved in two ways:

  • Running the package.sh in another restricted bash instance (empty PATH, -r option).
  • Shallow parsing the file to look for problems. I already have a very minimalistic parser (portable, sed, line based, support for double quotes) that could be potentially reused if anyone is interested.
@ccarpita

This comment has been minimized.

Show comment
Hide comment
@ccarpita

ccarpita Oct 14, 2014

Member

@alganet like the idea, though the security considerations are tough to resolve w/ restricted instance (can you export vars?) and shallow parsing can be worked around. Not sure we can guarantee security anyway, given that packages can be installed and executed without any checking. It would be nice to validate that all the required fields are called, and you get a nice clear error when the syntax breaks, both of which your package() proposal cover.

Regarding npm-compatible bin support, this is actually something I've been working on and will hopefully submit a PR soon.

Member

ccarpita commented Oct 14, 2014

@alganet like the idea, though the security considerations are tough to resolve w/ restricted instance (can you export vars?) and shallow parsing can be worked around. Not sure we can guarantee security anyway, given that packages can be installed and executed without any checking. It would be nice to validate that all the required fields are called, and you get a nice clear error when the syntax breaks, both of which your package() proposal cover.

Regarding npm-compatible bin support, this is actually something I've been working on and will hopefully submit a PR soon.

@alganet

This comment has been minimized.

Show comment
Hide comment
@alganet

alganet Oct 14, 2014

@ccarpita I guess the shallow parser is OK. It's still very limited, but it is token based, so it should be safe.

I've published a prototype with some examples:

  • A single .sed script generated on the fly (it can be cached) by a portable shell script.
  • Deal with blacklists for things like eval (easy to to turn into whitelist as well).
  • Deal with syntax errors.
  • Stops before invalid instructions and always produce a valid shell script (that displays the error, so it can be sourced in a clean way).
  • Very limited syntax. Only double quotes, no escapes, no multi-lines commands or multi line strings.

Many platforms use internal DSLs like this, although I'm not sure if they shallow parse before (we could!). One popular example is the Vagrantfile (which reuses the Ruby syntax in a configuration form).

alganet commented Oct 14, 2014

@ccarpita I guess the shallow parser is OK. It's still very limited, but it is token based, so it should be safe.

I've published a prototype with some examples:

  • A single .sed script generated on the fly (it can be cached) by a portable shell script.
  • Deal with blacklists for things like eval (easy to to turn into whitelist as well).
  • Deal with syntax errors.
  • Stops before invalid instructions and always produce a valid shell script (that displays the error, so it can be sourced in a clean way).
  • Very limited syntax. Only double quotes, no escapes, no multi-lines commands or multi line strings.

Many platforms use internal DSLs like this, although I'm not sure if they shallow parse before (we could!). One popular example is the Vagrantfile (which reuses the Ruby syntax in a configuration form).

@fidian

This comment has been minimized.

Show comment
Hide comment
@fidian

fidian Dec 3, 2015

It's been a sad year with no updates. Is there any movement on this issue that I have not seen? It looks like people are behind a new format and are just deciding what the new format would look like. I would love to see package.json renamed so people don't automatically think the repository is for node.

I'm in favor of the sanitized config files as linked earlier. Also there was talk about INI files. I have an INI file parser in mr-fusion. It still uses eval, but the values are sanitizes. Look at parse-ini, ini-var and encode-string and I'd welcome any critiques to improve the library.

I'm debating splitting up my existing bash code into reusable libraries and I think it would be wonderful to combine forces. I'm a fan of bpkg because I have to install just a single file and there's the online repository. I like basher because it is tested.

fidian commented Dec 3, 2015

It's been a sad year with no updates. Is there any movement on this issue that I have not seen? It looks like people are behind a new format and are just deciding what the new format would look like. I would love to see package.json renamed so people don't automatically think the repository is for node.

I'm in favor of the sanitized config files as linked earlier. Also there was talk about INI files. I have an INI file parser in mr-fusion. It still uses eval, but the values are sanitizes. Look at parse-ini, ini-var and encode-string and I'd welcome any critiques to improve the library.

I'm debating splitting up my existing bash code into reusable libraries and I think it would be wonderful to combine forces. I'm a fan of bpkg because I have to install just a single file and there's the online repository. I like basher because it is tested.

@jwerle

This comment has been minimized.

Show comment
Hide comment
@jwerle

jwerle Dec 3, 2015

Member

@fidian any suggestions on a new format? This would of course invalidate every package out there using package.json.

package.json isn't just owned by npm. It is used by other package management systems like clib and duo.

cc @stephenmathieson @ccarpita

Member

jwerle commented Dec 3, 2015

@fidian any suggestions on a new format? This would of course invalidate every package out there using package.json.

package.json isn't just owned by npm. It is used by other package management systems like clib and duo.

cc @stephenmathieson @ccarpita

@stephenmathieson

This comment has been minimized.

Show comment
Hide comment
@stephenmathieson

stephenmathieson Dec 3, 2015

Member

i stand by #17 (comment) still :)

Member

stephenmathieson commented Dec 3, 2015

i stand by #17 (comment) still :)

@jwerle

This comment has been minimized.

Show comment
Hide comment
@jwerle
Member

jwerle commented Dec 3, 2015

@fidian

This comment has been minimized.

Show comment
Hide comment
@fidian

fidian Dec 3, 2015

@stephenmathieson I am only talking about things I suffered through, not specifically the format of the file. Please consider scripts that set up repositories in a generic way. Here's a simplified example:

git clone http://github.com/whatever/whereever.git
(
    cd wherever
    git submodule update --init --recursive
    [[ -f composer.json ]] && composer install
    [[ -f bower.json]] && bower install
    [[ -f Makefile ]] && make
    [[ -f Gruntfile ]] && grunt
    [[ -f requirements.txt ]] && pip install -f requirements.txt
    [[ -f package.json ]] && npm install
    # How do we detect that bpkg is supposed to run instead or in addition to npm?
)
echo "Project checked out and dependencies installed"

With the popularity of node and npm, I would see people googling for package.json and hitting node instead of this package. Those unfortunate souls would be confused. Also, we now can't have both node modules and bpkg modules in the same project because the package.json files conflict with each other.

I'd advocate simply using bpkg.json and falling back to package.json when bpkg.json does not exit. Later on this format can be changed to be an INI file or a shell script or whatnot.

If appropriate, I can make a separate issue or a pull request and essentially split this issue into two (one for rename, a second for the format). I strongly support a new format and eliminating JSON, but that can come as a later step. For now I just want to have both npm and bpkg coexist.

fidian commented Dec 3, 2015

@stephenmathieson I am only talking about things I suffered through, not specifically the format of the file. Please consider scripts that set up repositories in a generic way. Here's a simplified example:

git clone http://github.com/whatever/whereever.git
(
    cd wherever
    git submodule update --init --recursive
    [[ -f composer.json ]] && composer install
    [[ -f bower.json]] && bower install
    [[ -f Makefile ]] && make
    [[ -f Gruntfile ]] && grunt
    [[ -f requirements.txt ]] && pip install -f requirements.txt
    [[ -f package.json ]] && npm install
    # How do we detect that bpkg is supposed to run instead or in addition to npm?
)
echo "Project checked out and dependencies installed"

With the popularity of node and npm, I would see people googling for package.json and hitting node instead of this package. Those unfortunate souls would be confused. Also, we now can't have both node modules and bpkg modules in the same project because the package.json files conflict with each other.

I'd advocate simply using bpkg.json and falling back to package.json when bpkg.json does not exit. Later on this format can be changed to be an INI file or a shell script or whatnot.

If appropriate, I can make a separate issue or a pull request and essentially split this issue into two (one for rename, a second for the format). I strongly support a new format and eliminating JSON, but that can come as a later step. For now I just want to have both npm and bpkg coexist.

@ccarpita

This comment has been minimized.

Show comment
Hide comment
@ccarpita

ccarpita Dec 3, 2015

Member

I've personally found package.json to be a point of confusion, and would prefer to see something along the lines of bpkg.json as bpkg "owns" the formatting. Maybe a crazy idea, but if we also required a dependency on jq then using json as a standard configuration format wouldn't be so bad.

Member

ccarpita commented Dec 3, 2015

I've personally found package.json to be a point of confusion, and would prefer to see something along the lines of bpkg.json as bpkg "owns" the formatting. Maybe a crazy idea, but if we also required a dependency on jq then using json as a standard configuration format wouldn't be so bad.

@isaacs

This comment has been minimized.

Show comment
Hide comment
@isaacs

isaacs Jan 31, 2016

So... I get that bpkg is for bash, and npm is for JavaScript.

Funny story. I have this bash program, which I'm currently putting in npm already (since it's primarily used by Node users), and I'd love to also put it in bpkg, since this is a bash thing.

Also, since bash programs are frequently going to be CLI utilities, and npm is frequently used for the same, it's not so weird an idea that things might exist in both places. (See also overlap between npm and rubygems with system package managers like apt-get and yum.) There's already a bpkg package that installs Node, not so strange that someone might use that to have a node-and-bash hybrid type thing. Quite a lot of Node users use bash pretty regularly, after all.

I think using a well-established filename convention like package.json, while making needlessly incompatible choices in how this file is interpreted, is not the best approach. Especially when there's already a well-established way to express exactly what bpkg's scripts field is being used for, and literally infinite ways to express this that don't conflict with the many years-long legacy of extant npm packages.

I'd suggest inventing a new json filename which isn't already in use, or using package.json in ways that don't conflict.

isaacs commented Jan 31, 2016

So... I get that bpkg is for bash, and npm is for JavaScript.

Funny story. I have this bash program, which I'm currently putting in npm already (since it's primarily used by Node users), and I'd love to also put it in bpkg, since this is a bash thing.

Also, since bash programs are frequently going to be CLI utilities, and npm is frequently used for the same, it's not so weird an idea that things might exist in both places. (See also overlap between npm and rubygems with system package managers like apt-get and yum.) There's already a bpkg package that installs Node, not so strange that someone might use that to have a node-and-bash hybrid type thing. Quite a lot of Node users use bash pretty regularly, after all.

I think using a well-established filename convention like package.json, while making needlessly incompatible choices in how this file is interpreted, is not the best approach. Especially when there's already a well-established way to express exactly what bpkg's scripts field is being used for, and literally infinite ways to express this that don't conflict with the many years-long legacy of extant npm packages.

I'd suggest inventing a new json filename which isn't already in use, or using package.json in ways that don't conflict.

@jwerle

This comment has been minimized.

Show comment
Hide comment
@jwerle

jwerle Feb 1, 2016

Member

@isaacs I agree. package.json was a poor choice now that several bpkg packages exist this creates confusion for people trying to differentiate bpkg from npm. Any suggestions for a new filename?

Member

jwerle commented Feb 1, 2016

@isaacs I agree. package.json was a poor choice now that several bpkg packages exist this creates confusion for people trying to differentiate bpkg from npm. Any suggestions for a new filename?

@isaacs

This comment has been minimized.

Show comment
Hide comment
@isaacs

isaacs Feb 1, 2016

I'd suggest a two-fold approach:

  1. If bpkg.json is present, use that.
  2. Else, if package.json is present, use that, but handle scripts/bin/etc. using npm's semantics.

Using npm's json semantics is probably a good idea for (1) anyway, if possible, because you take advantage of a lot of documentation and expectations, without having to explain (or invent!) the rationale for the differences.

Like the example with nave, people are already using npm for some bash packages. May as well use what works :)

isaacs commented Feb 1, 2016

I'd suggest a two-fold approach:

  1. If bpkg.json is present, use that.
  2. Else, if package.json is present, use that, but handle scripts/bin/etc. using npm's semantics.

Using npm's json semantics is probably a good idea for (1) anyway, if possible, because you take advantage of a lot of documentation and expectations, without having to explain (or invent!) the rationale for the differences.

Like the example with nave, people are already using npm for some bash packages. May as well use what works :)

@jwerle

This comment has been minimized.

Show comment
Hide comment
@jwerle

jwerle Feb 1, 2016

Member

@isaacs I like this approach. It ensures backwards compatibility with existing bpkg packages that are using the package.json file at the very least. I'l try to have a patch up soon if everyone agrees. cc @bpkg/owners @stephenmathieson @juanibiapina

Member

jwerle commented Feb 1, 2016

@isaacs I like this approach. It ensures backwards compatibility with existing bpkg packages that are using the package.json file at the very least. I'l try to have a patch up soon if everyone agrees. cc @bpkg/owners @stephenmathieson @juanibiapina

@brock

This comment has been minimized.

Show comment
Hide comment
@brock

brock Feb 1, 2016

Member

👍 for bpkg.json

Member

brock commented Feb 1, 2016

👍 for bpkg.json

@juanibiapina

This comment has been minimized.

Show comment
Hide comment
@juanibiapina

juanibiapina Feb 1, 2016

Cool!
On Feb 1, 2016 8:13 PM, "Brock Angelo" notifications@github.com wrote:

[image: 👍] for bpkg.json


Reply to this email directly or view it on GitHub
#17 (comment).

juanibiapina commented Feb 1, 2016

Cool!
On Feb 1, 2016 8:13 PM, "Brock Angelo" notifications@github.com wrote:

[image: 👍] for bpkg.json


Reply to this email directly or view it on GitHub
#17 (comment).

@jasonkarns

This comment has been minimized.

Show comment
Hide comment
@jasonkarns

jasonkarns Feb 15, 2016

Just dropping in to point out that this is coming to a head soon.

https://github.com/ztombol/bats-core
and https://github.com/ztombol/bats-assert

We (nodenv org) will likely soon be using bats-core and bats-assert for our bats test assertions. nodenv is a bash utility, but for managing node versions. being a bash program, our test suite is in bats. However, we use npm for managing nodenv (and its associated plugin repos). (for versioning, releases, tagging, scripting, dependencies etc)

I can't speak for @ztombol, but bats-core and bats-assert may very well have an npm-compatible package.json. And I suspect that bats-core and bats-assert would love to support bpkg as well. Personally, I think package.json works great as a manifest, since the majority of npm's manifest needs are shared with bpkg. It would only require that bpkg leverage npm's semantics for the package.json fields. I think this is already possible, given the current functionality. (just leveraging bin and files instead of scripts and leveraging scripts as hook scripts or run-scripts)

I don't see a reason to duplicate package.json's metadata into a bpkg.json file, especially if the semantics are identical, only with different keynames. This would be a pain to keep in sync, for any libs that need both. (see also, annoyance and pain in dealing with duplicate manifests between component.json, bower.json and package.json, when most/all the data therein was identical)

TL;DR +1 to keeping package.json, but with npm's semantics

jasonkarns commented Feb 15, 2016

Just dropping in to point out that this is coming to a head soon.

https://github.com/ztombol/bats-core
and https://github.com/ztombol/bats-assert

We (nodenv org) will likely soon be using bats-core and bats-assert for our bats test assertions. nodenv is a bash utility, but for managing node versions. being a bash program, our test suite is in bats. However, we use npm for managing nodenv (and its associated plugin repos). (for versioning, releases, tagging, scripting, dependencies etc)

I can't speak for @ztombol, but bats-core and bats-assert may very well have an npm-compatible package.json. And I suspect that bats-core and bats-assert would love to support bpkg as well. Personally, I think package.json works great as a manifest, since the majority of npm's manifest needs are shared with bpkg. It would only require that bpkg leverage npm's semantics for the package.json fields. I think this is already possible, given the current functionality. (just leveraging bin and files instead of scripts and leveraging scripts as hook scripts or run-scripts)

I don't see a reason to duplicate package.json's metadata into a bpkg.json file, especially if the semantics are identical, only with different keynames. This would be a pain to keep in sync, for any libs that need both. (see also, annoyance and pain in dealing with duplicate manifests between component.json, bower.json and package.json, when most/all the data therein was identical)

TL;DR +1 to keeping package.json, but with npm's semantics

@jasonkarns jasonkarns referenced this issue Feb 15, 2016

Merged

Support npm #2

@isaacs

This comment has been minimized.

Show comment
Hide comment
@isaacs

isaacs Feb 16, 2016

I agree with @jasonkarns. Essentially the same use-case as nave, which I mentioned initially: #17 (comment)

There is quite a lot of overlap between bash and node programmers, and bash and node programs. Node is a command line utility, and used for writing command line utilities, and bash is the unix command line. Any divergence from established patterns should be done only when there is a very significant benefit to be had, because that divergence is very costly. I don't see the benefit in this case.

isaacs commented Feb 16, 2016

I agree with @jasonkarns. Essentially the same use-case as nave, which I mentioned initially: #17 (comment)

There is quite a lot of overlap between bash and node programmers, and bash and node programs. Node is a command line utility, and used for writing command line utilities, and bash is the unix command line. Any divergence from established patterns should be done only when there is a very significant benefit to be had, because that divergence is very costly. I don't see the benefit in this case.

@jwerle

This comment has been minimized.

Show comment
Hide comment
@jwerle

jwerle Feb 16, 2016

Member

@isaacs @jasonkarns Okay, I think with about a year and a half of discussion on this subject, we should move towards implementing the package.json specification as much as possible. Global command utilities should be defined with .bin and .scripts =>.files` is an obvious change.

Member

jwerle commented Feb 16, 2016

@isaacs @jasonkarns Okay, I think with about a year and a half of discussion on this subject, we should move towards implementing the package.json specification as much as possible. Global command utilities should be defined with .bin and .scripts =>.files` is an obvious change.

@ztombol

This comment has been minimized.

Show comment
Hide comment
@ztombol

ztombol Feb 16, 2016

Hi! Developer of bats-core and bats-assert here. I'm not familiar with bpkg, basher or npm, but I would love to merge support for them. @juanibiapina has already started working on basher support, and @jasonkarns on npm support.

If these package managers could share a common manifest specification that would make everyone's life easier. Developers could easily support 3 different package managers, and users would have a wider selection of packages available for their favourite manager.

Like I've said, I'm not familiar with the intricacies of using a shared pacakge manifest (from this discussion it seems possible), but it seems that everybody would benefit.

ztombol commented Feb 16, 2016

Hi! Developer of bats-core and bats-assert here. I'm not familiar with bpkg, basher or npm, but I would love to merge support for them. @juanibiapina has already started working on basher support, and @jasonkarns on npm support.

If these package managers could share a common manifest specification that would make everyone's life easier. Developers could easily support 3 different package managers, and users would have a wider selection of packages available for their favourite manager.

Like I've said, I'm not familiar with the intricacies of using a shared pacakge manifest (from this discussion it seems possible), but it seems that everybody would benefit.

@fidian

This comment has been minimized.

Show comment
Hide comment
@fidian

fidian Feb 17, 2016

I have a concern that npm-specific properties do not overlap with bpkg-specific properties. For instance, npm uses "dependencies" like so:

{
    "dependencies": {
        "some-npm-module": "version"
    }
}

I would not want bpkg to reuse "dependencies" in their spec. Doing so would pollute the npm's list of dependencies and "npm install" would fail. Instead, the bpkg-specific dependencies should use their own property.

Here's a bad example so you can see what I mean. With this, now neither tool can fully complete their task of installing dependencies.

{
    "dependencies": {
        "some-npm-module": "version",
        "some-bpkg-dependency": "version"
    }
}

Further, one should not conflict with any other tool that has a tolerably significant market share that may overlap with bpkg. npm is one such tool, but there are also Mozilla add-ons and jspm's extensions to the package.json spec.

I see two options. You can define your own filename and stick to it, thus you own the spec. Alternately you can try to settle in nicely with package.json by carefully examining what others are doing to extend that file in their own way and try to ensure you don't conflict.

fidian commented Feb 17, 2016

I have a concern that npm-specific properties do not overlap with bpkg-specific properties. For instance, npm uses "dependencies" like so:

{
    "dependencies": {
        "some-npm-module": "version"
    }
}

I would not want bpkg to reuse "dependencies" in their spec. Doing so would pollute the npm's list of dependencies and "npm install" would fail. Instead, the bpkg-specific dependencies should use their own property.

Here's a bad example so you can see what I mean. With this, now neither tool can fully complete their task of installing dependencies.

{
    "dependencies": {
        "some-npm-module": "version",
        "some-bpkg-dependency": "version"
    }
}

Further, one should not conflict with any other tool that has a tolerably significant market share that may overlap with bpkg. npm is one such tool, but there are also Mozilla add-ons and jspm's extensions to the package.json spec.

I see two options. You can define your own filename and stick to it, thus you own the spec. Alternately you can try to settle in nicely with package.json by carefully examining what others are doing to extend that file in their own way and try to ensure you don't conflict.

@juanibiapina

This comment has been minimized.

Show comment
Hide comment
@juanibiapina

juanibiapina Feb 17, 2016

Just to reiterate my opinion, since it has been a while:

I think having two package managers for different languages use the same file name is not a good idea. Imagine the problems we could have if rubygems also used package.json. It would be very difficult to know which entries are used by which tool and how.

Basher and bpkg, on the other hand, could easily share a package format. But if they don't, it shouldn't be a major concern anyway, since they tackle the problem in very different ways.

Since it has been a while, I've already adopted package.sh for basher, and I think it has worked well. If you decide to use bpkg.json, basher could easily use it as a fallback.

juanibiapina commented Feb 17, 2016

Just to reiterate my opinion, since it has been a while:

I think having two package managers for different languages use the same file name is not a good idea. Imagine the problems we could have if rubygems also used package.json. It would be very difficult to know which entries are used by which tool and how.

Basher and bpkg, on the other hand, could easily share a package format. But if they don't, it shouldn't be a major concern anyway, since they tackle the problem in very different ways.

Since it has been a while, I've already adopted package.sh for basher, and I think it has worked well. If you decide to use bpkg.json, basher could easily use it as a fallback.

@ccarpita

This comment has been minimized.

Show comment
Hide comment
@ccarpita

ccarpita Feb 17, 2016

Member

If possible, I would retain as many semantics as possible from npm conventions, and still use a distinct file named bpkg.json, as it explicitly indicates support for bpkg implementations while avoiding the need for per-field conflict resolution. It just seems like unnecessary surface area for issues in the future to attempt to maintain partial compatibility with the evolving npm standards.

Member

ccarpita commented Feb 17, 2016

If possible, I would retain as many semantics as possible from npm conventions, and still use a distinct file named bpkg.json, as it explicitly indicates support for bpkg implementations while avoiding the need for per-field conflict resolution. It just seems like unnecessary surface area for issues in the future to attempt to maintain partial compatibility with the evolving npm standards.

@jasonkarns jasonkarns referenced this issue Jun 11, 2018

Merged

Publish to npm #100

0 of 5 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment