Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maybe need a package management tool #47

Closed
hstarorg opened this issue May 31, 2018 · 94 comments
Closed

Maybe need a package management tool #47

hstarorg opened this issue May 31, 2018 · 94 comments

Comments

@hstarorg
Copy link

Import from url is very hard to use. I think is not a good idea.
If so,

  1. We must manage more js files not our own development.
  2. Heavily dependent on the network, because the third package also can import another packages.

Like C#\Java, the nuget and maven is good for use. Maybe use that mode can help us manage the packages.

@qti3e
Copy link
Contributor

qti3e commented May 31, 2018

@hstarorg I'm against having a package manager.

I think other things including npm are more network depended, here is why:

  1. in NPM you must have a connection to easily initing your project.
  2. you need a network to run commands such as npm install, yarn add

Why (i think) importing from URLs is a good idea?

Deno uses a built-in cache system so you don't need to download a bunch of things. (that you already downloaded for your other projects)

We must manage more js files not our own development.

You already do that in NPM!

and it's easier in Deno as you only have to import one file per dependency wherever you need that : )

@sandangel
Copy link

@qti3e will the code autocompletion or IDE relate stuff work with that?

@mohsen1
Copy link
Contributor

mohsen1 commented May 31, 2018

package name in package.json is a shortcut to an actual URL anyways.

@qti3e
Copy link
Contributor

qti3e commented May 31, 2018

@sandangel not yet, but it's not a hard task, there are only two differences between a Deno file and a regular ts file:

  1. remote URL import
  2. top-level await

@sandangel
Copy link

@qti3e I'm curious, so what is the strategy to reuse thousands of js, ts library out there when they are using normal require() // js and import // ts

@qti3e
Copy link
Contributor

qti3e commented May 31, 2018

@sandangel well, I'll dive into this issue next week and see if I can write a Node to Deno converter or not : )

(FYI; Probably I'm gonna parse AST and insert file extensions to module names, also I'm thinking of converting npm imports (package name) to unpkg urls)

If you have any other idea please let me know : )

@ry
Copy link
Member

ry commented May 31, 2018

@sandangel The idea is to explore what a simpler runtime could be. It may indeed to too tedious to link to URLs (but Go seems to do fine by it). I think the project looks like it's is going well I will write a source rewriting tool for converting old programs.
For now, no package manager needed or desired.

@ry ry closed this as completed May 31, 2018
@sandangel
Copy link

sandangel commented May 31, 2018

@ry and the tool you have just said works just like a package manager ^^. It downloads the package, converts it so it can be import with url, adds the import to the source code. Am I wrong? :)

@wujohns
Copy link

wujohns commented May 31, 2018

Version control for package is important. Something like go-dep

@TehShrike
Copy link

@wujohns Bear in mind that with the third-party service unpkg, you can use URLs to depend on npm packages with the same version ranges that you use in your package.json file, ala https://unpkg.com/deepmerge@~1.4.2 -> https://unpkg.com/deepmerge@1.4.4/dist/cjs.js

@sandangel
Copy link

@TehShrike but having a cli tool just makes life easier ^^

@hstarorg
Copy link
Author

hstarorg commented Jun 1, 2018

How to process multilayer deps? For example: A depend B, B depend C?

@oridark
Copy link

oridark commented Jun 1, 2018

i think a package manage tool like npm is needed, because we are lazy, we want to make it easy to use, writing urls is really not a happy things to everyone. you want everyone use it, you should make them happy to use itk.

@yorkie
Copy link
Contributor

yorkie commented Jun 1, 2018

Deno is browser-compatible, so any packager(bower, unpkg and npm) are available, too.

@balupton
Copy link
Contributor

balupton commented Jun 7, 2018

i think a package manage tool like npm is needed, because we are lazy, we want to make it easy to use, writing urls is really not a happy things to everyone. you want everyone use it, you should make them happy to use itk.

problem of finding and typing URLs can be solved via vscode plugins, or a siri plugin - no need why that complexity should be in something which goal is to make things lighter

@cemremengu
Copy link

Even if you dont have a package manager there should be a central place to list dependencies.

Are we going to write the absolute url in each file? How does it work?

@wujohns
Copy link

wujohns commented Jun 7, 2018

after the version of npm3, the “black hole” is flattened

@kitsonk
Copy link
Contributor

kitsonk commented Jun 7, 2018

Are we going to write the absolute url in each file? How does it work?

Ryan's talk makes it explicit that a package server would handle semantic versioning as part of the URL.

after the version of npm3, the “black hole” is flattened

It just looks different, but in practice, it isn't very "flat".

@ry
Copy link
Member

ry commented Jun 7, 2018

The idea with the URL imports is that deno acts as its own package manager. I explicitly want to avoid the need for ancillary tooling.

@jedahan
Copy link

jedahan commented Jun 7, 2018

I love the idea of url imports because it makes it easier for us to experiment with ideas like ipfs://<contentHash>, dat://<similarWeirdAddress>, ssb://%<yepthisworkstoo>, and ssb-git://, etc.

@hstarorg
Copy link
Author

hstarorg commented Jun 7, 2018

@jedahan Use this style, your code should be very hard to share.

@jedahan
Copy link

jedahan commented Jun 7, 2018

@hstarorg hard to share in that not everyone will have a system-level resolver, I agree. But there are so many good solutions for this, and what I like is that it seems deno is making decisions that don't explicitly stop us from trying any of them:

  • have go/rust/c++ resolve the protocol itself directly (built in fetch that makes it easier to share without installing anything)
  • passthrough to system-level resolver (hard to share if you don't have something to handle those protocols, so I agree with you)
  • somthing else in the future we haven't thought of?

@wujohns
Copy link

wujohns commented Jun 7, 2018

When using go to program, the biggest problem I encountered is package manage.

I am afraid to use a language without official package manage tool. ex: go, it has many tools for
managing package because of the long time missing of official package manage tool(dep improve the current situation)

So I think the result of the “import from url” will be that many new tools are created for connecting npm and dyno. Maybe this result is good for most of the people

@hstarorg
Copy link
Author

hstarorg commented Jun 8, 2018

@wujohns Many options that will difficult to choose. Split package management will cause the package in many tools.

@wujohns
Copy link

wujohns commented Jun 8, 2018

@hstarorg So the official tool or central repository is important. Fortunately we have npm, the new tools's core will be it (there is actually only one choice)

@hstarorg
Copy link
Author

hstarorg commented Jun 9, 2018

@wujohns A npm like tool is very important, also need fix some npm defect.

@cojack
Copy link

cojack commented Jan 10, 2019

Guys, without any package manager, we will end up using plenty of lodash source in one single application, so instead of using 4mb of lodash, we're going to hack up the memory up to 40mb because of 10 different packages, use almost the same lodash version, but bundled so every import will take it also.

This is ridiculous.

@balupton
Copy link
Contributor

balupton commented Jan 10, 2019

@cojack that is what tree shaking is for

@randfur
Copy link

randfur commented May 18, 2020

People saying deno needs a package manager like NPM... what if you just use NPM as the package manager??
import {dog} from '../node_modules/dog/index.ts';

@Soremwar
Copy link
Contributor

Soremwar commented May 18, 2020

@randfur You could if you want to, as long as the module uses JavaScript and not CommonJS for it's resolution system( it's the beauty of Deno )

A package manager can't be implemented cause it would limit the resources from which a JS module can be imported from(requires a metadata file, a standard to validate versions, a min runtime version, etc)

According to ECMA standards, any valid JS file is a module of sorts, and Deno's team seems to be of this same mind

@dummyuser10
Copy link

The main issue I see with the URL-based imports is that it's not possible to upgrade the versions of a given package project-wide, at least not without doing a find-and-replace or via some utility.

Co-locate your dependencies in a single file and re-export them. (e.g. https://github.com/oakserver/oak/blob/master/deps.ts/)

One of my favorite things about package.json was that it gave you a way to easily view all dependencies used in a project/module when browsing/auditing code. Something like a deps.ts file isn't really as good as package.json IMO because it is left to best intentions. There is nothing to stop people from not using a deps.ts file or to have a URL import somewhere within their code even if they do have a deps.ts file. So looking at a deps.ts file isn't really a guarantee that there aren't other deps used elsewhere in the project.

just my 2 cents, but I personally wish Deno had created its own standard/blessed way of managing deps rather than a best-intentions recommendation of putting all your deps in a single file.

@cknight
Copy link
Contributor

cknight commented May 19, 2020

@dummyuser10 check out deno info mymodule.ts, for example, which will print all dependencies of your module, regardless of where they are kept.

@matiaslopezd
Copy link

matiaslopezd commented May 20, 2020

I'm still think this way to import packages will be really insecure. Security is first! I mean, over 80% of devs today copy and paste the classic npm i package-name and never check the source code. That could be end on malwares, virus, troyan, etc. in your dev environment. Let's imagine all services that now provide analytics and builds of projects, executing and reading malicious code!

Other way to understand this, is with the known express package. For example, if I want import to Deno:

import { express } from 'https://malicious-cdn.com/libs/express/express.ts'

All it's ok, run perfect and offer the lastest express package... But the package have malicious code that recognize the dev or production environment and can log all HTTP request undetected by developers.

This is 100% possible with NPM, but we know that npm can remove it in Deno not. I see the difference between both easily, but how I can trust in that CDN?

  1. npm i express-2
  2. https://malicious-cdn.com/libs/express/4.17.1/express.ts

I know that Deno have other philosophy of security over runtime, but obviously if a package need write and read permissions the developers will give them.

What plans have about this? Is the cost of decentralization?

Packages over "Deno Proxy"

Could be a optional global package register, but not store the files? Maybe like a proxy for deno packages because with that flow Deno could give some security to developers and block malicious packages.

For example:

import { express } from 'https://malicious-cdn.com/libs/express/4.17.1/express.ts';

I'm still using malicious package but when start app, Deno proxy check the URL if is not in the blacklist packages.

I say optional because can be good idea turn off "Deno proxy" if the developer don't want use it, like with a privates repositories. Also could integrate with a list of trusted domains will not pass to the Deno proxy trusted-domains.json.

@matiaslopezd
Copy link

Maybe my last comment could transform in a business model 😆

@matiaslopezd
Copy link

Also what can be happen with the most used packages when they don't have enough backers for host files and support all traffic with the URL model?. I mean think in lodash, tslib, axios, express, moment, etc. with the current developers demand.

I see the natural movement of companies to offer freemium/paid host to open source packages, that could end in a centralizated host.

I don't say will be a bad idea use URL but the developers need a secure model to import packages with the accessibility and security like fundamentals.

@mlogan
Copy link

mlogan commented May 20, 2020

The main issue I see with the URL-based imports is that it's not possible to upgrade the versions of a given package project-wide, at least not without doing a find-and-replace or via some utility.

Co-locate your dependencies in a single file and re-export them. (e.g. https://github.com/oakserver/oak/blob/master/deps.ts/)

This misunderstands the issue.

In Node, if I depend on A, and A depends on B, and B releases a critical security patch, I can apply that patch immediately by running a single command. I don't have to wait for A to release a new version. And if, as is often the case, I depend on packages A through Z, all of which depend on B, I don't have to wait for dozens of different package maintainers to release new versions before I can be rid of the vulnerable version of B.

As I understand it, in deno, whether you plumb imports through deps.ts or not, an import statement always refers to an exact package version. Therefore the only way to receive updates to indirect dependencies is to first wait for updates to your direct dependencies to be released.

Other languages do things this way, and for all I know it's better that way. But, the existing module culture of node ("make lots of packages, depend on loooooots of packages") will not be able to exist in deno due to this issue.

@crisnao2
Copy link

I'm still think this way to import packages will be really insecure. Security is first! I mean, over 80% of devs today copy and paste the classic npm i package-name and never check the source code. That could be end on malwares, virus, troyan, etc. in your dev environment. Let's imagine all services that now provide analytics and builds of projects, executing and reading malicious code!

Other way to understand this, is with the known express package. For example, if I want import to Deno:

import { express } from 'https://malicious-cdn.com/libs/express/express.ts'

All it's ok, run perfect and offer the lastest express package... But the package have malicious code that recognize the dev or production environment and can log all HTTP request undetected by developers.

This is 100% possible with NPM, but we know that npm can remove it in Deno not. I see the difference between both easily, but how I can trust in that CDN?

  1. npm i express-2
  2. https://malicious-cdn.com/libs/express/4.17.1/express.ts

I know that Deno have other philosophy of security over runtime, but obviously if a package need write and read permissions the developers will give them.

What plans have about this? Is the cost of decentralization?

Packages over "Deno Proxy"

Could be a optional global package register, but not store the files? Maybe like a proxy for deno packages because with that flow Deno could give some security to developers and block malicious packages.

For example:

import { express } from 'https://malicious-cdn.com/libs/express/4.17.1/express.ts';

I'm still using malicious package but when start app, Deno proxy check the URL if is not in the blacklist packages.

I say optional because can be good idea turn off "Deno proxy" if the developer don't want use it, like with a privates repositories. Also could integrate with a list of trusted domains will not pass to the Deno proxy trusted-domains.json.

The idea of a proxy seems interesting, I will even travel a bit, maybe it could even be maintained by the NPM itself, something like, I want such a package, it would do something like import {module} from "npm.org?package=namespace/module.ts&package_version=1.&platform=deno&npm_package_version=", there at NPM they would get the package, all its subdependencies, package and deliver everything only in a single module file, from the moment he assembled the package the first time , it would already be cache for all the others, to keep updated, they could generate a new package every time there is an update in the main module or in its dependencies and create a versioning system for the packaged modules, like an npm_package_version.

@juanpenarandaosf
Copy link

Another comment here from someone who sees benefit in the package.json listing all the different packages that a project has.

One thing that I wonder is, what happens if we need to upgrade a package to a new version? In node we could update the package.json and keep the "require('express')" for example. But in deno I understand we should go through every file and update it there. To me there should be a better way to address this issue because we could have the case in which some file is left unchanged and an old api is used unexpectedly or a security issue remains.

I like the solution from @kitsonk using the deps.ts, but I see we as developers should reexport the functions / classes from each one of our dependencies. In terms of maintenance I think deno could have something like the package.json to make this mapping, for example:

{
   "dependencies": {
       "react": "https://the-cdn-url-here@version"
    }
}

and then in the file just use import { something } from 'react' as normal. Also, I think this could improve the readability and could help more people transition to deno.

Keep up the good work guys! :)

@crowlKats
Copy link
Member

@juanpenarandaosf you are asking for import maps basically, which are already a thing

@juanpenarandaosf
Copy link

Beautiful! thank you

@balupton
Copy link
Contributor

balupton commented Jun 10, 2020

I've created make-deno-edition to make npm packages written in typescript compatible with deno - is working on badges - usage proof of this here https://repl.it/@balupton/badges-deno - has been used now to make 32 node packages compatible with deno - you can use project to automatically generate the readme instructions for the deno edition - and can use boundation to automatically scaffold your projects to automate the entire process - start-of-week is an example where different entries are used for node, deno, and web browsers

@TylerBre
Copy link

TylerBre commented Jun 28, 2020

It's been mentioned a few times, but it seems some of you fail to read the documentation. Deno has support for "importmaps", which serve to alias import URL's to human-readable strings or "package names".

https://deno.land/manual/linking_to_external_code/import_maps

For the lazy, this means you create a file called "import_map.json" and populate it with your projects dependencies. Shockingly (coincidence?) similar to how package.json works...

By definition, this is package management. Just because there is no magical CLI to update and manage this file for you, does not mean there is no way to manage and centralize your project's dependencies.

Further, if any of you wish to create a tool to do this, you are more than welcome.

Edit: for the people comparing a JS runtime to Go and Kotlin... Really?

@theoludwig
Copy link

In the official website manual, they are using the deps.ts file solution. So what is the best solution, import maps or deps file ?

There shouldn't be 2 ways of doing the same thing as it is promoted by deno, for example the builtin utilities like fmt, lint or tests so everyone in deno use the same linter and formatter, no need to struggle with ESLint or Prettier configuration since everyone follows the same configurations and conventions.

I don't know if it's related to the issue (excuse me if it's not) : In Node.js in package.json there is a "scripts" object to easily execute long repetitive command, in deno it doesn't exist and it is all the more true with deno because of the security flags, so everytime you want to execute your deno script you must remember all the flags and options ?

Note : Since I'm mostly a Node.js developer, I'm new to the deno world, but really like some of the concepts promoted by deno and looking forwards to see the improvements. Excuse me if I'm saying something wrong.

@crowlKats
Copy link
Member

importmaps are a web standard, so deno implemented it. deps.ts is just a convetion people like to use. the reason people use deps.ts is because importmaps cant be used by modules, only by the final product.

@chickencoding123
Copy link

chickencoding123 commented Oct 8, 2020

OK now I'm confused, why import maps and not standard typescript paths mapping? Specially since paths can be optionally placed in their own file using configuration inheritance? I must be missing something here 🤔

@lucacasonato
Copy link
Member

Because import maps are a browser standard that also work for JS (not only typescript)

@chickencoding123
Copy link

@lucacasonato how about support for paths can be an additional feature when using typescript.

@Soremwar
Copy link
Contributor

Soremwar commented Oct 8, 2020

@IRoninCoder TypeScript paths aren't meant to be used with URLs (which would require extra thinkering in the TS side to make it work like that) and it would complicate a lot more the module resolution system Deno uses, rather than just appending import maps to it.

Also, see #7732 which would effectively remove the capability to do that.

@kitsonk
Copy link
Contributor

kitsonk commented Oct 11, 2020

paths are only to provide the TypeScript compiler for information of where code exists at runtime. Since We no manages both the compilation of TypeScript and where code is "located" at runtime, paths is a non-sensical config operation and is ignored.

@balupton
Copy link
Contributor

Regarding the above, to reiterate, https://github.com/bevry/make-deno-edition generates a compatible edition with all necessary path remapping automatically performed; for local deps, deno deps, and node_module deps.

@loohoo
Copy link

loohoo commented Oct 20, 2020

I am new to deno. after reading the deno manual and this artical my mind mess up. have no idea what's the best solution for dependence management...

@lucacasonato
Copy link
Member

@loohoo You can use deps.ts and mod.ts without concern. That article and benchmark was written before we had incremental TypeScript compilation. It is not up to date anymore.

@azwar
Copy link

azwar commented Mar 22, 2021

NPM was easily abused, and Deno library is easier. See: https://www.veracode.com/blog/research/abusing-npm-libraries-data-exfiltration. We have to build both good package management and good repo central like Java and C# have.

@denoland denoland locked as resolved and limited conversation to collaborators Mar 22, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests