Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Roadmap for Nim #437

Closed
33 tasks done
Araq opened this issue Nov 19, 2021 · 47 comments
Closed
33 tasks done

Roadmap for Nim #437

Araq opened this issue Nov 19, 2021 · 47 comments

Comments

@Araq
Copy link
Member

Araq commented Nov 19, 2021

"The Cylons Were Created by Man. They Rebelled. They Evolved. They Look and Feel Human.
Some are programmed to think they are Human. There are many copies. And they have a Plan."

It seems to come up regularly that people don't know where Nim is heading, where our focus of development is and what's the PLAN. I'm sorry for not communicating it well but the plan is solid, doesn't change much and it "only" takes longer than I hoped it would take.

Incremental compilation (IC) / nimsuggest / IDE tooling

This is a single point as for me it's a single thing. If we had a compiler that only recompiles the module that was touched the "nimsuggest" tool can be a simple command line tool that answers a single question like "where is symbol X used?".

The good news is that we have this tool already, it's nim check --navigate, the bad news is that its progress is tied to PR nim-lang/Nim#17814. Which is in the state "on hold" until version 1.6.2 is out.

IC is the most important thing for me personally. It's not scheduled for any particular release as it's not about the language nor about its library. Ideally IC will stabilize and be backported to the 1.6.x line.

Update: Good IC relies on refactorings of the compilation pipeline. We want to finish these refactorings, release version 2.0 and then finish IC in version 2.x.

Version 1.6.2

  • We will release 1.6.2 near-term (ideally still in 2021) which is 1.6.0 with many bugs ironed out. It is expected that the 1.6.x line is a LTS version, much like 1.2.x is.

Version 2.0

Hard goals

For version 2.0 we will make --gc:orc, mimalloc and --threads:on the new defaults.

Furthermore, we consider the following experimental features to be stable enough for version 2.0:

Stretch goals

Language

Cleanup of experimental language features

Stable experimental language features

These features are stable and should be part of the manual.

Library

  1. Clean up the system module, move these submodules into their own modules:

  2. Clean up the OS module, make it more type safe, split it in up in e.g.:

  3. The JSON module is also in a poor shape, the API exposes public fields, sometimes is not intuitive (the %* operator) and has poor performance. It could be split up into:

    • [-] jsonbuilder
    • [-] jsonparser
    • [-] jsonlexer
    • [-] jsontree
    • [-] jsonmapper: it maps Nim types (objects) to Json types
  4. More of the standard library should be moved to external Nimble packages. How
    far we get with this is unclear.

Misc (in comments)

Version 2.x

Every stretch goal of 2.0 that didn't make it into version 2.0, should be
considered a goal for version 2.x.

Hard goals

We should remove the requirement for forward declarations and allow for recursive module dependencies. Then we can remove these experimental features:

  1. "Package level objects"
  2. "code reordering"

see also nim-lang/Nim#18818

Stretch goals

  1. Find a better design for "Term rewriting macros". Once done, deprecate the old feature.
  2. Find a better design for "Special Operators". Once done, deprecate the old feature.
  3. Implement "Aliasing restrictions in parameter passing". The compiler can always produce warnings first.
  4. "Strict funcs" / "view types" should be made stable for Nim 2.x.
  5. Concepts: Find a way of how to infer static values and stabilize the feature.
  6. Find a way of allowing a syntax like of Value(let x, var y) and embrace an official pattern matching solution that makes use of this syntax.

Nimble

We hope to be able to ship a new Nimble with the 2.0 release but we are happy to ship it whenever it's ready. See #398 for good ideas of how to further evolve Nimble. The most important missing feature of Nimble is "lock files" -- Nimble implements it, but not in a way that is compatible with the 1.6.0 Nim compiler.

How can you help?

The standard library cleanups should be rather easy to do.

@Vindaar
Copy link

Vindaar commented Nov 19, 2021

"AST based overloading" -- Should be removed.

Not sure how I feel about this. Would this also imply removal of the same feature for NimNodes? I rather like that (i.e. specifying a CT proc is only for idents).

To be fair I didn't know this could be used with runtime procedures.

@metagn
Copy link
Contributor

metagn commented Nov 19, 2021

Find a way of allowing a syntax like of Value(let x, var y) and embrace an official pattern matching solution that makes use of this syntax.

Reminds me, default values for object fields was quite popular, is this planned as well? (just saw this higher up in the post) are any object variants improvements from many proposals going to be considered? I've also seen a lot of complaints about object construction, but I don't know which useful changes there are other than default values, if there was going to be a =init I'm guessing it'll probably be more coherent for it to be similar to =destroy instead of a complete override of initialization (after zeroing memory/initializing default field values).

The JSON module ... could be split up into:

Would json/builder, json/lexer be better (with a re-export of every submodule in a module named json)? I remember this being discussed for other modules but vaguely something about it not working or being inconvenient. I believe it was proposed for the xml modules and times, but deemed problematic for os.

@Araq
Copy link
Member Author

Araq commented Nov 19, 2021

Would json/builder, json/lexer be better

Yes, most people would like that better but I don't want to implement and support package.module.name just for a refactoring so I'd go with the name prefixes.

@konsumlamm
Copy link

"Void type" -- We should optimize away the "empty tuple" type in the backend and remove this feature.

So eventually, void would be replaced by the empty tuple, so that e.g. proc test() = discard would return () (like it is the case in Rust)?

@Araq
Copy link
Member Author

Araq commented Nov 20, 2021

So eventually, void would be replaced by the empty tuple, so that e.g. proc test() = discard would return () (like it is the case in Rust)?

No, for the return type and discard-checking there would still be a void.

@levovix0
Copy link

+1 for moduleGroup/module instead of moduleGroupmodule

I don't want to implement and support package.module.name

why implementing and supporting package.module.name needed for package/module/name?

@Araq
Copy link
Member Author

Araq commented Dec 1, 2021

Update: Expanded the section on Nimble.

@juancarlospaco
Copy link
Contributor

"Parallel & Spawn" -- Should be removed. Better done by external packages.

But stdlib keeps std/tasks ?,
I think a way to run multiple stuff at the same time in stdlib is useful !,
at least with a basic API, maybe think of removing Parallel and keep Spawn?

@Araq
Copy link
Member Author

Araq commented Dec 5, 2021

std/tasks is a "building block", so it should stay.

@Varriount
Copy link

Varriount commented Dec 6, 2021

+1 for moduleGroup/module instead of moduleGroupmodule

I don't want to implement and support package.module.name

why implementing and supporting package.module.name needed for package/module/name?

I believe that this is because import only considers the last path component as the module name. The code from json/lexer import nil allows accessing all exported routines/types in "./json/lexer.nim" through the lexer symbol, not the expression json.lexer.

If an expression like json.lexer won't be supported, then the symbol "jsonLexer" may be preferable to "lexer", since "jsonLexer" is less ambiguous.

@Araq
Copy link
Member Author

Araq commented Dec 17, 2021

Update:

  • Nimble supports lockfiles and Nim 1.6.2 supports the new Nimble. (But it doesn't yet ship with it.)
  • Nim version 1.6.2 has been released.

@juancarlospaco
Copy link
Contributor

Would be nice to give the jsgen some love too!, at least improve jsgen a little,
the JS stdlib improved a lot and wont produce Showstopper complex bugs,
and the few JS bugs are actually jsgen related bugs actually so ... 🤔

@a-mr
Copy link

a-mr commented Dec 17, 2021

What is policy for breaking changes in 2.0?

I see 2 extreme options:

  1. almost no breaking allowed as it is currently in Nim 1.x. The transition to gc:orc is an exception.
  2. any breaking changes are potentially allowed if it is the right thing. Why would you call it version 2.0 otherwise...

Some concrete examples I know:

@Araq
Copy link
Member Author

Araq commented Dec 17, 2021

We should fix as much as reasonable, but we need to have a good upgrade path. The worst thing continues to be subtle changes in runtime behavior. That means:

  • I don't know what to do with slices.
  • Deprecate walkDir and introduce something that makes sense. Can also be in a module of its own.
  • Anchors for v2 are not worth preserving, details like that are not worth

@zedeus
Copy link

zedeus commented Jan 4, 2022

Very cool, nice roadmap. What's the current status, what is the primary project being worked on?

The good news is that we have this tool already, it's nim check --navigate, the bad news is that its progress is tied to PR nim-lang/Nim#17814. Which is in the state "on hold" until version 1.6.2 is out.

1.6.2 is now out, but there hasn't been any updates to that PR. What's the status of it?

@Araq
Copy link
Member Author

Araq commented Jan 5, 2022

1.6.2 is now out, but there hasn't been any updates to that PR. What's the status of it?

I never know how to update these RFCs. 1.6.2 is out, next week work on 2.0 will begin.

@Araq
Copy link
Member Author

Araq commented Jan 16, 2022

There is nim-lang/threading which is official plus a couple of external packages that do it better than we could hope to. There are currently no plans to bring this into the stdlib for these reasons:

  • We lack the resources; our job shifted to compiler and tooling work which is demanding enough already.
  • The stdlib should be trimmed down a bit, not expanded further.
  • Most importantly: As separate packages they can evolve much more quickly, not hindered by a deprecation cycle that spans a decade. And not tied to our slow release cycle.

We merely offer the building blocks with destructors, atomics and std/tasks.

@metagn
Copy link
Contributor

metagn commented Jan 16, 2022

Note for the checklist: guards & locks are in both experimental features to clean up and stable experimental features. Since guards were moved to the stable documentation, it can be marked under the stable experimental features section.

Some RFCs covered by the items:

A few simple non-breaking features that could be done at any time, but would be nice to have soon, probably also under 1.6 line:

There are also tons of RFCs related to default object fields and pattern matching but they tend to be larger in scope and discuss other things.

I think "optimize away empty tuple" will take a while compared to the steps for removing the other experimental features. But since it's not tied to a super well known experimental feature it should be fine if it's not 2.0.

Does the var in of Value(let x, var y) mean var y: T or y: var T? I don't think it should be the first one given for loops and proc parameters presumably don't have it for a reason. Second one is something the language could do that macros couldn't.

"Not nil annotation" -- Should be planned for Nim 2.x but removed from the "experimental" manual.

Don't understand this. Is it supposed to be kept in as experimental but undocumented?

@beef331
Copy link

beef331 commented Jan 18, 2022

I just got reminded, is there any chance marshal will get removed from the stdlib for 2.0, or at least removing the VM support for it. Almost every json library made can be used at compile time without hard coded VM support, and as most know marshal is just a json serializer. https://github.com/nim-lang/Nim/blob/devel/compiler/vmgen.nim#L2057-L2065 can cause 10% slower compilation in compile time heavy code(macros included). Since every single call is done with two string searches to see if it's from marshal.

@ringabout
Copy link
Member

Alternative way is to support vmmarshal properly with callback if it is worth.

@Araq
Copy link
Member Author

Araq commented Jan 19, 2022

@beef331 wow... that should be fixed indeed. It doesn't necessarily mean we need to disable marshal on the VM though, it should simply either use its own opcodes or the existing callback mechanism.

@ringabout
Copy link
Member

A simple POC using callback mechanism: https://gist.github.com/xflywind/551d84567cb39d95d0ddcd71f8279e6a

@ringabout
Copy link
Member

ringabout commented Jan 21, 2022

Path related RFCs:

#54 new path handling module that uses Nim's type system, using Path instead of string
#121 os module should have an "exists" function
#71 [compiler/pathutils] AbsoluteFile,AbsoluteDir,RelativeFile,RelativeDir is too complicated
#112 joinPath("foo", "/bar") should return "/bar"

@Menduist
Copy link

Menduist commented Mar 3, 2022

Could we consider #89 for nim V2?

A few options have been considered, eg the linked RFC, or deprecating naked excepts, all of them being better than the current status quo, imo

@juancarlospaco
Copy link
Contributor

juancarlospaco commented May 13, 2022

I would like to re-consider ideas proposed before this Nim 2.0 trend, like:

@beef331
Copy link

beef331 commented May 13, 2022

Slightly related to arne's point. In my quest to make code more reusable I did attempt to make parseutils use openarray[char] but it's showed a few issues with openarray/string/cstring that will need to get resolved. Presently due to implicit conversion from string to cstring the following does not work.

proc doThing(oa: openarray[char]) = discard
proc doThing(str: cstring) = discard
doThing("hello")

Similarly the following perhaps should work. Where openArray[char] is considered equivalent to string in the case of converters.

type NotString = distinct string
converter toString(s: NotString): string = string(s)
proc doThing(oa: openarray[char]) = discard
doThing(NotString("hello"))

@juancarlospaco
Copy link
Contributor

The Nim 2.0 can also be a good opportunity to make a decision about Nimble packages clean out,
packages that do not exist anymore and give 404 errors, but they are still in the packages list,
theres multiple issues open and closed in the repo about the problem,
and removing the packages from the JSON looks like it is not allowed (?),
but it is useless because the repo do not exist anymore,
and sometimes the user also do not exist anymore or is not coding anymore since years ago.

@CyberTailor
Copy link

"Parallel & Spawn" -- Should be removed. Better done by external packages

What about bootstrapping build systems/package managers that use threads? Will they need to supply their own implementation of threadpool?

@Araq
Copy link
Member Author

Araq commented Jul 5, 2022

No, package managers and bulid systems can use Nimble packages as dependencies if they do the "git clone" commands directly in the build script and/or can use git submodules. A package manager is not like a compiler, it doesn't have to "bootstrap" itself.

@CyberTailor
Copy link

CyberTailor commented Jul 5, 2022 via email

@Araq
Copy link
Member Author

Araq commented Jul 5, 2022

The sandbox would prevent nimble install nimble just as it would prevent git clone depsThatNimbleRequires.

@CyberTailor
Copy link

CyberTailor commented Jul 5, 2022 via email

@Araq
Copy link
Member Author

Araq commented Jul 5, 2022

Fine with me, you started the subdiscussion.

@tomasky
Copy link

tomasky commented Jul 15, 2022

I like nim, but cross-compiling is inconvenient. I wish I could cross-compile directly like Golang or ziglang, which would attract more users

@Araq
Copy link
Member Author

Araq commented Jul 15, 2022

Just use a CI, cross-compilation won't let you run any tests anyway... I wish we would attract more users who understand software development.

@tomasky
Copy link

tomasky commented Jul 22, 2022

Ok, I spent some time cross-compiling using CI:

- name: build-x86
  image: dockcross/linux-x86
  commands:
  - nimble build --passL:-static -o:out-linux-x86 -d:release
 
- name: build-x86-64
  image: dockcross/linux-x64
  commands:
  - nimble build --passL:-static -o:out-linux-amd64 -d:release
    
- name: build-linux-arm64
  image: dockcross/linux-arm64
  commands:
  - nimble build --cc:env --os:linux --cpu:arm64 -o:out-linux-arm64 -d:release

- name: build-android-arm64
  image: ghcr.io/tomasky/drones/arm64:nim
  commands:
  - nimble build --cc:env --os:android --cpu:arm64 -o:out-android-arm64 -d:release

@Araq
Copy link
Member Author

Araq commented Oct 15, 2022

Postponed:

@Araq
Copy link
Member Author

Araq commented Oct 25, 2022

Postponed:

  • The JSON module is also in a poor shape, the API exposes public fields, sometimes is not intuitive (the %* operator) and has poor performance.

@Araq
Copy link
Member Author

Araq commented Dec 28, 2022

I'm closing this since version 2.0 is effectively done, we'll only fix even more ORC bugs and release 2.0 in early 2023. The plan for 2023 will be published as an RFC here in the next couple of days.

@Araq Araq closed this as completed Dec 28, 2022
@Araq Araq unpinned this issue Dec 29, 2022
@ShalokShalom
Copy link

I like nim, but cross-compiling is inconvenient. I wish I could cross-compile directly like Golang or ziglang, which would attract more users

https://github.com/enthus1ast/zigcc

Just use the Zig compiler 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests