Relative file resolution (FILE_PWD) not working in configs (env.nu, config.nu)

[texastoland]

Describe the bug

My Nu config dir is symlinked to work around #893. I'm attempting to source-env default_env.nu to work around #7371. I'm seeing confusing error messages documented below.

I suspect there's a bug and/or poor error message and/or misunderstanding on my part and/or all of the above ❓

First Attempt

# env.nu
source-env default_env.nu

Error

Error: nu::parser::sourced_file_not_found

  × File not found
   ╭─[/Users/texas/Library/Application Support/nushell/env.nu:1:1]
 1 │ source-env default_env.nu
   ·            ───────┬──────
   ·                   ╰── File not found: default_env.nu
   ╰────
  help: sourced files need to be available before your script is run

Same error with ./default_env.nu.

Context

> '/Users/texas/Library/Application Support/nushell/env.nu'
➜ | path join .. default_env.nu
➜ | ls --long $in
➜ | reject name
╭─────┬────────┬──────────┬────────────┬─────────────┬─────────────┬────────────┬────────┬────────┬──────────┬─────────────┬─────────────┬─────────────╮
│   # │  type  │  target  │  readonly  │    mode     │  num_links  │   inode    │  user  │ group  │   size   │   created   │  accessed   │  modified   │
├─────┼────────┼──────────┼────────────┼─────────────┼─────────────┼────────────┼────────┼────────┼──────────┼─────────────┼─────────────┼─────────────┤
│   0 │ file   │          │ false      │ rw-r--r--   │           1 │  110565660 │ texas  │ staff  │  4.3 KiB │ 2 days ago  │ 2 days ago  │ 2 days ago  │
╰─────┴────────┴──────────┴────────────┴─────────────┴─────────────┴────────────┴────────┴────────┴──────────┴─────────────┴─────────────┴─────────────╯

Aside: I was surprised I couldn't pipe to ls.

Second Attempt

Supposing it was related to symlinks I tried to expand it.

# env.nu
source-env (default_env.nu | path expand)

Error

Error:   × Error: nu::shell::not_a_constant
  │ 
  │   × Not a constant.
  │    ╭─[/Users/texas/Library/Application Support/nushell/env.nu:1:1]
  │  1 │ source-env (default_env.nu | path expand)
  │    ·             ───────┬──────
  │    ·                    ╰── Value is not a parse-time constant
  │    ╰────
  │   help: Only a subset of expressions are allowed constants during parsing. Try using the 'const' command or typing the value literally.
  │ 
   ╭─[/Users/texas/Library/Application Support/nushell/env.nu:1:1]
 1 │ source-env (default_env.nu | path expand)
   ·            ───────────────┬──────────────
   ·                           ╰── Encountered error during parse-time evaluation
   ╰────

Same error with ./default_env.nu or assigning it to a const.

Final Attempt

I tried quoting it but it resolved relative to $env.PWD. That's entirely reasonable since source isn't doing the expansion.

# env.nu
source-env ('default_env.nu' | path expand)

Error

Error: nu::parser::sourced_file_not_found

  × File not found
   ╭─[/Users/texas/Library/Application Support/nushell/env.nu:1:1]
 1 │ source-env ('default_env.nu' | path expand)
   ·            ────────────────┬───────────────
   ·                            ╰── File not found: /Users/texas/default_env.nu
   ╰────
  help: sourced files need to be available before your script is run

Same error with './default_env.nu'.

Info

How to reproduce

Haven't identified the root cause yet.

Expected behavior

Expect to load.

Screenshots

No response

Configuration

key	value
version	0.88.1
branch
commit_hash
build_os	macos-x86_64
build_target	x86_64-apple-darwin
rust_version	rustc 1.74.0 (79e9716c9 2023-11-13) (Homebrew)
cargo_version	cargo 1.74.0
build_time	2023-12-14 17:14:47 +00:00
build_rust_channel	release
allocator	mimalloc
features	dataframe, default, sqlite, trash, which, zip
installed_plugins

Additional context

No response

[fdncred]

I'm kind of lost because this path will never resolve because env.nu is a file, right?

> '/Users/texas/Library/Application Support/nushell/env.nu'
➜ | path join .. default_env.nu

This is what it says on my system

❯ '/Users/fdncred/Library/Application Support/nushell/scripts/env.nu' | path join .. default_env.nu 
/Users/fdncred/Library/Application Support/nushell/scripts/env.nu/../default_env.nu

I'm also wondering if you do something like this, if it will work for you.

const var = ('/Users/fdncred/Library/Application Support/nushell/scripts/' | path join default_env.nu)
source $var

[texastoland]

> '/Users/texas/Library/Application Support/nushell/env.nu'
➜ | path join .. default_env.nu

Identical to:

> '/Users/texas/Library/Application Support/nushell/env.nu'
::: | path dirname | path join default_env.nu

I suppose path join .. obfuscates my issue:

1. source-env default_env.nu doesn't work despite being in the same directory
2. I don't understand the not_a_constant error in my second attempt
3. (Unrelated) ls unexpectedly can't be piped

I'm also wondering if you do something like this, if it will work for you.

Both the following work:

# where my dotfiles are
source-env ~/.config/nushell/default_env.nu

# symlinked dir until #7371 is resolved
source-env '~/Library/Application Support/nushell/default_env.nu'

I initially also tried to use $nu.default-config-dir but it wasn't exposed as a constant either.

Thanks for the help I'm loving it besides the 2 linked issues 🙌🏼

[fdncred]

This seems to work fine.

const x = ($nu.default-config-dir | path join env.nu)
source-env $x

I have no issues sourcing files from $nu.default-config-dir but maybe that may be because my $env.NU_LIB_DIRS has that folder in the list, which makes it source-able from anywhere without specifying a path.

❯ $env.NU_LIB_DIRS
╭───┬───────────────────────────────────────────────────╮
│ 0 │ C:\Users\username\AppData\Roaming\nushell\scripts │
│ 1 │ D:\nu_scripts                                     │
│ 2 │ C:\Users\username\AppData\Roaming\nushell         │
╰───┴───────────────────────────────────────────────────╯

A constant has to be a variable declared with const or a full path to a file. https://www.nushell.sh/book/how_nushell_code_gets_run.html and https://www.nushell.sh/book/thinking_in_nu.html go into how nushell works.

piping into ls isn't implemented but a motivated contributor could add it.

[texastoland]

This seems to work fine.

IFL I'm not communicating effectively. Starting from scratch but happy to rewrite the description too.

I have no issues sourcing files from $nu.default-config-dir but maybe that may be because my $env.NU_LIB_DIRS has that folder in the list, which makes it source-able from anywhere without specifying a path.

Nothing is loaded yet. Could that be the issue? I want to source the generated config in my user config as described in #893 (comment). My config directory happens to be symlinked as described in #7371 (comment):

# env.nu
source-env default_env.nu

I expect default_env.nu to be resolved relative to env.nu as described in the 0.67 release notes. Instead the error reports File not found: default_env.nu with no indication where it's relative to. It exists and is readable (output above). Additionally when piped inside a subexpression an error reports Value is not a parse-time constant unless quoted (example above). This may not be a bug but worth noting in Nushell's excellent docs because it confused me.

This seems to work fine.

const x = ($nu.default-config-dir | path join env.nu)
source-env $x

Thanks! I don't know what I did wrong before but the following successfully loads:

source-env ($nu.default-config-dir | path join default_env.nu)

piping into ls isn't implemented but a motivated contributor could add it.

Should I open an issue for backlog? It sounds like a couple lines at most. Rust isn't 1 of my languages but I look forward to collaborating on nushell/nushell.github.io#1189 🤘🏼

In conclusion is the import resolution and constant error expected behavior?

[texastoland]

I have no issues sourcing files from $nu.default-config-dir but maybe that may be because my $env.NU_LIB_DIRS has that folder in the list, which makes it source-able from anywhere without specifying a path.

Nothing is loaded yet. Could that be the issue?

I'm 50% thinking this is the issue.

find_in_dirs_env implements paths relative to FILE_PWD as well as NU_LIB_DIRS:

nushell/crates/nu-engine/src/env.rs

Lines 283 to 302 in 42bb42a

	/// This helper function is used to find files during eval
	///
	/// First, the actual current working directory is selected as
	/// a) the directory of a file currently being parsed
	/// b) current working directory (PWD)
	///
	/// Then, if the file is not found in the actual cwd, NU_LIB_DIRS is checked.
	/// If there is a relative path in NU_LIB_DIRS, it is assumed to be relative to the actual cwd
	/// determined in the first step.
	///
	/// Always returns an absolute path
	pub fn find_in_dirs_env(
	filename: &str,
	engine_state: &EngineState,
	stack: &Stack,
	dirs_var: Option<VarId>,
	) -> Result<Option<PathBuf>, ShellError> {
	// Choose whether to use file-relative or PWD-relative path
	let cwd = if let Some(pwd) = stack.get_env_var(engine_state, "FILE_PWD") {
	match env_to_string("FILE_PWD", &pwd, engine_state, stack) {

It relies on the presence of an environment variable I only found set in 2 places:

1. nushell/crates/nu-command/src/env/source_env.rs

   Line 66 in 42bb42a

   caller_stack.add_env_var("FILE_PWD".to_string(), file_pwd);

2. nushell/crates/nu-cmd-lang/src/core_commands/use_.rs

   Line 112 in 77f10eb

   callee_stack.add_env_var("FILE_PWD".to_string(), file_pwd);

3. nushell/crates/nu-cmd-lang/src/core_commands/overlay/use_.rs

   Line 136 in 42bb42a

   callee_stack.add_env_var("FILE_PWD".to_string(), file_pwd);

But AFAICT loading configs bypasses these pathways:

nushell/crates/nu-cli/src/config_files.rs

Lines 82 to 91 in 42bb42a

	pub fn eval_config_contents(
	config_path: PathBuf,
	engine_state: &mut EngineState,
	stack: &mut Stack,
	) {
	if config_path.exists() & config_path.is_file() {
	let config_filename = config_path.to_string_lossy();
	if let Ok(contents) = std::fs::read(&config_path) {
	eval_source(

Rust isn't 1 of my languages

I don't know Rust but hopefully that's a concrete starting point!

[texastoland]

I just confirmed use utils in my config.nu can't resolve from my NU_LIB_DIRS either.

[fdncred]

I just confirmed use utils in my config.nu can't resolve from my NU_LIB_DIRS either.

I don't get how that's possible. env.nu is loaded and parsed first (this is where NU_LIB_DIRS is supposed to be), then config.nu. so NU_LIB_DIRS is available to config.nu. I use it this way all the time.

[texastoland]

Can you confirm relative paths work in your env.nu and lib paths in config.nu? Neither works for me. If so I can at least identify where it's happening 👀 I can probably repro in a Codespaces later.

[fdncred]

I'm not really sure what you want me to test. I was just saying that this works.
nu_lib_dirs definition in my env.nu

$env.NU_LIB_DIRS = [
    ($nu.config-path | path dirname | path join 'scripts')
    '/Users/username/src/nu_scripts'
    ($nu.config-path | path dirname)
]

using scripts in my config.nu that only work if nu_lib_dirs is set.

# external completions
use "custom-completions/cargo/cargo-completions.nu" *

# git completions and aliases
use custom-completions/git/git-completions.nu *
use aliases/git/git-aliases.nu *

# temperature converter
use "sourced/temp.nu" *

# get_weather
use "modules/weather/get-weather.nu" get_weather

[texastoland]

I presented a simplified use case to test I don't understand? This errors for me OOTB (first line of file).

# env.nu
source-env default_env.nu

You're right about NU_LIB_DIRS. I thought use utils worked instead of use utils.nu. My bad 🙏🏼

[texastoland]

I reviewed the code again.

I don't get how that's possible. env.nu is loaded and parsed first (this is where NU_LIB_DIRS is supposed to be)

I was wrong NU_LIB_DIRS works. But FILE_PWD is used for relative paths and (I think) only added to the stack by the use and source commands:

1. nushell/crates/nu-command/src/env/source_env.rs

   Line 66 in 42bb42a

   caller_stack.add_env_var("FILE_PWD".to_string(), file_pwd);

2. nushell/crates/nu-cmd-lang/src/core_commands/use_.rs

   Line 112 in 77f10eb

   callee_stack.add_env_var("FILE_PWD".to_string(), file_pwd);

3. nushell/crates/nu-cmd-lang/src/core_commands/overlay/use_.rs

   Line 136 in 42bb42a

   callee_stack.add_env_var("FILE_PWD".to_string(), file_pwd);

But eval_config_contents doesn't defer to those commands:

But AFAICT loading configs bypasses these pathways:

nushell/crates/nu-cli/src/config_files.rs

Lines 82 to 91 in 42bb42a

	pub fn eval_config_contents(
	config_path: PathBuf,
	engine_state: &mut EngineState,
	stack: &mut Stack,
	) {
	if config_path.exists() & config_path.is_file() {
	let config_filename = config_path.to_string_lossy();
	if let Ok(contents) = std::fs::read(&config_path) {
	eval_source(

That's why find_in_dirs_env isn't resolving the path:

find_in_dirs_env implements paths relative to FILE_PWD as well as NU_LIB_DIRS:

nushell/crates/nu-engine/src/env.rs

Lines 283 to 302 in 42bb42a

	/// This helper function is used to find files during eval
	///
	/// First, the actual current working directory is selected as
	/// a) the directory of a file currently being parsed
	/// b) current working directory (PWD)
	///
	/// Then, if the file is not found in the actual cwd, NU_LIB_DIRS is checked.
	/// If there is a relative path in NU_LIB_DIRS, it is assumed to be relative to the actual cwd
	/// determined in the first step.
	///
	/// Always returns an absolute path
	pub fn find_in_dirs_env(
	filename: &str,
	engine_state: &EngineState,
	stack: &Stack,
	dirs_var: Option<VarId>,
	) -> Result<Option<PathBuf>, ShellError> {
	// Choose whether to use file-relative or PWD-relative path
	let cwd = if let Some(pwd) = stack.get_env_var(engine_state, "FILE_PWD") {
	match env_to_string("FILE_PWD", &pwd, engine_state, stack) {

[fdncred]

Does your example work if you use source versus source-env. I'm not even sure how well source-env is supported these days.

[texastoland]

No not use either. Based on the above code above I suspect relative paths were overlooked in eval_config_contents.

[fdncred]

could've been, that was written/changed 22 months ago so a lot has changed since then

[texastoland]

Ping @kubouch from Git blame (I hope that's okay) 🔔 I'm not a Rust person but I'm having an issue with relative file loading that I think (based on clicking around GitHub so maybe wrong) I identified the issue in #11465 (comment). Maybe can check eventually 🙏🏼 Anyway really thank you all Nushell is wonderful!

[kubouch]

The parser looks for currrently_parsed_cwd field which is set before parsing via start_in_file() before starting to parse a file. $env.FILE_PWD is intended just for users to use in their scripts. Anyway, it's totally possible we forgot to put start_in_file() somewhere.

It's hard to figure out what the issue is because it's missing a minimal working example. There are no clear steps to reproduce the issue. It would be great if you could summarize the reproduction steps, starting from scratch, as simple as possible, in a 1., 2., 3., ... list such that we can quickly reproduce it.

[texastoland]

This was my minimal example:

I presented a simplified use case to test I don't understand? This errors for me OOTB (first line of file).

# env.nu
source-env default_env.nu

I can't source any file relatively in env.nu. Same for config.nu (assuming $nu.default-config-dir is not in $env.NU_LIB_DIRS). I'm not launching nu with any flags. Steps:

1. Default install
2. Create env.nu above
3. Create default_env.nu in same directory (contents don't matter)

Result:

Error: nu::parser::sourced_file_not_found

  × File not found
   ╭─[/Users/texas/Library/Application Support/nushell/env.nu:1:1]
 1 │ source-env default_env.nu
   ·            ───────┬──────
   ·                   ╰── File not found: default_env.nu
   ╰────
  help: sourced files need to be available before your script is run

Same error with ./default_env.nu.

use gives an almost identical error message except module_not_found.

[kubouch]

Yeah, I can reproduce it. Seems like the start file is not set correctly when evaluating the config files. This should be an easy fix: Add start_in_file() calls to the appropriate locations (see my previous comment).

[texastoland]

Thanks so much!!! It's a small thing but it makes breaking up my config feel messier. I added this to my list to attempt if I have time (familiar with Rust but never written any).

[texastoland]

Closing as duplicate of #7849.