Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bib file not found even though known to kpse #416

Closed
AndrewSwann opened this issue Jun 29, 2022 · 68 comments
Closed

bib file not found even though known to kpse #416

AndrewSwann opened this issue Jun 29, 2022 · 68 comments

Comments

@AndrewSwann
Copy link

AndrewSwann commented Jun 29, 2022 •
edited
Loading

Running biber 2.18 on a mac, installed via macports texlive distribution, bib files that can be found by kpsewhich in the shell are not longer found by biber. This was previously working for me a few days ago.

Running with --trace, I get

[0] Config.pm:306> INFO - This is Biber 2.18
[0] Config.pm:309> INFO - Logfile is 'a.blg'
[80] biber-darwin:340> INFO - === Wed Jun 29, 2022, 08:39:55
[97] Biber.pm:418> INFO - Reading 'a.bcf'
[196] Biber.pm:978> INFO - Found 1 citekeys in bib section 0
[196] Biber.pm:981> DEBUG - The citekeys for section 0 are: test
[323] biber-darwin:426> TRACE - 
....
[335] Biber.pm:4401> INFO - Processing section 0
[335] Biber.pm:4515> DEBUG - Looking for directly cited keys: test
[347] Biber.pm:4592> INFO - Looking for bibtex file 'x.bib' for section 0
[347] Utils.pm:410> ERROR - Cannot find 'x.bib'!
[347] Biber.pm:135> INFO - ERRORS: 1

Looking at the source of Utils.pm suggests that kpsewhich was never tried, otherwise the report should have included messages such as Looking for file '$sourcepath' via kpsewhich etc. I have tried removing the biber cache and running again, but this does not change the result.
@plk
Copy link
Owner

plk commented Jun 29, 2022 •
edited
Loading

Hmm, that's the second report of this I've had but currently I can't reproduce it. You are right that it seems that it can't find kpsewhich in the PATH for some reason. Is this on x86 or ARM?

@AndrewSwann
Copy link
Author

It is x86.

@plk
Copy link
Owner

plk commented Jun 29, 2022 •
edited
Loading

Just to make sure we're comparing the same thing, can you download onto your desktop the universal binary from here:

https://sourceforge.net/projects/biblatex-biber/files/biblatex-biber/current/binaries/MacOS/biber-darwin_universal.tar.gz/download

(you may well have to clear the security complaint as this binary isn't signed like the TL one - try to open it once, close the complaint window, go into System Preferences->Security & Privacy and you'll see biber mentioned with a "Allow Anyway" button. Click that and on the next attempt to run it, just click "Open" in the popup window and after that it will run normally)

and then run:

~/Desktop/biber --debug --tool biblatex-examples.bib

and then look in the biblatex-examples.bib.blg

to see what's happening?

I just did this on an x86 laptop and it's fine for me.

Also check to make sure kpsewhich is in the PATH of the shell running biber?

@AndrewSwann
Copy link
Author

Many thanks, that downloaded version works. The version on my system with the corresponding invocation fails. kpsewhich runs fine from the same shell.
tlmgr info biber gives

package:     biber
category:    Package
shortdesc:   A BibTeX replacement for users of BibLaTeX
longdesc:    Biber is a BibTeX replacement for users of BibLaTeX. Biber supports full UTF-8, can (re)-encode input and output, supports highly configurable sorting, dynamic bibliography sets and many other features. The CTAN distribution offers a compressed tar archive of the sources, etc., together with "binary" distributions for a variety of platforms. Note: on SourceForge biber is formally named "biblatex-biber", to distinguish it from an earlier (now apparently moribund) project called "biber".
installed:   Yes
revision:    63708
sizes:       src: 1193k, doc: 269k, bin: 98113k
relocatable: No

and tlmgr restore provides

tlmgr restore --dry-run biber
tlmgr restore: dry run, no changes will be made
Available backups for biber: 61867 (2022-06-26 11:52)

So the texlive version has been bumped recently. Should I try to restore this earlier version?

@plk
Copy link
Owner

plk commented Jun 29, 2022

Would be interesting to see what difference that makes - I have no idea what is going on here currently - I just released the MacOS stuff as usual and it was pulled into TL.

@AndrewSwann
Copy link
Author

AndrewSwann commented Jun 29, 2022 •
edited
Loading

OK, I have now tried that. Unfortunately, restoring the texlive version 61867 doesn't help. Should I/we be asking the texlive maintainers instead? Incidentally reupdating texlive now gives revision number 63752, but it still has the same problem.

@plk
Copy link
Owner

plk commented Jun 29, 2022

I think the move to 6308 was just a documentation change as I had to request that yesterday. I just tried the same thing using the TL binary from macTeX TL 2021 and 2022 on both ARM and x86 and I still can't get it to fail. Tried version 61867 and 63708 and both work for me on ARM and x86_64. Very odd.

@AndrewSwann
Copy link
Author

AndrewSwann commented Jun 29, 2022 •
edited
Loading

Is there live dependence on perl modules or is everything included in the binary? I notice some parts of my perl installation have updated recently, but I haven't used perl for a long time, so am not sure what to check. The can_run command is from IPC:Cmd. But that works fine

#!/usr/bin/env perl

use IPC::Cmd qw( can_run );
print(can_run('kpsewhich'));

gives

/Library/TeX/texbin/kpsewhich%

as output

@plk
Copy link
Owner

plk commented Jun 29, 2022

There are no live dependencies - it's all internal to the cached perl tree "inside" biber. I was thinking it might be to do with what they do to the binaries to sign them in TL as that one you downloaded was the exact binary they pulled into CTAN and TL. They then have to sign them for MacOS. However, that still doesn't explain why I can run all of this without problems with the TL binaries ...

@plk
Copy link
Owner

plk commented Jun 29, 2022

Can you let me know which version of MacOS this is on?

@herbs
Copy link

herbs commented Jun 29, 2022

I'm on macOS 12.4, Monterey. I'm also using a fully updated TeX Live 2022 originally installed via MacTeX.

By the way, in an earlier email you noted that there were 2 versions of kpsewhich on your system. One was in /opt via MacPorts. The other seemed to be in TeX Live 2021 rather than 2022! I have only one active version of kpsewhich on my PATH.

@plk
Copy link
Owner

plk commented Jun 29, 2022

That situation with macports kpsewhich was a red herring I think. It's currently frustrating that I can't reproduce this. I also have TL 2022 via MacTeX on an x86 laptop but it's not 12.4 as it's an older laptop that can't take 12. I have 12.4 on ARM but that doesn't seem to be an issue. Let's see what @AndrewSwann has.

@AndrewSwann
Copy link
Author

AndrewSwann commented Jun 30, 2022 •
edited
Loading

I am also on macOS 12.4 Monterey. biber was working fine a week or so ago and I have been running Monterey for some time. I have only one version of kpsewhich on my system: /Library/TeX/texbin/kpsewhich

@plk
Copy link
Owner

plk commented Jun 30, 2022

Can you both try deleting the biber cache and trying this again? To do this, run

biber --cache

and just delete the entire directory this returns. Then, be sure to run biber once until it returns so it can unpack the cache again.

@herbs
Copy link

herbs commented Jun 30, 2022 via email

@AndrewSwann
Copy link
Author

I removed that entire parent directory to the cache and ran biber again, but unfortunately that doesn't solve the problem.

@plk
Copy link
Owner

plk commented Jun 30, 2022

Can you try the following and see what happens?

cp /usr/local/texlive/2022/bin/universal-darwin/biber ~/Desktop/biber
codesign --remove-signature ~/Desktop/biber
~/Desktop/biber --tool biblatex-examples.bib

@plk
Copy link
Owner

plk commented Jun 30, 2022

Can you also get this small binary:

https://www.dropbox.com/s/kjocvk15eb0sddi/test?dl=0

and run it on your machines with the problem? It just runs the perl code below to see what environment PAR is seeing:

use IPC::Cmd (can_run);
use v5.26;

say 'PATH=' . $ENV{'PATH'};
say can_run "kpsewhich";

@AndrewSwann
Copy link
Author

Moving biber to Desktop as you described, and running worked fine. I have sent you the result of the test script by email.

@plk
Copy link
Owner

plk commented Jun 30, 2022 •
edited
Loading

Hmm, did the copy to the Desktop work or was it the codesign signature removal that did the trick?

@AndrewSwann
Copy link
Author

Good point. Just copying to the Desktop without removing the signature works.

@plk
Copy link
Owner

plk commented Jun 30, 2022

That is really odd - let's see what happens for @herbs.

@plk
Copy link
Owner

plk commented Jun 30, 2022

Just out of interest - what happens if you overwrite the TL binary with the one copied to the Desktop?

@herbs
Copy link

herbs commented Jun 30, 2022 via email

@herbs
Copy link

herbs commented Jun 30, 2022 via email

@plk
Copy link
Owner

plk commented Jun 30, 2022

and zsh kills it every time now? Sometimes there is a second popup to allow unsigned after the System Preferences auth.

@plk
Copy link
Owner

plk commented Jun 30, 2022

One more test then - what about copying the test binary temporarily to /usr/local/texlive/2022/bin/universal-darwin/? I have a feeling that something horrible is happening with MacOS SIP here.

@AndrewSwann
Copy link
Author

Copying it from Desktop back to the texlive location gives the old error

@AndrewSwann
Copy link
Author

No, copying to universal-darwin directory does not help.

@plk
Copy link
Owner

plk commented Jun 30, 2022

What output does the "test" binary give when it's in /usr/local/texlive/2022/bin/universal-darwin/? Same as when it is in Desktop?

@plk
Copy link
Owner

plk commented Jul 3, 2022

Apologies:
https://www.dropbox.com/s/zu4ka1hpsjdp8wy/test?dl=0

@herbs
Copy link

herbs commented Jul 3, 2022 via email

@leonmoonen
Copy link

leonmoonen commented Jul 3, 2022 •
edited
Loading

gotcha!

➤ biber-test
PATH=/usr/local/Caskroom/miniconda/base/bin

➤ /Library/TeX/texbin/biber-test
PATH=/usr/local/Caskroom/miniconda/base/bin:/usr/local/Caskroom/miniconda/base/condabin:/Users/leon/bin:/usr/local/bin:/usr/local/sbin:/usr/local/opt/sqlite/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
/Library/TeX/texbin/kpsewhich

NB: renamed the executable to avoid conflicts with shell built-ins

@plk
Copy link
Owner

plk commented Jul 3, 2022 •
edited
Loading

@herbs - yes we preempted that issue and released a new biblatex version last night just after the cut-off for distribution so you will get 3.18a with the fix today.

For an emergency fix, see:

plk/biblatex#1234

@plk
Copy link
Owner

plk commented Jul 3, 2022

@leonmoonen - Aha, that's the issue then. How strange - and that doesn't happen with the raw perl if that's in /Library/TeX/texbin?

@leonmoonen
Copy link

nope, the raw perl script in /Library/TeX/texbin gives the complete path in both cases

@plk
Copy link
Owner

plk commented Jul 3, 2022 •
edited
Loading

Can you see what happens to the binary and the script when called through the real (no symlinks) absolute path, e.g:

/usr/local/texlive/2022/bin/x86_64-darwinlegacy/biber-test

@leonmoonen
Copy link

Through the absolute path, things still break when the executable is found via $PATH, direct calls work as expected

➤ /usr/local/texlive/2022/bin/universal-darwin/biber-test
PATH=/usr/local/Caskroom/miniconda/base/bin:/usr/local/Caskroom/miniconda/base/condabin:/Users/leon/bin:/usr/local/bin:/usr/local/sbin:/usr/local/opt/sqlite/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
/Library/TeX/texbin/kpsewhich

➤ /usr/local/texlive/2022/bin/universal-darwin/raw-biber-test
PATH=/usr/local/Caskroom/miniconda/base/bin:/usr/local/Caskroom/miniconda/base/condabin:/Users/leon/bin:/usr/local/bin:/usr/local/sbin:/usr/local/opt/sqlite/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
/Library/TeX/texbin/kpsewhich

➤ fish_add_path -p /usr/local/texlive/2022/bin/universal-darwin

➤ which biber-test
/usr/local/texlive/2022/bin/universal-darwin/biber-test

➤ biber-test
PATH=/usr/local/texlive/2022/bin/universal-darwin
/usr/local/texlive/2022/bin/universal-darwin/kpsewhich

➤ which raw-biber-test
/usr/local/texlive/2022/bin/universal-darwin/raw-biber-test

➤ raw-biber-test
PATH=/usr/local/Caskroom/miniconda/base/bin:/usr/local/Caskroom/miniconda/base/condabin:/Users/leon/bin:/usr/local/bin:/usr/local/sbin:/usr/local/opt/sqlite/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
/usr/local/texlive/2022/bin/universal-darwin/kpsewhich

@plk plk mentioned this issue Jul 3, 2022
Closed
@plk
Copy link
Owner

plk commented Jul 3, 2022

I can reproduce this on MacOS 10.12 Intel too. Have opened ticket to PAR::Packer as it seems to be something to do with the binary packaging - see mentioned ticket above.

@plk
Copy link
Owner

plk commented Jul 4, 2022

Looks like this might be an issue with an older version of PAR::Packer as I can fix the issue by upgrading this on MacOS 10.12. I will be upgrading the MacOS build servers and pushing new versions soon.

@herbs
Copy link

herbs commented Jul 4, 2022 via email

@amunn
Copy link

amunn commented Jul 4, 2022

Late to the party here, but I'm having the same problem (regularly installed MacTeX) MacOS 12.4 (Monterey). biber 2.18.

@herbs
Copy link

herbs commented Jul 4, 2022 via email

@amunn
Copy link

amunn commented Jul 4, 2022 •
edited
Loading

On Jul 4, 2022, at 1:13 PM, Alan Munn @.***> wrote: Late to the party here, but I'm having the same problem (regularly installed MacTeX) MacOS 12.4 (Monterey). biber 2.18.
Howdy, Just to be clear. Can we assume you are on an Intel system? It seems that the arm64 code runs. I've got one of each, so

Yes, Intel system. Using the full path works, but not the filename alone.

@plk
Copy link
Owner

plk commented Jul 4, 2022

It's a problem with the module used to package the binary I believe. I am in the process of getting a new release out with a fix but my main build server is having a few issues I have to deal with first, hopefully this evening.

@plk
Copy link
Owner

plk commented Jul 6, 2022

After coincidental and irritating server issues, I have rebuilt the binaries for 2.18 and before I push then to CTAN/TL, they are on SF:

https://sourceforge.net/projects/biblatex-biber/files/biblatex-biber/current/binaries/MacOS/

Can you get the universal binary from here and try it with non-absolute paths?

@herbs
Copy link

herbs commented Jul 6, 2022 via email

@plk
Copy link
Owner

plk commented Jul 6, 2022

@herbs - Did you do the usual unblocking routine via System Preferences? That isn't notarized and so will need to be manually authorized until it's in TL.

@herbs
Copy link

herbs commented Jul 6, 2022 via email

@amunn
Copy link

amunn commented Jul 6, 2022

Thanks, this binary works fine for me (Intel Mac).

@plk
Copy link
Owner

plk commented Jul 6, 2022

CTAN publishing will be done today and you should see 2.18-1 for MacOS in TL.

@herbs
Copy link

herbs commented Jul 6, 2022 via email

@herbs
Copy link

herbs commented Jul 8, 2022 via email

@plk
Copy link
Owner

plk commented Jul 8, 2022

Good, at least we know what the problem was and that it's fixed.

@plk plk closed this as completed Jul 8, 2022
@AndrewSwann
Copy link
Author

AndrewSwann commented Jul 25, 2022 via email

@herbs
Copy link

herbs commented Oct 11, 2022 via email

@herbs
Copy link

herbs commented Dec 17, 2022 via email

@leonmoonen
Copy link

yeah, it looks like I somehow ended with a mix of old and new versions of biber, and the path changed to prefer an old one...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@herbs @amunn @plk @leonmoonen @AndrewSwann