Allow any characters in filenames / labels #374

damienmg · 2015-08-13T08:24:10Z

Ultimately any character can be part of a filename. We should probably allow that.

Some mangling to generate the corresponding label should probably be done.

Original report on the mailing-list:
https://groups.google.com/d/msgid/bazel-discuss/CAN0GiO3__5jXo5rZqroSj0mFxpqCzUZZVkY%3DSNsJK1%2BZ1BdJLg%40mail.gmail.com

abergmeier · 2015-10-19T18:37:04Z

So are we talking Unicode?
Where does any character stop?
how do you treat characters that are not allowed on certain platforms?
When using mangling, how do you handle collisions?

kayasoze · 2015-10-19T18:59:37Z

In POSIX, filenames are "bags of bytes"--there is no encoding; however, NUL and / are not allowed. Windows has a few more restrictions. Perhaps the BUILD file should be parsed in the encoding of the system locale, usually UTF-8, and filenames run though a ValidForCurrentPlatform() function which checks for disallowed characters. However, opting for strict platform neutrality in this way means that Bazel would have to represent filenames as a bag of bytes and not a Unicode string, as there is no guarantee that the filename will roundtrip through Unicode correctly. The problem can probably be simplified by restricting filenames to be UTF-8 or UTF-16, which should cover most people's needs even though that's not strict POSIX.

ulfjack · 2015-10-22T11:05:41Z

Well, I think we can probably require valid UTF-8 file names and strongly recommend that people use UTF-8 for their file system. For labels / BUILD files, we probably need an escaping scheme, at least for the control characters. If there's a file that isn't valid UTF-8, we give an error message?

btelles · 2015-11-18T19:30:20Z

Our company codes mainly in C++, but our frontend uses a lot of JS and nodejs modules which have all sorts of characters in the filenames--for example, -, #, @, (, and ).

Right now this is a major blocker for getting all our codebase under one build system since we can't reference files with semi-special characters. I don't think Bazel should decide what characters are acceptable in file names, as that reduces file names to those that fit both (1) supported languages and (2) supported platforms. This seems unnecessarily restrictive, and is becoming a major pain point for us.

ulfjack · 2015-11-19T08:18:38Z

Agreed. Unfortunately, it's a bit tricky to fix, as a lot of code assumes that the mapping from labels to file names (and vice versa) is trivial, and doesn't require escaping. Any suggestions on an escaping scheme?

damienmg · 2015-11-19T09:02:46Z

URL based?

abergmeier · 2015-11-19T09:23:19Z

You mean an own URI scheme? Sounds good.

damienmg · 2015-11-19T09:25:17Z

I mean replacing special characters by %XX where XX is the UTF-8 code in hexa.

ulfjack · 2016-03-02T19:36:11Z

Sorry, I won't be able to work on this. @philwo had an interest, maybe he can make some progress here. :-/

kayasoze · 2016-03-22T17:16:30Z

This blocks our Bazel deployment as well.

RonnieAtOracle · 2016-03-22T17:20:02Z

This is blocking us. We have a templating system where we need to build our template files. The filenames themselves contains template variables (e.g. ${ServiceName}.java ). Both $, {, and } are not supported by Bazel in file names.

philwo · 2016-05-25T12:59:09Z

I totally agree that this is important, should be done, I want this myself, however I don't have the time to work on it in the coming months, thus I have to unassign it.

DemiMarie · 2016-09-01T17:14:51Z

Here is my proposal:

Metacharacters (:, %, =, any others) must be %-encoded
All other characters are allowed. This includes Unicode characters.
Non-UTF8 names are not allowed, even if escaped. This is because there is no good way to handle them cross-platform, nor to display them to the user.

mihnita · 2016-10-06T17:25:23Z

Plain ASCII (and even that partial) makes this feels like we are in the early 90s.

There are reasonable ways to handle that.
For POSIX using the default locale would be good enough.

If my project is C/C++, and it is cross-platform, and I have problems handling Unicode, then I will not use Unicode in file names. And the fact that bazel "explodes" is not such a problem.
But if I use something like Java, then "it just works", and bazel would work too.

Even better would be to to allow for a character-set option in the project file.
This is what maven does. And what Java does with -Dfile.encoding=UTF-8
So if it is there, use it. If not, then take the system charset.

I did not move one project to bazel because test units check that Unicode file names work.
So the files are there, make it through git, work with maven and ant and gradle and java.
But bazel fails because there is an "@" in the file name... Which is supported on all OSes.

There are 2 incompatibilities with the python install that pyenv creates: * empty directories - bazel's sandbox mode + rules_pkg doesn't work with them: bazelbuild/rules_pkg#135 * spaces in filenames - bazelbuild/bazel#374 So pass flags to glob() to exclude these sticking points, and you can now build a container (or tar) with rules_pyenv built python.

Bazel pep_deps rule can not handle files names with `=` in the name [1]. Unfortunately, there is no way to fix this issue in Bazel. [1] bazelbuild/bazel#374 Closes #12643 from iampat/iampat/fix_bazel_pep_import Authored-by: Ali Amiri <ali@amiri.dev> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

This is similar to 53e9d3e. Over at https://github.com/cockroachdb/cockroach, we use bazel and it chokes on this file: ``` java.io.IOException: Error extracting [...]/com_github_maruel_panicparse_v2-v2.2.1.zip to [...]: [...]/com_github_maruel_panicparse_v2/cmd/panic/internal/utf8/?tf8.go (Illegal byte sequence) ``` which is why we're on a fork so far. It would be nice to avoid this. There is an (eternal, it seems) [upstream discussion], so there is little home of bazel starting to support non-unicode filenames. I wonder if the loss in test coverage (if any) is acceptable in return for avoiding build problems like the ones we've been seeing. [upstream discussion]: bazelbuild/bazel#374

This is similar to 53e9d3e. Over at https://github.com/cockroachdb/cockroach, we use bazel and it chokes on this file: ``` java.io.IOException: Error extracting [...]/com_github_maruel_panicparse_v2-v2.2.1.zip to [...]: [...]/com_github_maruel_panicparse_v2/cmd/panic/internal/utf8/?tf8.go (Illegal byte sequence) ``` which is why we're on a fork so far. It would be nice to avoid this. There is an (eternal, it seems) [upstream discussion], so there is little home of bazel starting to support non-unicode filenames. [upstream discussion]: bazelbuild/bazel#374

adam-azarchs · 2023-03-21T17:36:45Z

I'm reconsidering what I said earlier about allowing :. Yes, it's in common usage in filenames in perl code, however given bazel is intended for cross-platform support, there's also an argument against supporting filenames which wouldn't be legal in Windows, specifically those containing any of <>:"|?*\. Most of those are also problematic in cases of careless rule authors writing shell scripts. Though there's a limit to how much bazel can protect them there¹, at least if " isn't legal in target names then it's always safe to "-quote them.

I'd really rather not to also prohibit ';&#()[]{} or whitespace, all of which are also potentially problematic for a carelessly written shell command. I think whitespace is probably the most important use case for loosening the current restrictions, as there are a lot of use cases where file names become user-visible and normal humans don't automatically translate _ into spaces. ↩

mihnita · 2023-03-23T18:04:05Z

given bazel is intended for cross-platform support, there's also an argument against supporting filenames which wouldn't be legal in Windows,

TLDR: I think it is not the job of the build tool to force naming conventions on a project

A project might be Linux and Mac only, no Windows.
Or might be Linux only, but using bazel just because they like it.
Or because use bazel because they us a mixture of targets / programming languages.
Or just because they know it (used it in other projects). Or want to learn.

The file naming rules would be selected by the developers looking at the big picture:
what they need to test, tools they use, version control tools, etc.
If each tool I use tries to force on me its own opinions on what's "the right thing" I end up with [A-Z0-9_]

Should a tool disallow the existence of the same file with different case?
(because there are case insensitive file systems)
If yes, what version of the case mapping table? (it can be system dependent)

Should a tool force a certain Unicode normalization?
Because the MacOS file system normalizes file names to decompose format.
(and what version of the normalization table?)

What about a Mac OS accessing a disk shared by a Linux machine using Samba (a Windows protocol)?

Long story short: not a bazel responsibility to force any limitations on what I can do or not.
IF bazel wants to help devs do "the right thing" then it can be as a validation step that can be disabled.
Not as a design limitation frozen in code.

adam-azarchs · 2023-03-24T01:48:15Z

I agree, mostly. My point was that there are reasons why we might want to/need to preserve some limitations on legal target names. Ideally any name would be supported, but if certain patterns are difficult to support, if they would anyway be illegal on some platforms that might be a sufficient excuse to not support them. I definitely feel like some of the proposed solutions start to get a bit convoluted.

thomaslangston-mongodb · 2023-03-27T14:00:30Z

Ideally any name would be supported, but if certain patterns are difficult to support, if they would anyway be illegal on some platforms that might be a sufficient excuse to not support them.

Illegal on some platforms is not a sufficient excuse to not support a filename pattern. Some filenames are system provided and cannot change to match the build system's preferences. Choosing some subset of valid filename patterns is equivalent to not having full support for any platform that is not encapsulated by the entire subset. We should be supporting the superset of all filename patterns instead.

atobiszei · 2024-02-21T09:53:38Z

Encountered this issue while using aws-sdk-cpp as a dependency. Luckily in that case it was only test file that we could safely remove with patch_cmds parameter in git_repository. Leaving it for anybody trying to use aws-sdk-cpp with bazel. It would be nice to see it resolved.

Additionally bump aws-sdk-cpp version to 1.11.279 Encountered issues with bazel. Non ASCII filenames don't work inside bazel. Workaround is to remove test file from aws-sdk-cpp. bazelbuild/bazel#374 Building with rules_foreign cmake doesn't work with parallel builds. This is something we could optimize later, for now hardcode some low value: bazelbuild/rules_foreign_cc#329 On Redhat to properly build targets we now have to add: --//:distro=redhat By default distro is set to ubuntu. This is workaround for not bazel not being able to differentiate between the two. There is difference in aws-sdk-cpp static libraries location after build. Ideally we should find how to use select on aws-sdk-cpp repo BUILD file which would rely on main repo flag, but I didn't find way to support it there. JIRA:CVS-130367

snakethatlovesstaticlibs · 2024-04-11T06:03:08Z

We're running into this in erlang 26 which bundles emoji filenames: https://github.com/erlang/otp/blob/master/lib/stdlib/test/edlin_expand_SUITE_data/.hidden%F0%9F%98%80_file

Reported upstream as not a bug: erlang/otp#8005

fmeum · 2024-04-11T07:27:58Z

@atobiszei @snakethatlovesstaticlibs Both your cases sound like you are in a slightly different situation: Files with complex file names are part of some external dependency you consume, but aren't relevant to its build (for example, because they are part of the test suite). The errors arise while extracting an archive. This use case should be fully supported in Bazel 6.4.0 and higher, could you give that a try?

snakethatlovesstaticlibs · 2024-04-11T16:50:42Z

@fmeum I worked around this by adding the test directories to an exclude in a filegroup, and it seems to be working now (because as you mentioned, the files are part of tests, so I don't need them)

For reference, the error I saw was:

ERROR: .../bazel_build/external/erlang/BUILD.bazel:10:15: Foreign Cc - Configure: Building erlang failed: error reading file '@@erlang//:erts/test/erlc_SUITE_data/src/😀/erl_test_unicode.erl': .../bazel_build/external/erlang/erts/test/erlc_SUITE_data/src/😀/erl_test_unicode.erl (No such file or directory)

sorry if this is unrelated to the issue in the original report, it sounded relevant while I was debugging

damienmg added type: bug P2 We'll consider working on this in future. (Assignee optional) labels Aug 13, 2015

kchodorow mentioned this issue Aug 31, 2015

Bazel doesn't like directories containing '+' #399

Closed

kchodorow assigned ulfjack Aug 31, 2015

kchodorow mentioned this issue Oct 19, 2015

Extracting file permission in zip fails in 0.1.1 #518

Closed

kchodorow mentioned this issue Nov 2, 2015

new_git_repository unable to handle Unicode characters in paths #551

Closed

ulfjack assigned philwo and unassigned ulfjack Mar 2, 2016

philwo removed their assignment May 25, 2016

damienmg added the category: misc > misc label Jun 14, 2016

damienmg modified the milestone: 0.7 Jun 14, 2016

dfabulich mentioned this issue Jun 20, 2016

Distributed caching: build fails when build inputs include directories #1415

Closed

nelhage mentioned this issue Aug 16, 2016

new_http_archive can't handle archives containing unicode-encoded filenames #1653

Closed

helenalt added the onbas-p1 label Aug 17, 2016

damienmg mentioned this issue Oct 31, 2016

# is not supported in srcs (duplicate of #374) #2006

Closed

lazcamus mentioned this issue Mar 10, 2022

make file globs compatible with rules_docker's py3_image() digital-plumbers-union/rules_pyenv#9

Merged

iampat mentioned this issue Mar 16, 2022

MINOR: [Python] Change a test file name for better bazel support apache/arrow#12643

Closed

tjgq mentioned this issue Apr 6, 2022

Wrong encoding of filename in bazel action inputs #14381

Closed

This was referenced May 11, 2022

Avoid unicode filename tbg/panicparse#1

Closed

Avoid unicode filename maruel/panicparse#78

Merged

giokara-oqton mentioned this issue Aug 1, 2022

Issue with Unicode characters in demo scripts streamlit/streamlit#5066

Closed

sfc-gh-tszerszen mentioned this issue Aug 1, 2022

remove unicode characters from demo script filenames streamlit/streamlit#5067

Closed

10 tasks

brandjon added untriaged team-Loading-API BUILD file and macro processing: labels, package(), visibility, glob and removed team-Build-Language untriaged labels Nov 4, 2022

nickbreen mentioned this issue Feb 16, 2023

source filenames containing colons #17509

Closed

atobiszei mentioned this issue Feb 21, 2024

Use bazel to build aws-sdk-cpp openvinotoolkit/model_server#2334

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow any characters in filenames / labels #374

Allow any characters in filenames / labels #374

damienmg commented Aug 13, 2015

abergmeier commented Oct 19, 2015

kayasoze commented Oct 19, 2015

ulfjack commented Oct 22, 2015

btelles commented Nov 18, 2015

ulfjack commented Nov 19, 2015

damienmg commented Nov 19, 2015

abergmeier commented Nov 19, 2015

damienmg commented Nov 19, 2015

ulfjack commented Mar 2, 2016

kayasoze commented Mar 22, 2016

RonnieAtOracle commented Mar 22, 2016

philwo commented May 25, 2016

DemiMarie commented Sep 1, 2016

mihnita commented Oct 6, 2016

adam-azarchs commented Mar 21, 2023

mihnita commented Mar 23, 2023

adam-azarchs commented Mar 24, 2023

thomaslangston-mongodb commented Mar 27, 2023

atobiszei commented Feb 21, 2024

snakethatlovesstaticlibs commented Apr 11, 2024 •

edited

Loading

fmeum commented Apr 11, 2024

snakethatlovesstaticlibs commented Apr 11, 2024 •

edited

Loading

Allow any characters in filenames / labels #374

Allow any characters in filenames / labels #374

Comments

damienmg commented Aug 13, 2015

abergmeier commented Oct 19, 2015

kayasoze commented Oct 19, 2015

ulfjack commented Oct 22, 2015

btelles commented Nov 18, 2015

ulfjack commented Nov 19, 2015

damienmg commented Nov 19, 2015

abergmeier commented Nov 19, 2015

damienmg commented Nov 19, 2015

ulfjack commented Mar 2, 2016

kayasoze commented Mar 22, 2016

RonnieAtOracle commented Mar 22, 2016

philwo commented May 25, 2016

DemiMarie commented Sep 1, 2016

mihnita commented Oct 6, 2016

adam-azarchs commented Mar 21, 2023

Footnotes

mihnita commented Mar 23, 2023

adam-azarchs commented Mar 24, 2023

thomaslangston-mongodb commented Mar 27, 2023

atobiszei commented Feb 21, 2024

snakethatlovesstaticlibs commented Apr 11, 2024 • edited Loading

fmeum commented Apr 11, 2024

snakethatlovesstaticlibs commented Apr 11, 2024 • edited Loading

snakethatlovesstaticlibs commented Apr 11, 2024 •

edited

Loading

snakethatlovesstaticlibs commented Apr 11, 2024 •

edited

Loading