StreamWrapper support for glob()

[timint]

Description

glob() doesn't support StreamWrappers. It does surprisingly return an empty array instead of triggering an error or notice when passing a stream other than file://.

I couldn't find that any discussion has been brought up before regarding the topic.
Is glob() fit the way it is, or should it have streamwrapper support in the future?

[cmb69]

PHP's glob() function is a relatively thin wrapper over glob(3). The result in case of no matches somewhat depends on the implementation, but PHP favors to return an empty array:

php-src/ext/standard/dir.c

Lines 453 to 468 in 5370f34

	if (0 != (ret = glob(pattern, flags & GLOB_FLAGMASK, NULL, &globbuf))) {
	#ifdef GLOB_NOMATCH
	if (GLOB_NOMATCH == ret) {
	/* Some glob implementation simply return no data if no matches
	were found, others return the GLOB_NOMATCH error code.
	We don't want to treat GLOB_NOMATCH as an error condition
	so that PHP glob() behaves the same on both types of
	implementations and so that 'foreach (glob() as ...'
	can be used for simple glob() calls without further error
	checking.
	*/
	goto no_results;
	}
	#endif
	RETURN_FALSE;
	}

Anyhow, I don't think we should add support for arbitrary stream wrappers. In some cases it just makes no sense (e.g. compress.zlib:// and data://), in some cases it is impossible (e.g. http://), and in some cases it still might be impractical (e.g. ftp://). For the others, we still would need to implement the logic ourselves instead of just calling glob(3) (although we do that on Windows already).

[timint]

Thanks for your input on this. Ok so glob() quite much mirrors and promotes the behaviors and features of Linux glob(3). And implementing new possibilities might make it seem a bit off course or inconsistent. It might be a better idea to suggest a new function that isn't bound to the nature of glob(). StreamWrappers is an awesome thing in PHP. Supported by opendir(), readdir(), scandir() and many other file functions. I'm just hoping we can close the gap on where it's not usable. On a side note, a new function could also have new features like double globstar.

[KapitanOczywisty]

Not sure about what StreamWrappers are you talking, but you can do that with Iterators and Regex, eg:

$Directory = new RecursiveDirectoryIterator('somewrapper:///my/path/to/directory');
$Iterator = new RecursiveIteratorIterator($Directory);
$Regex = new RegexIterator($Iterator, '/^.+\.php$/i');

And with regex pattern being more powerful than glob, I wouldn't wait for glob equivalent with StreamWrappers.

[bukka]

Not sure if you are aware of that but there is already glob stream wrapper which sort of does what you are asking for. Passing stream to the glob function doesn't really make much sense because it accepts pattern which is not a file path.

In addition you might also look to GlobIterator which makes use of glob wrapper and provides iteration of the glob result.

If you still think there are still some missing use cases, please elaborate.

[KapitanOczywisty]

@bukka Both glob:// and GlobIterator are performed on local filesystem and are using glob() internally, so nothing is changing for the issue author. There are legitimate reasons to find files on remote (e.g. ftp, ssh2.sftp) or virtual filesystems, but still I think existing iterators are sufficient to do this job.

[bukka]

Ok that would be quite a bit job though as we would need to re-implement glob on other platforms than Windows. It's really useful just for few additional stream wrappers like ftp which is probably no longer that much used and ssh2 which is not even in the core so not sure if there's a big value in it...

[timint]

@KapitanOczywisty That looks like something nice and lightweight that could be useful as an alternative to dual globstars. But wouldn't the DirectoryIterator go through everything recursively? In the long run I rather see something friendly to the system resources. 🙂

Not sure about what StreamWrappers are you talking

Any one of them. :) There is not one that plays with glob. In my specific case I implemented a virtual file system in a framework that works like a charm. But I had to swap out all uses of glob() in the platform. I tried substituting it with scandir() and fnmatch(). But fnmatch does not support brace expansions. And so I ended up reinventing glob, but for streamwrappers.

@bukka Yeah no, this topic is not about accessing glob() via a streamwrapper. It's the other way around; accessing the streamwrapper via glob, e.g. glob('virtualfs://*').

[KapitanOczywisty]

But wouldn't the DirectoryIterator go through everything recursively? In the long run I rather see something friendly to the system resources. 🙂

Any potential glob implementation for virtual filesystem would do the same, with the exception that Iterators are not loading everything at once, just reading more data until next step is satisfied e.g. RegexIterator will fetch data sufficient to return next match. Since regex is evaluated for the every file in directory, for slow filesystems or big number of files, you may need to do some additional filtering with own RecursiveIteratorIterator and remember to use longest possible path for RecursiveDirectoryIterator. However, best to measure first before getting into any optimizations 😉

Also I have no idea if glob on local filesystem employs any optimizations or path traversing is just fast (which is more likely).

As a side note there is user-land implementation of glob with support for StreamWrappers, and seems to work on mentioned above principles with RecursiveDirectoryIterator and some minor optimizations.

[timint]

@KapitanOczywisty If I wanted to grab all subfolders containing x like glob('wrapper://path/to/app/*x*/', GLOB_ONLYDIR). Would this be what you are suggesting:?

$Directory = new RecursiveDirectoryIterator('wrapper://path/to/app/');
$Iterator = new RecursiveIteratorIterator($Directory);
$Regex = new RegexIterator($Iterator, '#^wrapper://path/to/app/[^/]*x[^/]*/$#i');

Wouldn't that traverse through the files and folders of wrapper://path/to/app/cache/?

Yeah I have seen webmozarts implementation of glob. It's a good example of a workaround needed for what we are missing.

[KapitanOczywisty]

Wouldn't that traverse through the files and folders of wrapper://path/to/app/cache/?

Yes It will. In that case only FilesystemIterator is needed. webmozart/glob splits pattern by slashes so it can skip traversing into the folder.

My original example, was made just to present alternative, but at this point I have to make some notes, since It's not how it would be used in the wild:

• RecursiveIteratorIterator by default returns files only (RecursiveIteratorIterator::LEAVES_ONLY), which means you need ::SELF_FIRST or ::CHILD_FIRST to match also directories

• in most cases you don't want paths like wrapper://path/to/app/cache/.. (with dots) to be tested, so add FilesystemIterator::SKIP_DOTS to RecursiveDirectoryIterator

• RegexIterator without flags matches by value, which in this case is SplFileInfo casted to string before performing match. You may want to add flags: RegexIterator::USE_KEY to RegexIterator constructor.

• To match only filename, use FilesystemIterator::KEY_AS_FILENAME in RecursiveDirectoryIterator, just be aware that keys may repeat, so make sure to pass preserve_keys: false in iterator_to_array.

• There is no slash at the end in pathname.

So for example to match files and directories with test:

$Directory = new RecursiveDirectoryIterator('wrapper://path/to/app/', FilesystemIterator::SKIP_DOTS | FilesystemIterator::KEY_AS_FILENAME);
$Iterator = new RecursiveIteratorIterator($Directory, RecursiveIteratorIterator::SELF_FIRST);
$Regex = new RegexIterator($Iterator, '/test/i', flags: RegexIterator::USE_KEY);

Going back to your example, it would go something like this:

$Directory = new FilesystemIterator('wrapper://path/to/app/', FilesystemIterator::SKIP_DOTS | FilesystemIterator::KEY_AS_FILENAME);
$Regex = new RegexIterator($Directory, '/x/i', flags: RegexIterator::USE_KEY);
// keep only directories
$Iterator = new CallbackFilterIterator($Regex, fn($current, $key, $iterator) => $current->isDir());

It's not exactly lightweight anymore, however packed into some function can be easy to reuse. I'm personally don't mind that, but if anyone do, use webmozart/glob.

[timint]

KapitanOczywisty Yeah, pattern separated by slashes is how I solved it too. My workaround is here.

Looking at win32/glob.c I think it looks well made. Wouldn't it be worth considering if future PHP glob() could be all based on the win32 make, instead of the thin wrapper for Linux glob(3). That way we could have a standalone globbing implementation in the nature of "globbing" rather than locking ourselves in a corner by wrapping "glob(3)".

[cmb69]

Wouldn't it be worth considering if future PHP glob() could be all based on the win32 make, instead of the thin wrapper for Linux glob(3).

I'm not against that (Windows user speaking), but I think that requires discussion on the internals mailing list, and likely an RFC. Do you want to pursue this process?