TYPO3 is no longer installed in vendor-dir

[stucki]

According to 2d078ab, typo3/cms should always be installed in the vendor directory.

I just tried installing the latest master, but it installs TYPO3 into {$web-dir}/typo3/. In older versions, a symlink was created instead when I specified cms-package-dir.
According to the message in the commit above, this should now be the default. However it seems like this doesn't work as expected...

Here is my composer.json:

{
    "config": {
        "git-clone-depth": 100,
        "process-timeout": 900,
        "preferred-install": "dist",
        "optimize-autoloader": true
    },
    "minimum-stability": "dev",
    "prefer-stable": true,
    "repositories": [
        { "type": "composer", "url": "https://composer.typo3.org/" }
    ],
    "require": {
        "typo3/minimal": "dev-master",
        "typo3/cms-composer-installers": "dev-master"
    },
    "extra": {
        "typo3/cms": {
            "web-dir": "web"
        }
    }
}

[stucki]

@helhum Can you please take a look at this? Not sure if it's a bug or intended.

[helhum]

@stucki Hey. Thanks for your report. Since typo3/cms is not installable at all any more with current TYPO3 master (and composer installers 2.0), the configuration where to install typo3/cms is obsolete.

Previously typo3/cms was installed and then individual core extensions were symlinked to {$web-dir}/typo3/sysext (in fact the whole typo3 folder was symlinked).

Now that we use the subtree split packages, these are installed directly into the place TYPO3 expects them, which is {$web-dir}/typo3/sysext

So, yes, current behavior is intended.

[stucki]

Thanks for the feedback. Nevertheless, I'd see this as a missing feature then...

[helhum]

Nevertheless, I'd see this as a missing feature then...

Can you elaborate what exactly you miss?

[stucki]

I'm not sure if I just don't understand the concept or if I'm unaware of any planned features, but the status right now seems to be less powerful compared to before:

I would like to keep system extensions outside of the document root. It may not make a difference security-wise (because access was always possible via the symlink) but structure wise I like to have everything (or: as much as possible) that does not belong to my project in the vendor folder.

[stucki]

Related to this, I have a second question: Like you mention, typo3/cms is no longer installable. I have the possibility to require typo3/minimal metapackage, however this depends on a stable release, so I cannot use master with it right now.

For example, https://forge.typo3.org/issues/83479 was fixed a few days ago in master, but typo3/minimal depends on the last 9.0 release, so the fix is currently missing.

Is there an easy way to regain this possibility without explicitely having to specify all system extensions?

[helhum]

I'm not sure if I just don't understand the concept or if I'm unaware of any planned features, but the status right now seems to be less powerful compared to before

Well depends how you look at it.

I would like to keep system extensions outside of the document root

I agree :)

It may not make a difference security-wise (because access was always possible via the symlink) but structure wise I like to have everything (or: as much as possible) that does not belong to my project in the vendor folder

I also agree. However TYPO3 still requires core extensions to be installed in typo3/sysext/.
While we are striving to remove this limitation (in TYPO3 core), it currently still exists and there is nothing we can do about that within the composer installers.

The power of the approach having everything directly installed in typo3/sysext/ is that we won't require any symlinks to be created any more. The code is just there where it needs to be, just like for third party extensions which were always installed in typo3conf/ext/ by composer installers.

So yes, structure wise it isn't all too nice, but unavoidable when we want to avoid symlinks.
Other than theoretical issues with the structure, do you see any practical issues with that?

[helhum]

Related to this, I have a second question: Like you mention, typo3/cms is no longer installable. I have the possibility to require typo3/minimal metapackage, however this depends on a stable release, so I cannot use master with it right now

Using branches indeed is a bit more complicated. However from a composer point of view it makes sense. If you need a package in a branch, add this package as branch version to your composer.json. In your case you could add "typo3/cms-backend": "9.1.*@dev as 9.0.0" to pull this fix in (including other changes within that branch).

In general, if you need a fix for a project, I would always prefer to use a composer patch plugin over having to pull in a branch with composer.

[helhum]

Oh, and last thing. If you are concerned about security, with extension code being in document root, you may want to look at https://packagist.org/packages/helhum/typo3-secure-web

You can use this with TYPO3 8 and higher and using this will get you a directory structure with document root only containing public assets.

[helhum]

but unavoidable when we want to avoid symlinks

Hm, since I mentioned typo3-secure-web (which also uses symlinks to set up the document root),
we could indeed change composer installers to have the composer install path of all extension in vendor and create symlinks (and junctions on Windows) to the place TYPO3 requires them to be.

Shouldn't be too hard to do. Question would be. Should we do this by default (again having symlinks) or as option.

I'll re-open this here for now.

[stucki]

Thanks for the long answer.

So yes, structure wise it isn't all too nice, but unavoidable when we want to avoid symlinks.
Other than theoretical issues with the structure, do you see any practical issues with that?

I think there are no other drawbacks except this one.

Using branches indeed is a bit more complicated. However from a composer point of view it makes sense. If you need a package in a branch, add this package as branch version to your composer.json. In your case you could add "typo3/cms-backend": "9.1.*@dev as 9.0.0" to pull this fix in (including other changes within that branch).

Ok this will do, thanks. I'm still not too happy with it. I was hoping for a way to specifiy the version for all typo3/cms-* packages in a single constraint. However, I don't think that this is possible...

In general, if you need a fix for a project, I would always prefer to use a composer patch plugin over having to pull in a branch with composer.

Yes and no. The issue that I mentioned above breaks the backend for a new installation. It is definitely a bad end user experience if he expects to take a first look at the current master without being aware that this is actually 9.0 and not master...

Oh, and last thing. If you are concerned about security, with extension code being in document root, you may want to look at https://packagist.org/packages/helhum/typo3-secure-web

Looks interesting, thanks! But according to the README, it doesn't work with TYPO3 version 9. So what is correct?

Hm, since I mentioned typo3-secure-web (which also uses symlinks to set up the document root),
we could indeed change composer installers to have the composer install path of all extension in vendor and create symlinks (and junctions on Windows) to the place TYPO3 requires them to be.
Shouldn't be too hard to do. Question would be. Should we do this by default (again having symlinks) or as option.

If we can provide better security by default then I'd go for this. I know that there were issues with symlinks and I had some of them too while using Linux in Docker on Windows. However, the easy fix for this could be a configuration flag to turn symlinks on or off without any further auto-detection. What do you think?

I'll re-open this here for now.

Thanks!

[helhum]

If we can provide better security by default then I'd go for this

There is no impact on security if typo3 extensions need to be in document root for TYPO3 to work,
whether we install code directly there or just symlink code to be there.

[helhum]

Looks interesting, thanks! But according to the README, it doesn't work with TYPO3 version 9. So what is correct?

Here is the quote from the README: "While it would still be possible to require typo3/cms (the complete TYPO3 package) for TYPO3 version 8.7, it is not recommended any more and won't work with TYPO3 9."

So it is all about the typo3/cms package, which you can still use with TYPO3 8, but not with 9.
typo3-secure-web can deal with both.

[helhum]

However, the easy fix for this could be a configuration flag to turn symlinks on or off

Configuration is always easy to implement but not simple for users. Every option you hand out needs to be know and understood. I would avoid configuration wherever possible.

[helhum]

I gave this another thought and even started implementing some code, when I came (again) to the conclusion that every change would make the situation worse with no actual benefit.

Here are the summarized reasons:

1. No matter what we change here, there is no security benefit unless TYPO3 changes its requirements (or the mentioned package is used as mid term workaround)
2. Even if we install the code in vendor folder, we need to expose it for TYPO3 in typo3/sysext and typo3conf/ext, which requires symlinking or mirroring.
3. Since neither symlinking nor mirroring the code satisfies all needs (symlinks are an actual practical bummer with PHP on Windows), we would need to add a config option (again), which is a burden for users in contrast to what we have now, where it works reliably in all cases on all systems without any config option
4. The current state actually is the most consistent state we ever had in terms of data structure. All code is exactly where it needs to be for our application to run. No duplication/ symlinking needed. No more looking in more places for the code (either over the symlink or directly in vendor).
5. Last but not least, the code that needs to be added is significant. Not thousands of lines but a few hunderts of lines. Only to support something with no practical benefit, quite some practical constraints and only some minor theoretical/ structural benefit.

[helhum]

Regarding

typo3/cms is no longer installable

While I see the practical benefit in having only one line to change to get a different version or branch, it was a huge burden in multiple areas, that you only were able to get everything, even if you needed a fraction of the TYPO3 packages.

I'm sure we'll figure out something until 9LTS to make it easier to use branches, but for now you would need to change minimum-stablity to dev (which I wouldn't recommend without at the same time setting prefer-stable to true) or requiring dev branches for every package in your root composer.json

[bmack]

Hey @stucki,

there are quite some different issues in your ticket:

a) typo3/cms is not working with dev-master or v9 anymore. The main feature (!) for composer-mode installations is that typo3 is not downloading nor installing e.g. workspaces if you don't need it, thus actually "requiring" it.

b) Keeping as much as possible in the vendor/ dir - I agree. However, TYPO3 cannot handle this as of yet. We are aware but pushing on thing at a time. If you look at it with my argument above: now TYPO3 does not expose everything but only the required system extensions to the public anymore.

Actually, for composer, it does not matter where they are put, as the vendor directory should not be exposed for your application, also not the location of your packages themselves. But for TYPO3 it still matters.

I don't see why symlinking helps in any way to continue further down our road to only expose what we need, but only downsides we had with symlinks in the past.

My personal goal for v9 LTS would be that all active extensions are in one place (e.g. typo3conf/ext/) no matter if they are sysexts or not.

Once we figure out a "core" way to only have the public assets exposed in TYPO3 (don't get me wrong, helmuts secure-web is certainly great), which needs to fit to all systems (symlinking or not) and for non-composer mode, this is the best way we had in years.

[stucki]

Hi @helhum,

many thanks for your detailed answers. It took a few days for me to get back to this:

There is no impact on security if typo3 extensions need to be in document root for TYPO3 to work,
whether we install code directly there or just symlink code to be there.

Sorry, I misunderstood. I was assuming you want to include the typo3-secure-web functionality by default, but I guess that's not the case...

So it is all about the typo3/cms package, which you can still use with TYPO3 8, but not with 9. typo3-secure-web can deal with both.

I'm not sure if I got the point. To me it looks like replacing typo3/cms with small repositories has the side-effect that configuration now can't be controlled in the same nice way like it could be done before. A setting like vendor-dir would have to be provided for each repository separately, and that's why you don't implement it anymore. Is that correct?

Here are the summarized reasons:

No matter what we change here, there is no security benefit unless TYPO3 changes its requirements (or the mentioned package is used as mid term workaround)

Like I said, it's probably not a security problem, just inconsistency. Packages are usually installed in the vendor folder except if they are TYPO3 core parts or extensions... In the long term, I'd like to see TYPO3 having only index.php, resources and configuration in the document root, and the rest is stored entirely in vendor/. I'm aware that this is currently out of scope, however it feels like going one step back after we had stored typo3/cms in the vendor folder for quite some time before...

Even if we install the code in vendor folder, we need to expose it for TYPO3 in typo3/sysext and typo3conf/ext, which requires symlinking or mirroring.
Since neither symlinking nor mirroring the code satisfies all needs (symlinks are an actual practical bummer with PHP on Windows), we would need to add a config option (again), which is a burden for users in contrast to what we have now, where it works reliably in all cases on all systems without any config option

I appreciate that you focus on simplicity, however I don't see it as a problem to have a flag to control these things. Also I don't see too many other config options coming up...
Besides this, symlinks are also used by Composer when specifying repositories with local paths, and it works quite nice. (I don't know how it behaves on Windows, but on Linux the symlinks are used with no additional configuration needed...

The current state actually is the most consistent state we ever had in terms of data structure. All code is exactly where it needs to be for our application to run. No duplication/ symlinking needed. No more looking in more places for the code (either over the symlink or directly in vendor).

True. But that could still be the case if everything is moved into the vendor folder and included from a single entry point. (Again, Composer provides a nice example for how this could work using the wrapper in vendor/autoload.php...)

Last but not least, the code that needs to be added is significant. Not thousands of lines but a few hunderts of lines. Only to support something with no practical benefit, quite some practical constraints and only some minor theoretical/ structural benefit.

There's not much to say against this. The main point why I miss this is really just because of the structure.

[stucki]

Hello @bmack,

thanks for trying to clear things up!

a) typo3/cms is not working with dev-master or v9 anymore. The main feature (!) for composer-mode installations is that typo3 is not downloading nor installing e.g. workspaces if you don't need it, thus actually "requiring" it.

That's clear, see my previous comment. This is indeed a nice feature.

b) Keeping as much as possible in the vendor/ dir - I agree. However, TYPO3 cannot handle this as of yet. We are aware but pushing on thing at a time. If you look at it with my argument above: now TYPO3 does not expose everything but only the required system extensions to the public anymore.

True.

My personal goal for v9 LTS would be that all active extensions are in one place (e.g. typo3conf/ext/) no matter if they are sysexts or not.

Indeed this would be nice to have. However, see my other comment: It would be even nicer if all extensions are stored in vendor/ just like it's done for any other package...

Greetings, and thanks to both of you for sharing your thoughts!
Michael

[sypets]

I just found this thread. Thanks for providing the information. I would like to update the documentation.

Can you correct me, if I got anything wrong, I will try to summarize:

• the option extra -> cms-package-dir in composer.json no longer has any effect (since subtree split which is mandatory in TYPO3 9) and should not be used. (though it will not do any harm - however I like to remove options that have no effect).
• since subtree split, it is not possible to install TYPO3 system extensions in vendor directory. They will by default be installed in /typo3/sysext
• this may change in the future.
• the previous limitations can however be overridden by using e.g. helhum/secure-web

[helhum]

• the option extra -> cms-package-dir in composer.json no longer has any effect (since subtree split which is mandatory in TYPO3 9) and should not be used. (though it will not do any harm)

Correct

• since subtree split, it is not possible to install TYPO3 system extensions in vendor directory.

correct.

They will by default be installed in /typo3/sysext

They will be installed directly into the directory where TYPO3 needs them,
instead of installing them in the vendor folder and symlinking them to the directory where TYPO3 needs them.

We got rid of symlinks and only install the packages (TYPO3 system extensions) required, not always all, like it was before.

• this may change in the future.

Mid term plan is to "teach" TYPO3 to recognize extensions when they are located in vendor folder.
Then the installer can be removed and TYPO3 will use Composer default directory structure regarding packages.

• the previous limitations can however be overridden by using e.g. helhum/secure-web

Not sure what you mean by that. Sounds wrong to me.

First of all, I would not consider the above a "limitation".
In fact, these are all steps on the road to make TYPO3 a default Composer citizen,
thus these are step by step improvements in terms of consistency.

helhum/typo3-secure-web implements an additional folder, but the rest stays exactly the same:
no TYPO3 extensions in vendor folder, no symlinked packages.

[sypets]

Thanks for the answers. That was very helpful.

Not sure what you mean by that. Sounds wrong to me.
First of all, I would not consider the above a "limitation".

I was referring to a possible goal of moving things out of the web directory which do not have to be in the web directory and the goal of using vendor for TYPO3 system extensions (which would also move these out of the web directory). So there is some overlap between these goals and secure-web. But I admit the sentence was vague and not ideal.

I will strike both sentences entirely:

1. this may change in the future.
   Anything that may or may not change in the future can be discussed in a gist or decisions or some other place but I think it is not a good idea to put a vague sentence like that in the docs. Wasn't meant to be used like that.

2. the previous limitations can however be overridden by using e.g. helhum/secure-web
   This too is too vague and as you pointed out misleading and not entirely correct.

---

About "limitations" which are not - I think it is amazing how much work has been done and how smooth the transition to Composer can actually be. I was also quite pleased that I could find a lot of information. Just needs to get updated in some places which I am currently trying to help with.

[helhum]

Thanks for the answers. That was very helpful.

You are very welcomome.

But I admit the sentence was vague and not ideal.

No worries, really. There are many things here that are intertwined and I'm trying to explain
the different aspects and how they are related.

I was referring to a possible goal of moving things out of the web directory

That is one aspect. TYPO3 currently can not cope with that. It expects a directory structure
where almost all files have to live in the web directory. It is only two years ago, when
the official recommenddation to have all files in the web directory was dropped.
So the current state is, that in non Composer installations everything besides the third party
packages have to be in the web directory. In Composer installations var/* and config/sites live
outside the web directory, too. But typo3 and typo3conf still have to stay there.
This is deeply engrained in the code base, where it isn't distinguished between public and private
files, thus not easy to change.

This aspect is indeed tackled by helhum/typo3-secure-web, by providing a
web directory structure, which only contains public files and the PHP entry points.
This is done by leading TYPO3 to believe that typo3 and typo3conf are in the web directory,
but in fact are in a directory that isn't accessible by the web server.

the goal of using vendor for TYPO3 system extensions

This is another aspect. While being releated, it still is different.
The TYPO3 code base needs to be changed to allow extensions to be placed in other directories
than typo3/sysext and typo3conf/ext

Both aspects are intertwined, because once TYPO3 recognizes extensions in the vendor folder
(and this folder being outside the web directory), TYPO3 needs to "publish" the public assets
directory from these extensions to the web directory.

One aspect cannot be solved without solving the other.

I will strike both sentences entirely:

1. this may change in the future.
   Anything that may or may not change in the future can be discussed in a gist or decisions or some other place but I think it is not a good idea to put a vague sentence like that in the docs. Wasn't meant to be used like that.

I think it is inevitable that both aspects will be tackled eventually. But yes, there is no date or version number for that.

2. the previous limitations can however be overridden by using e.g. helhum/secure-web
   This too is too vague and as you pointed out misleading and not entirely correct.

About "limitations" which are not - I think it is amazing how much work has been done and how smooth the transition to Composer can actually be. I was also quite pleased that I could find a lot of information. Just needs to get updated in some places which I am currently trying to help with.

helhum/typo3-secure-web can be used to additionally secure current TYPO3 versions, by only exposing public assets to the web directory