New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TYPO3 is no longer installed in vendor-dir #75

Closed
stucki opened this Issue Jan 15, 2018 · 19 comments

Comments

Projects
None yet
3 participants
@stucki
Member

stucki commented Jan 15, 2018

According to 2d078ab, typo3/cms should always be installed in the vendor directory.

I just tried installing the latest master, but it installs TYPO3 into {$web-dir}/typo3/. In older versions, a symlink was created instead when I specified cms-package-dir.
According to the message in the commit above, this should now be the default. However it seems like this doesn't work as expected...

Here is my composer.json:

{
    "config": {
        "git-clone-depth": 100,
        "process-timeout": 900,
        "preferred-install": "dist",
        "optimize-autoloader": true
    },
    "minimum-stability": "dev",
    "prefer-stable": true,
    "repositories": [
        { "type": "composer", "url": "https://composer.typo3.org/" }
    ],
    "require": {
        "typo3/minimal": "dev-master",
        "typo3/cms-composer-installers": "dev-master"
    },
    "extra": {
        "typo3/cms": {
            "web-dir": "web"
        }
    }
}
@stucki

This comment has been minimized.

Member

stucki commented Jan 15, 2018

@helhum Can you please take a look at this? Not sure if it's a bug or intended.

@helhum

This comment has been minimized.

Contributor

helhum commented Jan 15, 2018

@stucki Hey. Thanks for your report. Since typo3/cms is not installable at all any more with current TYPO3 master (and composer installers 2.0), the configuration where to install typo3/cms is obsolete.

Previously typo3/cms was installed and then individual core extensions were symlinked to {$web-dir}/typo3/sysext (in fact the whole typo3 folder was symlinked).

Now that we use the subtree split packages, these are installed directly into the place TYPO3 expects them, which is {$web-dir}/typo3/sysext

So, yes, current behavior is intended.

@helhum helhum closed this Jan 15, 2018

@stucki

This comment has been minimized.

Member

stucki commented Jan 15, 2018

Thanks for the feedback. Nevertheless, I'd see this as a missing feature then...

@helhum

This comment has been minimized.

Contributor

helhum commented Jan 15, 2018

Nevertheless, I'd see this as a missing feature then...

Can you elaborate what exactly you miss?

@stucki

This comment has been minimized.

Member

stucki commented Jan 15, 2018

I'm not sure if I just don't understand the concept or if I'm unaware of any planned features, but the status right now seems to be less powerful compared to before:

I would like to keep system extensions outside of the document root. It may not make a difference security-wise (because access was always possible via the symlink) but structure wise I like to have everything (or: as much as possible) that does not belong to my project in the vendor folder.

@stucki

This comment has been minimized.

Member

stucki commented Jan 15, 2018

Related to this, I have a second question: Like you mention, typo3/cms is no longer installable. I have the possibility to require typo3/minimal metapackage, however this depends on a stable release, so I cannot use master with it right now.

For example, https://forge.typo3.org/issues/83479 was fixed a few days ago in master, but typo3/minimal depends on the last 9.0 release, so the fix is currently missing.

Is there an easy way to regain this possibility without explicitely having to specify all system extensions?

@helhum

This comment has been minimized.

Contributor

helhum commented Jan 15, 2018

I'm not sure if I just don't understand the concept or if I'm unaware of any planned features, but the status right now seems to be less powerful compared to before

Well depends how you look at it.

I would like to keep system extensions outside of the document root

I agree :)

It may not make a difference security-wise (because access was always possible via the symlink) but structure wise I like to have everything (or: as much as possible) that does not belong to my project in the vendor folder

I also agree. However TYPO3 still requires core extensions to be installed in typo3/sysext/.
While we are striving to remove this limitation (in TYPO3 core), it currently still exists and there is nothing we can do about that within the composer installers.

The power of the approach having everything directly installed in typo3/sysext/ is that we won't require any symlinks to be created any more. The code is just there where it needs to be, just like for third party extensions which were always installed in typo3conf/ext/ by composer installers.

So yes, structure wise it isn't all too nice, but unavoidable when we want to avoid symlinks.
Other than theoretical issues with the structure, do you see any practical issues with that?

@helhum

This comment has been minimized.

Contributor

helhum commented Jan 15, 2018

Related to this, I have a second question: Like you mention, typo3/cms is no longer installable. I have the possibility to require typo3/minimal metapackage, however this depends on a stable release, so I cannot use master with it right now

Using branches indeed is a bit more complicated. However from a composer point of view it makes sense. If you need a package in a branch, add this package as branch version to your composer.json. In your case you could add "typo3/cms-backend": "9.1.*@dev as 9.0.0" to pull this fix in (including other changes within that branch).

In general, if you need a fix for a project, I would always prefer to use a composer patch plugin over having to pull in a branch with composer.

@helhum

This comment has been minimized.

Contributor

helhum commented Jan 15, 2018

Oh, and last thing. If you are concerned about security, with extension code being in document root, you may want to look at https://packagist.org/packages/helhum/typo3-secure-web

You can use this with TYPO3 8 and higher and using this will get you a directory structure with document root only containing public assets.

@helhum

This comment has been minimized.

Contributor

helhum commented Jan 15, 2018

but unavoidable when we want to avoid symlinks

Hm, since I mentioned typo3-secure-web (which also uses symlinks to set up the document root),
we could indeed change composer installers to have the composer install path of all extension in vendor and create symlinks (and junctions on Windows) to the place TYPO3 requires them to be.

Shouldn't be too hard to do. Question would be. Should we do this by default (again having symlinks) or as option.

I'll re-open this here for now.

@helhum helhum reopened this Jan 15, 2018

@stucki

This comment has been minimized.

Member

stucki commented Jan 15, 2018

Thanks for the long answer.

So yes, structure wise it isn't all too nice, but unavoidable when we want to avoid symlinks.
Other than theoretical issues with the structure, do you see any practical issues with that?

I think there are no other drawbacks except this one.

Using branches indeed is a bit more complicated. However from a composer point of view it makes sense. If you need a package in a branch, add this package as branch version to your composer.json. In your case you could add "typo3/cms-backend": "9.1.*@dev as 9.0.0" to pull this fix in (including other changes within that branch).

Ok this will do, thanks. I'm still not too happy with it. I was hoping for a way to specifiy the version for all typo3/cms-* packages in a single constraint. However, I don't think that this is possible...

In general, if you need a fix for a project, I would always prefer to use a composer patch plugin over having to pull in a branch with composer.

Yes and no. The issue that I mentioned above breaks the backend for a new installation. It is definitely a bad end user experience if he expects to take a first look at the current master without being aware that this is actually 9.0 and not master...

Oh, and last thing. If you are concerned about security, with extension code being in document root, you may want to look at https://packagist.org/packages/helhum/typo3-secure-web

Looks interesting, thanks! But according to the README, it doesn't work with TYPO3 version 9. So what is correct?

Hm, since I mentioned typo3-secure-web (which also uses symlinks to set up the document root),
we could indeed change composer installers to have the composer install path of all extension in vendor and create symlinks (and junctions on Windows) to the place TYPO3 requires them to be.
Shouldn't be too hard to do. Question would be. Should we do this by default (again having symlinks) or as option.

If we can provide better security by default then I'd go for this. I know that there were issues with symlinks and I had some of them too while using Linux in Docker on Windows. However, the easy fix for this could be a configuration flag to turn symlinks on or off without any further auto-detection. What do you think?

I'll re-open this here for now.

Thanks!

@helhum

This comment has been minimized.

Contributor

helhum commented Jan 15, 2018

If we can provide better security by default then I'd go for this

There is no impact on security if typo3 extensions need to be in document root for TYPO3 to work,
whether we install code directly there or just symlink code to be there.

@helhum

This comment has been minimized.

Contributor

helhum commented Jan 15, 2018

Looks interesting, thanks! But according to the README, it doesn't work with TYPO3 version 9. So what is correct?

Here is the quote from the README: "While it would still be possible to require typo3/cms (the complete TYPO3 package) for TYPO3 version 8.7, it is not recommended any more and won't work with TYPO3 9."

So it is all about the typo3/cms package, which you can still use with TYPO3 8, but not with 9.
typo3-secure-web can deal with both.

@helhum

This comment has been minimized.

Contributor

helhum commented Jan 15, 2018

However, the easy fix for this could be a configuration flag to turn symlinks on or off

Configuration is always easy to implement but not simple for users. Every option you hand out needs to be know and understood. I would avoid configuration wherever possible.

@helhum

This comment has been minimized.

Contributor

helhum commented Jan 16, 2018

I gave this another thought and even started implementing some code, when I came (again) to the conclusion that every change would make the situation worse with no actual benefit.

Here are the summarized reasons:

  1. No matter what we change here, there is no security benefit unless TYPO3 changes its requirements (or the mentioned package is used as mid term workaround)
  2. Even if we install the code in vendor folder, we need to expose it for TYPO3 in typo3/sysext and typo3conf/ext, which requires symlinking or mirroring.
  3. Since neither symlinking nor mirroring the code satisfies all needs (symlinks are an actual practical bummer with PHP on Windows), we would need to add a config option (again), which is a burden for users in contrast to what we have now, where it works reliably in all cases on all systems without any config option
  4. The current state actually is the most consistent state we ever had in terms of data structure. All code is exactly where it needs to be for our application to run. No duplication/ symlinking needed. No more looking in more places for the code (either over the symlink or directly in vendor).
  5. Last but not least, the code that needs to be added is significant. Not thousands of lines but a few hunderts of lines. Only to support something with no practical benefit, quite some practical constraints and only some minor theoretical/ structural benefit.

@helhum helhum closed this Jan 16, 2018

@helhum

This comment has been minimized.

Contributor

helhum commented Jan 16, 2018

Regarding

typo3/cms is no longer installable

While I see the practical benefit in having only one line to change to get a different version or branch, it was a huge burden in multiple areas, that you only were able to get everything, even if you needed a fraction of the TYPO3 packages.

I'm sure we'll figure out something until 9LTS to make it easier to use branches, but for now you would need to change minimum-stablity to dev (which I wouldn't recommend without at the same time setting prefer-stable to true) or requiring dev branches for every package in your root composer.json

@bmack

This comment has been minimized.

bmack commented Jan 21, 2018

Hey @stucki,

there are quite some different issues in your ticket:

a) typo3/cms is not working with dev-master or v9 anymore. The main feature (!) for composer-mode installations is that typo3 is not downloading nor installing e.g. workspaces if you don't need it, thus actually "requiring" it.

b) Keeping as much as possible in the vendor/ dir - I agree. However, TYPO3 cannot handle this as of yet. We are aware but pushing on thing at a time. If you look at it with my argument above: now TYPO3 does not expose everything but only the required system extensions to the public anymore.

Actually, for composer, it does not matter where they are put, as the vendor directory should not be exposed for your application, also not the location of your packages themselves. But for TYPO3 it still matters.

I don't see why symlinking helps in any way to continue further down our road to only expose what we need, but only downsides we had with symlinks in the past.

My personal goal for v9 LTS would be that all active extensions are in one place (e.g. typo3conf/ext/) no matter if they are sysexts or not.

Once we figure out a "core" way to only have the public assets exposed in TYPO3 (don't get me wrong, helmuts secure-web is certainly great), which needs to fit to all systems (symlinking or not) and for non-composer mode, this is the best way we had in years.

@stucki

This comment has been minimized.

Member

stucki commented Jan 22, 2018

Hi @helhum,

many thanks for your detailed answers. It took a few days for me to get back to this:

There is no impact on security if typo3 extensions need to be in document root for TYPO3 to work,
whether we install code directly there or just symlink code to be there.

Sorry, I misunderstood. I was assuming you want to include the typo3-secure-web functionality by default, but I guess that's not the case...

So it is all about the typo3/cms package, which you can still use with TYPO3 8, but not with 9. typo3-secure-web can deal with both.

I'm not sure if I got the point. To me it looks like replacing typo3/cms with small repositories has the side-effect that configuration now can't be controlled in the same nice way like it could be done before. A setting like vendor-dir would have to be provided for each repository separately, and that's why you don't implement it anymore. Is that correct?

Here are the summarized reasons:

No matter what we change here, there is no security benefit unless TYPO3 changes its requirements (or the mentioned package is used as mid term workaround)

Like I said, it's probably not a security problem, just inconsistency. Packages are usually installed in the vendor folder except if they are TYPO3 core parts or extensions... In the long term, I'd like to see TYPO3 having only index.php, resources and configuration in the document root, and the rest is stored entirely in vendor/. I'm aware that this is currently out of scope, however it feels like going one step back after we had stored typo3/cms in the vendor folder for quite some time before...

Even if we install the code in vendor folder, we need to expose it for TYPO3 in typo3/sysext and typo3conf/ext, which requires symlinking or mirroring.
Since neither symlinking nor mirroring the code satisfies all needs (symlinks are an actual practical bummer with PHP on Windows), we would need to add a config option (again), which is a burden for users in contrast to what we have now, where it works reliably in all cases on all systems without any config option

I appreciate that you focus on simplicity, however I don't see it as a problem to have a flag to control these things. Also I don't see too many other config options coming up...
Besides this, symlinks are also used by Composer when specifying repositories with local paths, and it works quite nice. (I don't know how it behaves on Windows, but on Linux the symlinks are used with no additional configuration needed...

The current state actually is the most consistent state we ever had in terms of data structure. All code is exactly where it needs to be for our application to run. No duplication/ symlinking needed. No more looking in more places for the code (either over the symlink or directly in vendor).

True. But that could still be the case if everything is moved into the vendor folder and included from a single entry point. (Again, Composer provides a nice example for how this could work using the wrapper in vendor/autoload.php...)

Last but not least, the code that needs to be added is significant. Not thousands of lines but a few hunderts of lines. Only to support something with no practical benefit, quite some practical constraints and only some minor theoretical/ structural benefit.

There's not much to say against this. The main point why I miss this is really just because of the structure.

@stucki

This comment has been minimized.

Member

stucki commented Jan 22, 2018

Hello @bmack,

thanks for trying to clear things up!

a) typo3/cms is not working with dev-master or v9 anymore. The main feature (!) for composer-mode installations is that typo3 is not downloading nor installing e.g. workspaces if you don't need it, thus actually "requiring" it.

That's clear, see my previous comment. This is indeed a nice feature.

b) Keeping as much as possible in the vendor/ dir - I agree. However, TYPO3 cannot handle this as of yet. We are aware but pushing on thing at a time. If you look at it with my argument above: now TYPO3 does not expose everything but only the required system extensions to the public anymore.

True.

My personal goal for v9 LTS would be that all active extensions are in one place (e.g. typo3conf/ext/) no matter if they are sysexts or not.

Indeed this would be nice to have. However, see my other comment: It would be even nicer if all extensions are stored in vendor/ just like it's done for any other package...

Greetings, and thanks to both of you for sharing your thoughts!
Michael

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment