Skip verification of dependencies in compile time. How ?

[loguntsov]

Didn't found way to skip verification of dependencies in compile time. How i can do that ?
I think it is important feature which allow save much time.

What is right flow for debugging of code ?

Thanks.

[ferd]

What do you mean by skipping verifications of dependencies? Rebar3 currently does not allow that to happen if I interpret it as 'skipping making sure the deps are in place and compiled'.

[loguntsov]

Yes, you interpreted right. Rebar2 had option to skip it, but rebar3 have no this option, and i should spend my time for every run

[ferd]

There isn't a way to do it right now and I don't think we have any way planned for it at the moment. I don't think we would be closed to a way to work around it, but all the commands in rebar3 may declare one or more dependencies that get executed before their own provider. As such, if the dialyzer task requires compiled application and declares it needs the compiler to run before itself, there's no clear path to saying "skip the compile step".

[tsloughter]

@loguntsov it doesn't take anytime. It only verifies that that they exist before compiling your apps that depend on them.

[ferd]

I'll close this one as I don't think this will be happening, since it does not fit the current design. The bare compiler as mix uses may be opening the way there, but that's probably as far as this goes in terms of current plans.

[Zalastax]

@ferd can you highlight what the troublesome areas are to make this happen? Is it just a matter of putting the time in or is it something you are fundamentally opposed to?

[tsloughter]

I'm fundamentally opposed unless it can be shown to be an issue. Maybe on projects with an extremely large number of dependencies it can be shown there is N seconds of time spent that is unacceptable. That could change my opposition.

[loguntsov]

Couple times of runs for debugging:

make local | ts '[%Y-%m-%d %H:%M:%S]'

[2018-09-30 22:13:00] rebar3 as `cat env` shell
[2018-09-30 22:13:00] ===> Verifying dependencies...
[2018-09-30 22:13:00] ===> Package <<"rebar_gdb_plugin">> not found. Fetching registry updates and trying again...
[2018-09-30 22:13:00] ===> Updating package registry...
[2018-09-30 22:13:01] ===> Writing registry to /home/begemot/.cache/rebar3/hex/default/registry
[2018-09-30 22:13:01] ===> Generating package index...
[2018-09-30 22:13:01] ===> [appsignal:1.6.2], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:02] ===> [appsignal:1.6.6-beta.1], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:02] ===> [appsignal:1.6.0], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:03] ===> [appsignal:1.7.0-alpha.4], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:03] ===> [appsignal:1.6.0-beta.1], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:03] ===> [appsignal:1.6.3], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:04] ===> [appsignal:1.7.0-alpha.3], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:04] ===> [appsignal:1.7.0-alpha.2], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:05] ===> [appsignal:1.7.0-alpha.1], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:06] ===> [appsignal:1.6.5], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:06] ===> [appsignal:1.6.1], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:06] ===> [appsignal:1.6.6-beta.2], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:07] ===> [appsignal:1.6.6], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:07] ===> [appsignal:1.6.7], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:07] ===> [appsignal:1.6.4], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:07] ===> [appsignal:1.6.0-alpha.1], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:13:07] ===> Writing index to /home/begemot/.cache/rebar3/hex/default/packages.idx
[2018-09-30 22:13:07] ===> Plugin rebar_gdb_plugin not available. It will not be used.
[2018-09-30 22:13:14] ===> Compiling client_test
[2018-09-30 22:13:14] ===> Compiling ctv
[2018-09-30 22:13:16] Eshell V8.3.5  (abort with ^G)

make local | ts '[%Y-%m-%d %H:%M:%S]'
[2018-09-30 22:15:40] rebar3 as `cat env` shell
[2018-09-30 22:15:41] ===> Verifying dependencies...
[2018-09-30 22:15:41] ===> Package <<"rebar_gdb_plugin">> not found. Fetching registry updates and trying again...
[2018-09-30 22:15:41] ===> Updating package registry...
[2018-09-30 22:15:43] ===> Writing registry to /home/begemot/.cache/rebar3/hex/default/registry
[2018-09-30 22:15:43] ===> Generating package index...
[2018-09-30 22:15:44] ===> [appsignal:1.6.2], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:45] ===> [appsignal:1.6.6-beta.1], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:45] ===> [appsignal:1.6.0], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:45] ===> [appsignal:1.7.0-alpha.4], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:45] ===> [appsignal:1.6.0-beta.1], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:46] ===> [appsignal:1.6.3], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:46] ===> [appsignal:1.7.0-alpha.3], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:47] ===> [appsignal:1.7.0-alpha.2], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:47] ===> [appsignal:1.7.0-alpha.1], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:48] ===> [appsignal:1.6.5], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:48] ===> [appsignal:1.6.1], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:48] ===> [appsignal:1.6.6-beta.2], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:49] ===> [appsignal:1.6.6], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:49] ===> [appsignal:1.6.7], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:50] ===> [appsignal:1.6.4], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:50] ===> [appsignal:1.6.0-alpha.1], Bad dependency version for httpoison: ~> 0.11 or ~> 1.0.
[2018-09-30 22:15:50] ===> Writing index to /home/begemot/.cache/rebar3/hex/default/packages.idx
[2018-09-30 22:15:50] ===> Plugin rebar_gdb_plugin not available. It will not be used.
[2018-09-30 22:15:51] ===> Compiling client_test
[2018-09-30 22:15:51] ===> Compiling ctv
[2018-09-30 22:15:51] Eshell V8.3.5  (abort with ^G)
[2018-09-30 22:15:51] 1> ===> The rebar3 shell is a development tool; to deploy applications in production, consider using releases (http://www.rebar3.org/docs/releases)

@tsloughter you see i'm wasting 10-15 seconds to check dependencies. it's going to the server to check something every time. So i wanna skip this step.

[tsloughter]

If the time is spent on fetching the repo because of a missing plugin why aren't you removing rebar_gdb_plugin? If it is listed in a dependency you can use overrides to replace the deps plugins list without that plugin.

[loguntsov]

Because i just want skip this step for my debugging. I know project is compiled, and i want just run it. I don't need check something everytime when i need launch it. Just give us this way. What we are asking you.
Overides is not good way any way, because dependency can be changed later, something can be fixed inside this dependency, but you never will know about this fact and never will fix your overide. So i prefer don't use it anyway

[Zalastax]

For me it's about dependencies that are built somewhere else and where the result of the compilation will be bundled with the rest later. We will probably come up with a work around but I wanted to first probe to see if there was any willingness to merge such a solution upstream.

[tsloughter]

@loguntsov your issue won't be solved by skipping dep verification, it is a plugin that is your issue.

[loguntsov]

I had many projects since start of this topic, it is not the same. Many projects have this trouble. It doesn't mean that i should fix all them by overides. Another question from other point. How i can start application (like shell) without something else like dependencies and others ? Yes i can use simple erl command, but what about rebar3 with his profiles and another possibilities.

[tsloughter]

@Zalastax oh, hm, that is an interesting case. But if the dependencies aren't needed when building why have them in the deps list at all?

It is possible we'd accept a PR that allows for skipping dependencies. But keep in mind that it needs to not cause issues with providers that compile also depends on, like lock. I'm not sure what the solution would be in that case if the deps aren't resolved then it can't lock anything but it also shouldn't overwrite the possibly existing lock file with an empty list.

Another issue is paths. rebar3 doesn't simply add everything in _build/default/lib/ to the code path, it only adds dependencies it knows about. So if dependencies are skipped in a naive way it would mean parse transforms or headers from deps that are needed for compilation of the project will not be in the code path.

Those issues and possibly others would have to be figured out if it were to be a feature in rebar3. On the other hand a plugin would be simple.

Have a project_plugin that overrides the compile command, copy the compile provider to this new plugin and replace:

-define(DEPS, [lock]).

with

-define(DEPS, [app_discovery]).

Using that plugin as a project_plugin means you can still just run rebar3 compile, or you can optionally have the plugin use a different provider name or namespace, like rebar3 skipdeps compile.

[tsloughter]

@loguntsov do you mean how to start the shell even if the project isn't compiling?

That is something I've wanted to fix as I've found it annoying at times. However if you just want it to not start applications you can do rebar3 shell --start-clean.

Because of the way rebar3 provider dependencies work it isn't so easy to have a version of shell that doesn't compile the project, so the way right now is to use rebar3 path with erl. For example:

erl -pa `rebar3 as test path`

That will bring up a shell with everything, including tests and test deps, in the code path.

[Zalastax]

@tsloughter we primarily need the source files from the dependencies. Resolving the dependencies but then not building them seemed like the smoothest and fastest solution but we are also looking into creating a plugin that downloads our built artifacts instead.
I'll take your info back to the team - it will be very helpful when we decide on how to proceed!

[ferd]

@loguntsov

Because i just want skip this step for my debugging. I know project is compiled, and i want just run it. I don't need check something everytime when i need launch it. Just give us this way. What we are asking you.

No. As Tristan mentioned, the entire Rebar3 project works by having tasks depend on other tasks to work. The deps handling task loads the applications in the path, the lock task takes the deps and ensures they're well-represented in the lock file, and the compile task is what ensures they are compiled (if they are compiled they do not get recompiled -- if they do, that's a different bug).

This is automated because compile depends on lock, which depends on get-deps, which depends on application_discovery (finds top level apps). Asking to 'compile without caring about the deps' is essentially asking to change how the entire rebar3 model works.

You know the project is debugged, but Rebar3 does not. It needs to check and ensure that all artifacts are there for a build so that you don't get a crash every time, and that it can take corrective action otherwise. That's why when you call rebar3 compile, it knows what needs to be fetched and updated, and you don't need to tell it to get the deps by hand every time.

That was a conscious decision made there.

As Tristan says, if we can find a way to 'skip the compilation of deps' then sure that could be done, but it doesn't mean you could skip it when calling rebar3 shell or rebar3 release because these tasks do not understand compiling arguments: they just ask for the files to be compiled to work. The only way to configure these tasks then is through the rebar.config file.

You can come in and ask to "just give you the feature", but what I'm telling you is that doing that would not necessarily accomplish much of what you want. Like sure you can now run rebar3 compile without compiling the deps, but if you want to use the shell to debug, it still compiles. Because the shell asks for compiled deps and they need a check.

@Zalastax this is what I meant by "does not fit the current design".

As Tristan mentioned, rebar3 does not grab everything in _build/<profile>/lib to run, it only adds in the dependencies you need according to your config. This prevents us from having to go manually remove old applications that are no longer specified, and basically makes rebar3 work context-free -- it won't mistakenly load dependencies when switching branches with no way to know what was used or not, and should drastically reduce how often people need to 'rm -rf _build && rebar3 compile' every time they check out a new code branch

[Zalastax]

Thanks for all the insight you have provided here. The suggestion by @tsloughter is looking fruitful and I'm confident my team will be able to make the pipeline fit our needs. I'm not sure if we can avoid downloading the artifacts but we'll be able to avoid recompiling them. We'll investigate some more and I'll reach out to you if we think it can be integrated with the rest of the ecosystem.

[ferd]

One other thing to note is that if an app has beam files in a top-level ebin/ directory, those are copied as-is to the build directory.