Client-supplied bootstrappers can override the default behaviour when scanning assemblies for extensions #130

Closed
blairconrad opened this Issue Jun 11, 2013 · 54 comments

Projects

None yet

6 participants

@blairconrad
Member

The ImportsModule currently scans all assemblies in the current AppDomain and current directory for implementations of IArgumentValueFormatter, IDummyDefinition, and IFakeConfigurator. While this is an elegant way of discovering user-supplied implementations, it comes with a few drawbacks for a certain class of projects.

  1. The scanning can be slow. On a large project that I work on, we have nearly 100 assemblies in the unit test build directory. This is not ideal for a variety of reasons, but it's not something that's going to change soon. Scanning these for custom implementations of the above-mentioned interfaces takes several seconds, which is not a big deal for the automated build, but when trying to run just a few unit tests, it introduces an annoying delay.
  2. Scanning all the DLLs in the directory can trigger a LoaderLock, as managed code is exercised at an inopportune time. The popup can be turned off in the IDE, but it's still an annoyance.
  3. Worse, with certain inputs (boost seems to be a cause of this problem, but we're investigating that separately), there's a Runtime error R6033 caused by attempting to use MSIL code during native code intialization. This popup breaks the JetBrains.ReSharper.TestRunner.CLR4.exe, causing it never to terminate. The user has to kill the process before running another test.
    image

I realize that the last two of these problems are not caused by FakeItEasy, but they are exacerbated by its scanning behaviour.

I propose allowing FakeItEasy users to configure the amount of scanning done to find helpers. I'm keen to work on the issue, and would welcome some guidance so the feature fits in with the FakeItEasy philosophy.

Design options I've considered:

  1. Having FakeItEasy read a configuration file for the assemblies it should scan.
  2. Allowing explicit programmatic configuration of the assemblies to scan, possibly via a static method. This would be most useful when run under unit testing frameworks that allow one to ensure that code runs before any tests, à la NUnit's SetupFixture attribute
  3. Performing very limited-scope discovery for a class that extends another bootstrapping class, similar to the way Nancy's bootstrapper is replaced.

The goal with any of these approaches would be to ensure that ease of configuration is combined with the need not to disrupt existing usage. For example, if there was no override in a configuration file or in a custom bootstrapper, the existing "broad scan" behaviour would be preserved.

@philippdolder
Member

sounds reasonable to me to have a way to limit the discovery of the extension types.
I would go for a way in code (2), not in config files (1) and as @blairconrad suggests don't need any change to existing code bases.

Atm I have no clue how I would implement this and I won't have time this week to look into it.

@FakeItEasy/owners thoughts?

@adamralph
Member

I think this is a good idea. My initial reaction is to go with option 3. I've been doing a lot of work with Nancy recently so I'm quite familiar with that approach. It's a very elegant way of doing things. I'm not yet sure what 'shape' this type might take. I'd have to take a look at the scanning code more closely before offering any opinion on that. Perhaps later today.

@blairconrad
Member

I'm heartened by the support. @adamralph, I prefer option 3 (discoverable bootstrapper) as well. The only hiccough that I see is that Nancy's bootstrap locator scans the entire AppDomain looking for the first overriding implementation. For my purposes, this is better than scanning the AppDomain and the whole working directory, but still problematic. I've not yet had a chance to explore how to scan just the test assembly. I'm going to have to travel until late Sunday, but will be back on the case after that.

I guess another question would be whether to work only on the discovery of the extension types or to expand to a more comprehensive configuration system where many other facets could be replaced. I'm inclined to start small and see how it goes, but am happy to look into a more ambitious solution as well.

@adamralph
Member

I think this should be something general so that it can allow overriding of all kinds of behaviour. I imagine it looking something like DefaultFakeBootsrapper : IFakeBootrapper. However, I'm just commenting blind right now, there may already be some abstraction like this. I don't have time to look into the code right now.

It would be good for scanning for the bootstrapper to start with the test assembly and then work 'outwards'(?), to cover the whole AppDomain, stopping on the first match.

I agree with starting small. Allow the overriding of the extension type discovery for this issue, building more in later in separate issues.

@blairconrad
Member

Hmm. Interesting. I'm of two minds about scanning outwards from the test assembly.

At first I was going to say that if most people don't use this feature, then having the scan go outward just to find nothing is a little bit of a waste, and that we could have the people who want to use a custom bootstrapper that's not in their test project (in a helper, maybe) just provide a tiny class in the test assembly that just extends (with no implementation) their preferred bootstrapper.

On the other hand, with your suggested approach, the people who don't want a customer bootstrapper still don't get one if they don't define one, and those who are bothered by the extra scan time are going to be hit by the default assembly scanning for the existing extensions anyhow. So either people don't notice the scan today (like probably most don't today) or they provide a bootstrapper. I see the advantage. The only bit of confusion is that we'd have 2 kinds of scanning right out of the box:

  1. scan for bootstrappers incrementally, starting with the test fixture and going out to the AppDomain (and the directory?)
  2. if we're using the default extension location mechanism, scan the AppDomain and all assemblies in the directory

But with proper documentation, this is probably not a big deal.

@adamralph
Member

If we restrict bootstrapper scanning to only the test assembly then I think the problem might be in identifying the test assembly. How do you think this could be done? Bear in mind that a test assembly doesn't have to be calling FIE directly, it may be going via any number of methods in any number of intermediate assemblies.

I'm thinking that an incremental scan might be the only practical way of doing this. An implementation which springs to mind is to walk up the call stack, scanning each assembly found in turn and when the top of the call stack is reached, any remaining assemblies in the AppDomain are scanned.

Moreoever, if we then continuted to scan all assemblies in the directory for bootstrappers, we would have the same breadth of scan as the scan for the other types, which leads me to think that scanning for everything can be done in one go. I.e. as we walk up the stack/across the appdomain and through the directory, we can keep a record of all the types we find as we go along. If no boostrapper is found then the default bootstrapper is invoked given the results of the completed scan as an input. If a custom boostrapper is found, then it is invoked instead.

What do you think? Perhaps I'm missing something and there is an easy way to identify the test assembly.

@blairconrad
Member

@adamralph, I had exactly the same thoughts regarding the difficulty of finding the test assembly. And initial research shows nothing. I was zeroing in on the same approach you were - "walk the stack". I suppose it's still possible (but unlikely) that inlined methods could bite us, but I can't imagine that we'd inline across assemblies and lose out method there. And yes, once we popped off the top of the stack, I think that looking in the AppDomain and then the directory is the right thing to do. If we find nothing (other than the default bootstrapper), we use the default. Otherwise, we use the first one we find. If multiple bootstrappers are found within an assembly (or AppDomain or directory), that's the test-writers problem - there's only so much a framework can do.

@blairconrad
Member

I'll continue to look around for a reliable way to find the test assembly (I've low hopes), but aside from that, I'm thinking of an internal static class Bootstrapper that will take over some of the bootstrapping code we already have in FakeItEasy (initially the determination of where the ImportsModule will look for extension classes). It will do this by performing the "ever expanding search" described above, looking for the first class that extends either the IFakeBootstrapper interface @adamralph mentioned or, if we don't see a need for the interface, the DefaultFakeBootstrapper, which would be implemented in FakeItEasy. If no implementations besides DefaultFakeBootstrapper are found, then that would be used.

The various bootstrapper implementations could either know about the container and perform their own registrations, which would mean exposing the container to the outside world, or could more just return the data that will be overridden. I think that's cleaner. It would look something like

public class DefaultFakeBootstrapper: IFakeBootstrapper
{
    public virtual IEnumerable<Assembly> GetAssembliesToScanForImports()
    {
        // innards of ApplicationDirectoryAssembliesTypeCatalogue.GetAllAvailableAssemblies();
    }
}
@adamralph
Member

@blairconrad thanks for the continued work on this. Should we assume that you are taking on the implementation of this feature?

@blairconrad
Member

Hi, @adamralph. I would be happy to do so, but have a small wrinkle I'd like to throw into the works. I was looking at the NancyFX code earlier today, just to get a feel for when they initiate the bootstrapper scanning and whatnot, and noticed some interesting details about how they implement the scanning. I thought right off that it may be something FakeItEasy could use, not instead of this issue, but alongside it, and possibly first. Here's the gist:

  1. the locations of all Assemblies in the AppDomain are found, then any DLLs found when scanning the Application Directory that are already in the list are not loaded
  2. when DLLs are loaded, they're first loaded using Assembly.ReflectionOnlyLoadFrom, a lighter load
  3. then the newly-loaded assembly's references are checked - if the assembly doesn't reference a Nancy assembly, it's skipped, since none of its classes can extend the bootstrapper
  4. if we pass all these checks, the assembly is fully loaded and only then is its types scanned

Assemblies in the AppDomain have their references checked as well, before their types are scanned.

This sounded like it would resolve the problems I encountered before creating this issue - potentially quicker scans as many assembly files don't need to be opened at all, and we only scan for types when there's a chance that we implement one of the ArgumentValueFormatter, DummyDefinition, and FakeConfigurator interfaces. By not fully loading many of other assembly files in the directory, I figured, we'd lose the LoaderLock and Runtime Errors as well.

So I hacked something up, changing the innards of the ApplicationDirectoryAssembliesTypeCatalogue to implement what I just described. The FakeItEasy tests all passed (except one, which I think was not quite correct anyhow). My application's tests passed. There were no popups or crashes in my runner. And the tests started up quickly, so long as I didn't let my test runner make shadow copies of my assemblies (it messes up the "have we loaded the assembly from this path already" shortcut).

Since we only use the ApplicationDirectoryAssembliesTypeCatalogue to provide a list of types for the ImportsModule, I think the change would be safe - we wouldn't lose functionality. Also, we'd get more intelligent default scanning behaviour that would benefit all users (or at worst not harm them) even if they didn't provide a custom bootstrapper. I think the best next steps are to spin off an issue for an improved ApplicationDirectoryAssembliesTypeCatalogue, implement it, and then reassess this issue, which I'd be happy to work on, but we'd need to decide if the "assemblies to scan" override is the best first extension point. It's no worse than any, I think.

Does this make sense? Sorry for the thrash.

@adamralph
Member

I think this makes a lot of sense! Have you pushed this branch to your fork so we can take a look?

@philippdolder
Member

@blairconrad do we really need to completely load an assembly to scan its types (step 4)? Can't we scan after loading it reflection only? something like this?

        public bool FindFakeItEasyExtensions(string assemblyPath)
        {
            var reflectedAssembly = Assembly.ReflectionOnlyLoadFrom(assemblyPath);
            foreach (Type type in reflectedAssembly.GetTypes())
            {
                Type[] foundInterfaces = type.FindInterfaces(this.FilterFakeItEasyTypes, null);

                if (foundInterfaces.Length > 0)
                {
                    return true;
                }
            }

            return false;
        }

        private bool FilterFakeItEasyTypes(Type m, object filtercriteria)
        {
            return m == typeof(IFakeConfigurator);
        }

@blairconrad
Member

@adamralph. Thanks. I haven't pushed it yet. For three reasons.

  1. the code needs tidying,
  2. I changed the failing unit test to pass, but still think it's incorrect (but in the same way that it always was), and
  3. I'm a git virgin

3 is only a problem for me, so I'll see if I can get the code pushed before I head into work. Caveat lector.

Then I'll see about @philippdolder's comment. It sounds really interesting.

@blairconrad blairconrad reopened this Jun 19, 2013
@blairconrad
Member

(oops)

@blairconrad
Member

@philippdolder, scanning for types when the assembly is loaded for reflection only sounds great. I thought I read that there can be problems if the types in the scanned assembly referenced types in other assemblies that weren't fully loaded, but until 2 days ago, I hadn't heard of ReflectionOnlyLoadFrom, that may be not true. I'll look into it (and try it) after I push what I have.

@adamralph
Member

@blairconrad as far as pushing to your fork goes, it really doesn't matter what state the code is in. You should be working in a branch so you can just rebase/squish/etc. later as required and force push or even just kill the branch and redo the work/cherry pick into a new branch etc. before sending a PR. Plenty of possibilities.

@blairconrad
Member

@adamralph, oh sure. I know that the branch is disposable, but I have some pride. I don't mind pushing something that's not perfect, but I wanted it to at least not be an affront to the eyes. Besides, first impressions are important. What if I pushed a sea of pink? My karma plummets, and then how do I get my pet issues fixed? ;-)

Anyhow, I just pushed two commits to an issue133 branch in my fork. The second one just removes a commented-out line of code. You probably want to take a peek at the first commit.

@adamralph
Member

LOL

Thanks - I'll take a look at little later 😄

@blairconrad
Member

I'm back. Now that #133 looks fixed and in master and soon to be in a stable release, I can come back to this, but questions spring to mind:

  1. Do we still want to do this? My immediate need for the bootstrapper disappears with the new scanning for extensions, but that's just me. If we still see a need, then I'd happily work on the implementation.
  2. If we still want to do this, what initial behaviours do we want to customize via the bootstrapper? The extension scanning is still an option, but I'll confess that I haven't put much thought into anything else. I'll look for time over the (long for me - I'm in Canada) weekend to find some likely customization points. Does anyone have any areas they want targeted?
  3. How about just using the new scanning algorithm to find the bootstrapper? It was ripped off from Nancy's bootstrapper locator, after all. We could just grab the first non-default bootstrapper we find in all the scanned assemblies. Or do we still have interest in the inside-out scanning for the bootstrapper? Me, I'd say no - I'm a little concerned at the idea of baking in two kinds of scanning behaviour.
@adamralph
Member

If we see no immediate need for this feature then I say we don't bother implementing it. The custom bootstrapper approach is a good thing to keep in mind for the future, if we find other stuff we want to override.

If we ever do implement it then I think we should just use the first bootstrapper found. There's probably no need to try and be clever about its location and order of preference etc. We could also provide an equivalent of Nancy.Bootstrapper.NancyBootstrapperLocator.Bootstrapper which allows setting of the bootstrapper explicitly.

@blairconrad
Member

I agree on all counts. Close for now? Or do you keep the issue around in a limbo state?

@adamralph
Member

Closed - can always raise a new one or re-open.

@adamralph adamralph closed this Jul 1, 2013
@aliostad
aliostad commented Jul 4, 2013

That is poor judgement I am afraid. People are really frustrated by the slowness of the tests.

We have now dropped FakeItEasy in favour of NSubstitute. One of the tests takes 4 seconds in FakeItEasy while only 700ms in NSubstitute.

@blairconrad
Member

@aliostad, is that 4 seconds using FakeItEasy 1.13.0, or an earlier version? I found that 1.13.0 (which includes the fix for #133) to be snappier than the version I'd been using previously.

I don't want to speak for @adamralph, but he did say such things as "if we see no immediate need" and "can always raise a new one or re-open". If you can provide more information about where FakeItEasy is taking too much time, maybe we can have a conversation about this that would lead us to a solution.

@aliostad
aliostad commented Jul 4, 2013

@blairconrad We tried with the latest version 1.13 and it was a tad better coming to around 2.5 secs.
I am afraid I do not have the time to send a repro as it involves sending sensitive code and stripping it down is difficult. Our app has dependency on many DLLs and I suppose a repro need to replicate that.

@blairconrad
Member

@aliostad, do you rely on the imported IArgumentValueFormatters, IDummyDefinitions, or IFakeConfigurators? If not, perhaps you could try the same trick I mentioned at the end of the blog post where I detailed my problems and introduced this issue - I changed FakeItEasy to just scan the FakeItEasy dll. I can provide the hacked-up version of the FakeItEasy DLL if you've time to try it. That way you don't even have to take the time to modify the code. If it turns out that the hacked DLL solves your problem, we can look into things further.

Try the FakeItEasy I put up in Dropbox. It's built for .NET 4.0. If that's a problem, the source code is in there as well.

Oh! Something I just thought of. Is your test runner using shadow copies? The new scanning algorithm short-circuits loading of DLLs whose locations correspond to locations that have been loaded in the AppDomain. Shadow copying defeats this optimization, since all the DLLs come from the shadow directory instead of the application directory. If you are using shadow copies, and have a few minutes, consider turning the shadow copying off for a test run or two to see if it makes a difference.

@adamralph
Member

@aliostad as @blairconrad explains, we closed the issue because we did not see an immediate need but it looks like your example may display an immediate need so I'm reopening the issue. Thanks for bringing it to our attention.

I think before tackling this we need to do a spike which involves the creation of two test projects referencing a large number of assemblies. One using FIE and the other using NSub. We compare the performance, switch off assembly scanning and then compare the performance again. If indeed we see a large difference then we should decide how to tackle it.

@adamralph adamralph reopened this Jul 6, 2013
@blairconrad
Member

Your plan sounds good. Of course, we may have to distinguish between

  1. referencing a large number of assemblies,
  2. having a large number of assemblies in the application directory,
  3. referencing a large number of assemblies that reference FakeItEasy, and
  4. having a large number of assemblies in the application directory that reference FakeItEasy

I'm not saying we have to test each of those, necessarily, just be aware of the possible differences in the performance profiles.

I'd also been thinking about @aliostad's problem. One approach that I'd considered is seeing if we have to scan eagerly. We don't need the IArgumentValueFormatters until something goes wrong. Is the same true for IDummyDefinition and IFakeConfigurator? Now that I write those interface names, I doubt it. Okay. That's probably not going to help.

If it turns out that we need to further optimize the scanning, the configurability is an obvious route. Another approach could be to not use the full path to determine whether or not an assembly has already been loaded (this helps only if there are lots of assemblies around that aren't in the AppDomain). If we match on filename only, the shadow copying will no longer fool us. Of course, if we reference multiple assemblies with the same filename, that could be disaster.

@blairconrad
Member

I didn't take the time to grab NSub and convert my existing tests, but I focused on the amount of time that FakeItEasy is taking to start up. I used the solution that was giving me problems before I started this issue. When FakeItEasy starts up, there are

  • 39 assemblies in the AppDomain, and
  • 97 assemblies in the current directory.
  • 2 assemblies referencing FakeItEasy

I wrapped a Stopwatch around the first call that exercises FakeItEasy, and there's what I found, when I had shadow copying on using 1.13.0, when I had shadow copying off using 1.13.0, and when I turned scanning completely off using the hacked up FakeItEasy I referenced above:

test type average time all run times
shadow copies 6.44 6.119, 6.822, 5.965, 5.900, 6.424, 6.041, 6.434, 7.219, 7.198, 6.278
no shadow 0.37 0.375, 0.367, 0.373, 0.375, 0.373, 0.373, 0.386, 0.376, 0.369, 0.370
no scan 0.23 0.234, 0.231, 0.227, 0.228, 0.233, 0.226, 0.229, 0.227, 0.239, 0.230

I can try to add a reference to NSubstitute and replicate my findings later, if there's interest.
@aliostad, is the setup I described in any way similar to your situation?

@aliostad
aliostad commented Jul 8, 2013

Hi guys. Thanks for taking this matter seriously. We have around 4400 tests in just one project. Even 0.5 second more means around 20 minutes later feedback.

I believe benchmarking is a good start and you need to keep an eye on it and after improving, perhaps publish the results. If you produce better results, we could be reconsidering going back to FakeItEasy.

@blairconrad
Member

Hi, @aliostad. To be clear, the times I pasted up above are one-time hits only (per test runner run). The assemblies are only ever scanned the first time FakeItEasy does something. There may be recurring slowness caused by something in FakeItEasy, but it's not the thing I measured.

Have you had time to try the .dll I put in dropbox, or to turn off shadow copies (if they're on)? I understand that maybe neither is a fix, but the information you glean may provide insight into what's causing your pain.

@adamralph
Member

I'll have to take a moment to understand the shadow copy issue. If the assemblies in the AppDomain are those from the original app folder, what are the shadow copies used for?

@blairconrad
Member

The ones in the AppDomain are from the shadow copy dir. The ones from the original app folder are the ones that we sometimes avoid scanning. I probably mislead you.

@blairconrad
Member

Okay. Another thought on the exact path match vs. filename and how its affected be shadow copies. (Granted this may not be the main cause of aliostad's problems, but it's something that's been on my mind - if we don't have to force people to disable shadow copies to get speed improvements, then why do?)

My big concern about making such a change up 'til now has been "What happens if someone references two assemblies with the same name, but from different directories?" For example, maybe they need two different log4nets. Then we may see that we've already loaded one of the copies and not do a scan, when really we should've.

I guess the thing to remember is that the only assemblies we're worried about are the ones that would reference FakeItEasy and provide implementations of the extension points, so probably the test assemblies themselves, as well as whatever helper libraries they may use. The chance of a collision is probably pretty low. And worst case, if there is such a collision and some extensions aren't found, I'm not sure that "don't name the assemblies the same" isn't an acceptable response. I suppose if we ever do implement configurable scanning or whatever, then we have an out in case this kind of problem comes up.

I'm not saying we should rush out and change the filter to ignore assemblies just based on the filename, but we could keep it in mind if it turns out that the "no shadow copies if you want speed" condition becomes too onerous.

@megakid
megakid commented Jan 29, 2014

I'm also getting the sharp end of this. I had 1.7 installed, recently updated to 1.17 and saw a dramatic increase in overall test timing. Have over 150 assemblies, approximately half of them test assemblies so there are plenty of *.dlls to scan. Initially I used the silverlight dll which fixed the performance which at the time I didn't understand. I read this and realised that the dll scanning/reflection stuff would have probably been limited in the SL5 dll. I would roll back to 1.7 but want the async method fake support. Be great to have some configurable option or way of disabling this feature.

@adamralph
Member

Thanks @megakid. Those symptoms sound quite bad. Bumping up to P2.

@blairconrad
Member

Indeed, thanks, @megakid. I'm a little confused, though. 1.17.0's assembly scanning should be faster than 1.7's—the only changes to the scanning that have been made since January 2011 are the ones that I made that should really have helped matters.
Given the size of your solution, I can't ask for a "small sample that reproduces the problem", but if I may ask, can you confirm that you see the slowness during assembly scanning time? (Which would basically be at the time you make the first FakeItEasy call.)
Depending on your test runner, this may manifest as a very long first test time (which is what I saw in ReSharper), or maybe a large amount of unaccounted-for time for a test fixture, maybe because the Setup (or equivalent) method is initializing FakeItEasy.

I'm not against returning to this issue. I just want to make sure we have a reasonable shot at fixing the problem you're seeing. It'd be a shame to implement the configurable scanning and then find out that something else was causing problems.

Oh, and as I have suggested elsewhere, consider checking whether you are making shadow copies when tests are run. If you have the option of turning this off, it can speed up the scanning considerably. If you do attempt this and find it makes a difference (or no difference!) (or not enough difference!) I'd be very interested in your results.

@blairconrad
Member

Oh. I see. @megakid, when you switched from 1.7 to 1.17, you also switched from the SL DLL to the .NET 4.0 DLL?
I'm a little slow. Can you explain why you think the scanning would've been limited in the SL DLL?

@adamralph
Member

Also, @megakid did you perform an isolated before and after comparison with only the FakeItEasy version being changed? Or did you upgrade anything else at the same time? Other libraries, test runners, IDE, etc.?

@adamralph
Member

Even better would be if the repo was open source so we could try it 😉

@blairconrad
Member

@megakid, I understand why switching from SilverLight would make the difference now. I see it has its own version of ImportsModule that just doesn't look for extension points such as argument formatters and Dummy definitions.

@megakid
megakid commented Jan 30, 2014

Hi chaps. I am trying to produce a test to demonstrate this behaviour. If you say the folder scanning was in 1.7 that adds another mystery to the mix. With our live build, I can simply swap the FakeItEasy DLL between 1.7 and 1.17 and see a test such as the one below go from taking 0.4seconds (1.7) to 10+seconds (1.17). More to come, hopefully.

interface IBleb { }

class Bob
{
    public Bob(IBleb b, decimal? value)
    {
        if (b == null) throw new ArgumentNullException("b");
        if (value == null) throw new ArgumentNullException("value");
    }
}

[TestFixture]
public class BobTests
{
    [Test]
    public void Ensure_Null_Exception_When_Argument_Null()
    {
        IBleb bleb = A.Fake<IBleb>();

        Assert.Throws<ArgumentNullException>(() => new Bob(bleb, null));
    }
}
@adamralph
Member

@megakid thanks for the info, the plot thickens indeed. Can you please tell us exactly which version you are using? There are many versions of the form 1.7.x.y (we only switched to SemVer from 1.8 onwards).

@blairconrad
Member

Of course, maybe I read the commit log wrong. I think the first 1.7.* release was in early April 2011? I can recheck once I'm back in front of a computer.

@adamralph
Member

they range from April 2011 to August 2012 https://www.nuget.org/packages/FakeItEasy/

@megakid
megakid commented Jan 30, 2014

The version we are using is 1.7.4257.42.

@megakid
megakid commented Jan 30, 2014

Ok - I've just recreated it using 1.17 code. Initially I attempted to write a console app basically ran the unit test above in it's main method. I dumped our production DLLs into the directory and discovered that we are loading 5177 types into ApplicationDirectoryAssembliesTypeCatalogue.GetAvailableTypes(). Stopwatched the test and gave me a time of ~600ms. I then went one step further and changed it into a proper NUnit test dll and ran that using nunit-console-x86.exe (2.6.2), suddenly the time to run the same code was ~13,000ms.

Definitely something odd going on here when nunit is involved then...

@megakid
megakid commented Jan 30, 2014

Hi again - When running with the /noshadow option - to disable shadow copying - on nunit, the tests run fine. Here's a back to back comparison timings (they are all different DLLs but I've disguised the names):

/noshadow OFF
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 67.0788ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 64.0267ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 64.8375ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 75.4405ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 63.7843ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 61.7613ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 62.3777ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 65.0277ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 62.081ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 65.6368ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 64.6196ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 66.687ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 64.2773ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 61.7166ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 62.2262ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 74.3237ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 67.2294ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 65.0316ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 71.9535ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 66.9484ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 67.8745ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 63.4583ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 61.3918ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 62.3267ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 66.3024ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 60.8219ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 64.1547ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 61.6671ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 63.1692ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 63.2893ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 60.1234ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 65.4744ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 66.5632ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 62.6427ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 62.1733ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 63.8049ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 61.0344ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 67.9551ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 68.6291ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 62.4601ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 60.888ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 64.1327ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 61.4703ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 60.9363ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 60.9261ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 60.1575ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 69.6337ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 78.6927ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 75.2963ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 65.4551ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 62.5908ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 60.1609ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 68.0885ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 61.4978ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 62.6892ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 63.5549ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 68.6663ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 64.8285ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 65.7084ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 69.7255ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 67.2904ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 66.1328ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 60.9919ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 62.452ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 60.6785ms
Total: 14060.2964ms 'C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\'
/noshadow ON
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.9077ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.4609ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.4187ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.5814ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.7414ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.4075ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.4531ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.591ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.1984ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.0704ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.905ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.9367ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.7694ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.6493ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.4745ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.4676ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.9122ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.585ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.6638ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.1117ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.1434ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.4975ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.6599ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.6605ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.9572ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.4972ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.2283ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.4643ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.227ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.4932ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.4703ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.7344ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.8145ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.5388ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.921ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.617ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.5282ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.557ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.0007ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.029ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.1721ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.4525ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.2041ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.7057ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.665ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.5089ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.7873ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 3.2084ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.6367ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.1794ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.5922ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.4685ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.54ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.5666ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.4778ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.5062ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.521ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.4742ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.5192ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 2.0586ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.5135ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.9922ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.8186ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.7223ms
Load assembly C:\Projects\FakeItEasyTest\FakeItEasyC\bin\Debug\Something.Tests.dll: 1.4664ms
Total: 945.8182ms '' 
@megakid
megakid commented Jan 30, 2014

Was meant to add a thanks to @blairconrad for pointing me in the right direction regarding shadow copies. I am quite concerned that it adds a 3000% (+60ms) performance penalty to the scanning! Lesson learnt.

@blairconrad
Member

@megakid, thanks for performing the /noshadow test. I really appreciate the info.
It's all about not being able to tell that the shadow copy DLL is really the same as a DLL that's already in the app domain. I'd like to skip them automatically or something, but investigations haven't really lead to a safe way to do it, and it was considered a little niche, and nobody was complaining, so it was deferred in favour of adding the boostrapper if the problem started bothering enough people.

(The shadow copies KILL me when I run the tests at the Day Job, so I disabled the shadow copies in my ReSharper test runner and became happy.)

Not that this will keep me from working hard on the bootstrapper, but is adding /noshadow an option for you in your work? As something that can ease your pain until a better solution is released?

@blairconrad
Member

Okay. I'll start on this. @adamralph and I chatted, and if I may put words in his mouth, the approach will be to introduce a Bootstrapper. We'll supply a default implementation that will make sure the things we do now keep getting done that way.

The Bootstrapper will have (at first) one overrideable method that will control the list of DLLs that might be loaded from the working directory and scanned for extension points.
Tentative name for that method: GetAssemblyFilenamesToScanForExtensions. If anyone has something better, please chime in.

The bootstrapper locator will scan the assemblies in the AppDomain (only) and if it finds an overriding bootstrapper, that one will be used. If it doesn't, we'll use the default.

@blairconrad blairconrad was assigned Jan 31, 2014
@blairconrad
Member

Assigning to me and marking as Working.

@patrik-hagne
Member

Nice to see this one happening! Great job!

@adamralph adamralph added a commit to adamralph/FakeItEasy that referenced this issue Feb 9, 2014
@adamralph adamralph #130 refactored BootstrapperLocator 42972ff
@blairconrad blairconrad added a commit to blairconrad/FakeItEasy that referenced this issue Feb 10, 2014
@blairconrad blairconrad #130 - applying code review comments
- fixing comment in AssebmliesTypeCatalogue integration test
- no longer explicitly disregarding DefaultBootstrapper when scanning - we're excluding the FakeItEasy assembly, so there's no need
216fb7c
@blairconrad blairconrad added a commit to blairconrad/FakeItEasy that referenced this issue Feb 11, 2014
@blairconrad blairconrad #130 - Renaming ApplicationDirectoryAsseembliesTypeCatalogue to TypeC…
…atalogue and fixing its docs, per code review.
6f31729
@adamralph adamralph added 3 - Done and removed 2 - Working labels Feb 22, 2014
@adamralph adamralph added this to the 1.18 milestone Feb 22, 2014
@adamralph adamralph closed this Feb 22, 2014
@adamralph adamralph added a commit to adamralph/FakeItEasy that referenced this issue Feb 24, 2014
@adamralph adamralph #130 refactor: docs and internal structure of TypeCatalogue 967a0a6
@adamralph adamralph added a commit to adamralph/FakeItEasy that referenced this issue Feb 24, 2014
@adamralph adamralph #130 refactor: moved type loading from TypeCatalogue constructor to L…
…oad() method
42884d9
@adamralph adamralph added a commit to adamralph/FakeItEasy that referenced this issue Feb 24, 2014
@adamralph adamralph #130 green: satisfied TypeCatalogueTests.Should_warn_of_bad_assembly_…
…files
72dbf96
@adamralph adamralph added a commit to adamralph/FakeItEasy that referenced this issue Feb 24, 2014
@adamralph adamralph #130 added warnings when failing to load assembly or types 08ab7b2
@adamralph adamralph added a commit to adamralph/FakeItEasy that referenced this issue Feb 24, 2014
@adamralph adamralph #130 refactor: refactored TypeCatalogueTests 4c61ef6
@adamralph adamralph added a commit to adamralph/FakeItEasy that referenced this issue Feb 27, 2014
@adamralph adamralph #130 refactor: docs and internal structure of TypeCatalogue 27750bc
@adamralph adamralph added a commit to adamralph/FakeItEasy that referenced this issue Feb 27, 2014
@adamralph adamralph #130 refactor: moved type loading from TypeCatalogue constructor to L…
…oad() method
605d2aa
@adamralph adamralph added a commit to adamralph/FakeItEasy that referenced this issue Feb 27, 2014
@adamralph adamralph #130 green: satisfied TypeCatalogueTests.Should_warn_of_bad_assembly_…
…files
d864251
@adamralph adamralph added a commit to adamralph/FakeItEasy that referenced this issue Feb 27, 2014
@adamralph adamralph #130 added warnings when failing to load assembly or types 7ffb312
@adamralph adamralph added a commit to adamralph/FakeItEasy that referenced this issue Feb 27, 2014
@adamralph adamralph #130 refactor: refactored TypeCatalogueTests 877f878
@adamralph adamralph added a commit to adamralph/FakeItEasy that referenced this issue Mar 1, 2014
@adamralph adamralph #130 change message when failing to load types 4f40b05
@Dashue Dashue added a commit to Dashue/FakeItEasy that referenced this issue Mar 3, 2014
@adamralph @Dashue adamralph + Dashue #130 refactor: docs and internal structure of TypeCatalogue 77c0a2e
@Dashue Dashue added a commit to Dashue/FakeItEasy that referenced this issue Mar 3, 2014
@adamralph @Dashue adamralph + Dashue #130 refactor: moved type loading from TypeCatalogue constructor to L…
…oad() method
274f8b3
@Dashue Dashue added a commit to Dashue/FakeItEasy that referenced this issue Mar 3, 2014
@adamralph @Dashue adamralph + Dashue #130 green: satisfied TypeCatalogueTests.Should_warn_of_bad_assembly_…
…files
1ee599a
@Dashue Dashue added a commit to Dashue/FakeItEasy that referenced this issue Mar 3, 2014
@adamralph @Dashue adamralph + Dashue #130 added warnings when failing to load assembly or types 10a2899
@Dashue Dashue added a commit to Dashue/FakeItEasy that referenced this issue Mar 3, 2014
@adamralph @Dashue adamralph + Dashue #130 refactor: refactored TypeCatalogueTests 11fb4a0
@Dashue Dashue added a commit to Dashue/FakeItEasy that referenced this issue Mar 3, 2014
@adamralph @Dashue adamralph + Dashue #130 change message when failing to load types a1276ec
@adamralph
Member

@aliostad, @megakid thanks very much for the input on this issue. Look out for your names in the release notes 🏆

https://www.nuget.org/packages/FakeItEasy/1.18.0
https://github.com/FakeItEasy/FakeItEasy/releases/tag/1.18.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment