Implemented support for mirrors #3157

JohanMabille · 2024-01-25T20:17:56Z

No description provided.

AntoinePrv

My first few comments to get started since this seems to still be a moving target.

AntoinePrv · 2024-01-29T13:43:20Z

libmamba/src/download/curl.cpp

+            std::ofstream out("trace.txt", std::ios_base::app);
+            out << "configuring CURL handle:"
+                << "\n\turl = " << url << std::endl;


What is this trace.txt?

A leftover, I need to remove it (and the other occurences), I just wanted to get all the tests green before removing them.

AntoinePrv · 2024-01-29T13:52:53Z

libmamba/include/mamba/specs/channel.hpp

@@ -109,6 +109,7 @@ namespace mamba::specs
        auto clear_platforms() -> platform_list;
        void set_platforms(platform_list platforms);

+        [[nodiscard]] auto id() const -> const std::string&;


Can we a add a doc saying what this is (a cross URL id) and its limitations (dependent on channel_alias).

AntoinePrv · 2024-01-29T13:55:07Z

libmamba/src/specs/channel.cpp

@@ -45,6 +45,7 @@ namespace mamba::specs
    Channel::Channel(std::vector<CondaURL> mirror_urls, std::string display_name, platform_list platforms)
        : m_mirror_urls(std::move(mirror_urls))
        , m_display_name(std::move(display_name))
+        , m_id(m_display_name)


That's still not making it a proper ID, but its something.

Suggested change

, m_id(m_display_name)

, m_id(util::rstrip(m_display_name, '/'))

AntoinePrv · 2024-01-29T13:56:13Z

libmamba/src/specs/channel.cpp

+    auto Channel::id() const -> const std::string&
+    {
+        return m_id;
+    }
+


Should we consider

auto operator==(Channel const& lhs, Channel const& rhs) -> bool { return lhs.id() == rhs.id(); }

Ok let's revert this change for now, the current Channel's API is not really consistent with that comparison operator (especially because we can change attributes of an already created channel, without having its ID changed.).

JohanMabille · 2024-01-29T20:29:15Z

@AntoinePrv I've addressed the comments but the last one, I think this one is ready to go.

SylvainCorlay · 2024-01-29T22:16:38Z

Bravo ! This is getting close.

Hind-M · 2024-01-30T08:45:16Z

libmamba/tests/src/core/test_mirror.cpp

@@ -10,7 +10,7 @@

 namespace mamba
 {
-    TEST_SUITE("mirror")
+    /*TEST_SUITE("mirror")


Is there a reason the tests are commented here?

Yes, I have started to partially move the mirror implementation to the download folder / namespace. Since I want to make incremental steps, I have not port everything yet, but I had to remove core/mirror.cpp from the CMakeLists.txt to avoid symbol duplication, and I had to comment this test until the missing functions are ported to download.

libmamba/include/mamba/download/downloader.hpp

libmamba/include/mamba/download/mirror.hpp

AntoinePrv · 2024-01-30T13:48:27Z

libmamba/include/mamba/download/request.hpp

+    // This class is used to create strong alias on
+    // string_view. This helps to avoid error-prone
+    // calls to functionsthat accept many arguments
+    // of the same type
+    template <int I>
+    class string_view_alias
+    {
+    public:
+
+        explicit string_view_alias(std::string_view s);
+        operator std::string_view() const;
+
+    private:
+
+        std::string_view m_wrapped;
+    };
+
+    using MirrorName = string_view_alias<0>;


Why not?

Suggested change

// This class is used to create strong alias on

// string_view. This helps to avoid error-prone

// calls to functionsthat accept many arguments

// of the same type

template <int I>

class string_view_alias

{

public:

explicit string_view_alias(std::string_view s);

operator std::string_view() const;

private:

std::string_view m_wrapped;

};

using MirrorName = string_view_alias<0>;

struct MirrorName

{

std::string_view name;

}

Because initially I also wanted to have aliases for the url_path and why not thename fields of the request, and I found this solution more elegant than repeating the same code for wrapping the same type. I had some problem with wrapping the url that I postponed to a future PR. Ideally, this structure should be moved to some util files, and accept an additional template argument (for the type it should alias), and could be used for aliasing other IDs, such as the Channel ID for instance. But I agree it may seem overkill actually, and I can remove it in a future PR if we decide to alias the MirrorName only.

A minor difference is that the structure forces a {} initialization while the alias lets the choice in the initialization syntax.

libmamba/src/download/downloader_impl.hpp

AntoinePrv · 2024-01-30T13:52:00Z

libmamba/src/download/mirror.cpp

+    std::string MirrorID::to_string() const
+    {
+        return fmt::format("MirrorID <{}>", m_value);
+    }


Should we write a custom fmt::formatter instead?

It would have to be friend to access the m_value member, and that would make the code heavier. compared to this single line. I am not super familiar with the internals of FMT though, does a custom fmt::formatter have some advantages over calling fmt::format in this case?

AntoinePrv · 2024-01-30T13:53:15Z

libmamba/src/download/mirror.cpp

@@ -0,0 +1,169 @@
+#include <fmt/format.h>


Suggested change

#include <fmt/format.h>

// Copyright (c) 2023, QuantStack and Mamba Contributors

//

// Distributed under the terms of the BSD 3-Clause License.

//

// The full license is in the file LICENSE, distributed with this software.

#include <fmt/format.h>

AntoinePrv · 2024-01-30T13:53:50Z

libmamba/src/download/mirror_map.cpp

@@ -0,0 +1,51 @@
+#include "mamba/download/mirror_map.hpp"


Suggested change

#include "mamba/download/mirror_map.hpp"

// Copyright (c) 2023, QuantStack and Mamba Contributors

//

// Distributed under the terms of the BSD 3-Clause License.

//

// The full license is in the file LICENSE, distributed with this software.

#include "mamba/download/mirror_map.hpp"

libmamba/tests/src/download/test_downloader.cpp

Klaim

This needs more documentation and maybe some tweaks, but LGTM in general.

libmamba/include/mamba/core/context.hpp

Klaim · 2024-01-30T10:51:06Z

libmamba/include/mamba/download/downloader.hpp

-        DownloadOptions options = {},
-        DownloadMonitor* monitor = nullptr
+        Options options = {},
+        Monitor* monitor = nullptr


Side note: this could be a tl::optional<Monitor> too, as tl::optional handles references (not the standard one, unfortunately - being discussed again BTW)

Yes, but I am a bit reluctant to add another dependency just for this here.

libmamba/include/mamba/download/mirror.hpp

Klaim · 2024-01-30T10:58:41Z

libmamba/include/mamba/download/mirror.hpp

+        MirrorRequest& operator=(MirrorRequest&&) = default;
+    };
+
+    class Mirror


I would prefer to have some documentation here about what this specific type is about for this PR .

libmamba/include/mamba/download/mirror.hpp

libmamba/src/api/install.cpp

Klaim · 2024-01-30T11:16:27Z

libmamba/src/download/downloader.cpp

+     ********************************/
+
+    /*
+     * MirrorAttempt FSM:


Interesting way to specify it! 👍🏽

Klaim · 2024-01-30T11:23:05Z

libmamba/src/download/downloader.cpp

+     * LAST_REQUEST_FAILED:
+     *     - m_retries == p_mirror->max_retries ? => SEQUENCE_FAILED
+     */
+    MirrorAttempt::MirrorAttempt(Mirror* mirror)


If it is not valid to take nullptr, then replace this pointer by a reference and store in the member pointer (take dependency by reference, store by pointer). Then add an assertion in the constructor body about this pointer, just to clarify for future maintainers.

Mmh not sure about the assertion here, if the constructor takes the mirror parameter by reference (and I agree that it should), then this parameter has a valid address and it is clear the pointer cannot be null.

libmamba/src/download/mirror.cpp

JohanMabille force-pushed the mirror_oci branch from 7cb3b6e to 7efd632 Compare January 25, 2024 20:19

JohanMabille requested review from Klaim, AntoinePrv and Hind-M January 25, 2024 20:20

JohanMabille force-pushed the mirror_oci branch 5 times, most recently from 1c5aa2c to 424e1db Compare January 29, 2024 13:44

AntoinePrv reviewed Jan 29, 2024

View reviewed changes

JohanMabille force-pushed the mirror_oci branch from 424e1db to 9ff330a Compare January 29, 2024 14:53

Implemented support for mirrors

2ac58d8

JohanMabille force-pushed the mirror_oci branch from 9ff330a to 2ac58d8 Compare January 29, 2024 19:43

Hind-M reviewed Jan 30, 2024

View reviewed changes

Hind-M approved these changes Jan 30, 2024

View reviewed changes

AntoinePrv approved these changes Jan 30, 2024

View reviewed changes

Klaim approved these changes Jan 30, 2024

View reviewed changes

Changes according to reviews

51eb4d3

JohanMabille force-pushed the mirror_oci branch from d7099f2 to 51eb4d3 Compare January 30, 2024 16:21

SylvainCorlay merged commit f7c7630 into mamba-org:main Jan 30, 2024
27 checks passed

JohanMabille deleted the mirror_oci branch January 31, 2024 08:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implemented support for mirrors #3157

Implemented support for mirrors #3157

JohanMabille commented Jan 25, 2024

AntoinePrv left a comment

AntoinePrv Jan 29, 2024

JohanMabille Jan 29, 2024 •

edited

Loading

AntoinePrv Jan 29, 2024

AntoinePrv Jan 29, 2024

AntoinePrv Jan 29, 2024

JohanMabille Jan 29, 2024

JohanMabille commented Jan 29, 2024 •

edited

Loading

SylvainCorlay commented Jan 29, 2024

Hind-M Jan 30, 2024

JohanMabille Jan 30, 2024 •

edited

Loading

AntoinePrv Jan 30, 2024

JohanMabille Jan 30, 2024 •

edited

Loading

AntoinePrv Jan 30, 2024

JohanMabille Jan 30, 2024

AntoinePrv Jan 30, 2024

AntoinePrv Jan 30, 2024

Klaim left a comment

Klaim Jan 30, 2024

JohanMabille Jan 30, 2024

Klaim Jan 30, 2024

Klaim Jan 30, 2024

Klaim Jan 30, 2024

JohanMabille Jan 30, 2024 •

edited

Loading

	, m_id(m_display_name)
	, m_id(util::rstrip(m_display_name, '/'))

-#include <fmt/format.h>
+// Copyright (c) 2023, QuantStack and Mamba Contributors
+//
+// Distributed under the terms of the BSD 3-Clause License.
+//
+// The full license is in the file LICENSE, distributed with this software.
+#include <fmt/format.h>

Implemented support for mirrors #3157

Implemented support for mirrors #3157

Conversation

JohanMabille commented Jan 25, 2024

AntoinePrv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JohanMabille Jan 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JohanMabille commented Jan 29, 2024 • edited Loading

SylvainCorlay commented Jan 29, 2024

Choose a reason for hiding this comment

JohanMabille Jan 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JohanMabille Jan 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Klaim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JohanMabille Jan 30, 2024 • edited Loading

Choose a reason for hiding this comment

JohanMabille Jan 29, 2024 •

edited

Loading

JohanMabille commented Jan 29, 2024 •

edited

Loading

JohanMabille Jan 30, 2024 •

edited

Loading

JohanMabille Jan 30, 2024 •

edited

Loading

JohanMabille Jan 30, 2024 •

edited

Loading