packages windows: redirect to GitHub Releases to download latest Groonga #1432

HashidaTKS · 2022-10-18T08:34:24Z

Changed to redirect to GitHub Releases by .htaccess to download latest Groonga.

Changes

Skip downloading Groonga packages to local machine when target is Windows.
Create a new task named prepare.
- This task creates .htaccess for redirecting when target is Windows.

Created .htaccess is like below.

Redirect /windows/groonga/groonga-latest-x64-vs2022.zip https://github.com/groonga/groonga/releases/download/v12.0.9/groonga-12.0.9-x64-vs2022.zip
Redirect /windows/groonga/groonga-latest-x64-vs2022-with-vcruntime.zip https://github.com/groonga/groonga/releases/download/v12.0.9/groonga-12.0.9-x64-vs2022-with-vcruntime.zip
Redirect /windows/groonga/groonga-latest-x64-vs2019.zip https://github.com/groonga/groonga/releases/download/v12.0.9/groonga-12.0.9-x64-vs2019.zip
Redirect /windows/groonga/groonga-latest-x64-vs2019-with-vcruntime.zip https://github.com/groonga/groonga/releases/download/v12.0.9/groonga-12.0.9-x64-vs2019-with-vcruntime.zip
Redirect /windows/groonga/groonga-latest-x86-vs2019.zip https://github.com/groonga/groonga/releases/download/v12.0.9/groonga-12.0.9-x86-vs2019.zip
Redirect /windows/groonga/groonga-latest-x86-vs2019-with-vcruntime.zip https://github.com/groonga/groonga/releases/download/v12.0.9/groonga-12.0.9-x86-vs2019-with-vcruntime.zip

HashidaTKS · 2022-10-18T08:56:07Z

packages/packages-groonga-org-package-task.rb

+    if (target_namespace == :windows)
+      htaccesss_value = "RewriteRule groonga-((1[2-9]|[2-9][0-9])[\\d\\.]+?)-.+?-vs(?!.*(2012|2013|2015|2017)).+?\\.zip"\
+                        " #{github_releases_base_url}/v$1/$0\n"
+      File.write(htaccess_path, htaccesss_value, mode: "w")


Memo:

When vs2019+ reaches EOL, I think we need to add RewriteRules for the latest version of them at this point.

For example, if vs2019 reaches EOL and #{target_namespace}_targets not contains vs2019, we don't create RewriteRules for vs2019 on the lines 191 - 203.
So users will not get latest vs2019 from URLs like https://packages.groonga.org/windows/groonga/groonga-latest-x86-vs2019-with-vcruntime.zip, they will return 404 not found.
In order to avoid 404 not found, we need to add RewriteRules like below.

RewriteRule groonga-latest-x86-vs2019-with-vcruntime.zip https://github.com/groonga/groonga/releases/download/{old-version}/groonga-{old-version}-x86-vs2019-with-vcruntime.zip

If we don't need to provide the way to download EOL versions from packages.groonga.org, the RewriteRules for latest of EOL versions is needless.

この英語では何を言っているのかわからない可能性があるので日本語でも記載。

191行目から203行目では、現在サポート対象となっているVisual Studioのバージョン(vsXXXX)のlatestに対するリダイレクトルールを作っている。（groonga-latest-x86-vs2019のようなやつ）
なので、サポート対象外になったvsXXXXのlatestへのリダイレクトルールは作られない。
毎回新しく.htaccessを作っているので、.htaccessの中に古いバージョン用のlatest用のリダイレクトルールは作られない。

vs2017まではpackages.groonga.orgに正しいlatestの実体があるからそれでも問題ないのだが（.htaccessのルールに引っかからなくても、実体のほうが使われるだけだから）、この対応が入った後は、vs2019以降では、packages.groonga.orgに実体がない or 実体が古すぎるという状態になる（この変更の後はpackages.groonga.orgのパッケージは更新しないから）。
そのため、例えばvs2019のサポートが終了して、latest-xxx-vs2019のリダイレクトルールが作られなくなると、 https://packages.groonga.org/windows/groonga/groonga-latest-x86-vs2019-with-vcruntime.zip のようなURLにアクセスした時に、404エラーになるか古すぎるものがダウンロードされるようになる。
なので、例えばvs2019のサポートが終了した際には、このあたりの処理で https://packages.groonga.org/windows/groonga/groonga-latest-x86-vs2019-with-vcruntime.zip からvs2019を最後にサポートしたバージョンへのリダイレクトルールを作ってあげる必要があるのではないかということ。

ただ、サポートが終了したバージョンのlatestはpackages.groonga.orgからはダウンロードさせなくても良いというのであれば、そのような対処は不要。

なお、latestを指定せずにバージョンを明示的に指定すれば最初のリダイレクトルールに引っかかるのでリダイレクトされる。

すでにEOLになったものは、サポート終了しているので、 https://packages.groonga.org/windows/groonga/groonga-latest-x86-vs2019-with-vcruntime.zip でダウンロードできなくてもいいように思います。
EOLになったパッケージはできる限り使ってほしくないので。

なるほど、であれば特に将来的に追加で対応が必要になることはなさそうですね！

HashidaTKS · 2022-10-19T03:03:58Z

@kou @komainu8

Would you review this when you have time?

komainu8 · 2022-10-19T08:20:43Z

packages/packages-groonga-org-package-task.rb

+
+    if (target_namespace == :windows)
+      htaccesss_value = "RewriteRule groonga-((1[2-9]|[2-9][0-9])[\\d\\.]+?)-.+?-vs(?!(2012|2013|2015|2017)).+?\\.zip"\
+                        " #{github_releases_base_url}/v$1/$0\n"


Do we need this rewrite rule? I think that we only need a link to -latest.

I had written the reason why this ReqiteRule is needed on description.

The regexp of the first line is for 12.0.0+ (but not latest) and vs2019+.
If we omit this, for example, after Groonga 12.1.0 is released, we won't be able to download old versions by version specifying URL like https://packages.groonga.org/windows/groonga/groonga-12.0.9-x64-vs2019-with-vcruntime.zip because v12.0.9 will not exist on packages.groonga.org.

But if we do not need to provide a way to donwnload old version Groonga from packages.groonga.org, it is needless.

Thanks. I see.

In my opinion, this is needless.
Because a download URL that user can know is only https://packages.groonga.org/windows/groonga/groonga-latest-xxx.zip normally.

In addition, I would like to use in the user the latest version when Groonga up to version.
Because the user doesn't need to use a specific version in when Groonga up to version generally.

Ok, I will remove this.

Removed this RewriteRule.

komainu8 · 2022-10-19T08:26:18Z

@HashidaTKS Could you confirm my comments?

kou · 2022-10-20T00:31:08Z

Fixed to redirect to GitHub Releases by .htaccess to download latest (and 12.0.0+) Groonga.

Fixed? I think that the current behavior (provide packages for Windows on packages.groonga.org) isn't a bug.

kou · 2022-10-20T00:34:10Z

packages/packages-groonga-org-package-task.rb

+    github_releases_base_url = "https://github.com/groonga/groonga/releases/download"
+    htaccess_path = "#{repositories_dir}/.htaccess"
+
+    if (target_namespace == :windows)


Suggested change

if (target_namespace == :windows)

if target_namespace == :windows

kou · 2022-10-20T00:36:45Z

packages/packages-groonga-org-package-task.rb

@@ -175,19 +175,32 @@ def download_packages(target_namespace)
    mkdir_p(repositories_dir)
    download_dir = "#{base_dir}/tmp/downloads/#{@version}"
    mkdir_p(download_dir)
+
+    github_releases_base_url = "https://github.com/groonga/groonga/releases/download"


packages-groonga-org-package-task.rb is shared with other packages that use packages.groonga.org such as Mroonga and PGroonga. So putting groonga/groonga here isn't a good idea.

kou · 2022-10-20T00:37:33Z

packages/packages-groonga-org-package-task.rb

-        end
+        latest_link_base_name = target.gsub(@version, "latest")
+        htaccesss_value = "RewriteRule #{latest_link_base_name} #{github_releases_base_url}/v#{@version}/#{target}\n"
+        File.write(htaccess_path, htaccesss_value, mode: "a+")


Can we avoid reusing download_packages for this because the new behavior downloads nothing?

I have fixed it.

I have created a new task named create_htaccess for createing .htaccess because it seemed that neither download_packages(target_namespace)norrelease(target_namespace)` is good method for creating .htaccess in it.
How do you feel that?

Changes

Skip downloading packages when target_namespace is :windows.

Create new task named create_htaccess for each namespaces.

This task creates .htaccess.

It works only when namespaces is :windows. Do nothing in other namespaces.

I think there is a more suitable name for this task. Do you have a good idea for the task name?

How about prepare or something? It seems that create_htaccess is too specific to use in packages-groonga-org-package-task.rb.

Renamed to prepare

This reverts commit 2c87903.

This reverts commit e0bca3a.

kou · 2022-10-20T04:29:54Z

packages/packages-groonga-org-package-task.rb

+          desc "Make .htaccess for #{target_namespace} packages"
+          task :create_htaccess do
+            create_htaccess(target_namespace) if enabled
+          end
+          tasks << "#{target_namespace}:create_htaccess"


It seem that this isn't related to use_built_package?.

I will fixed it.

I focused on the case of redirecting URL so it added to this line because there should be some built packages to redirect.
But as you said, create_htaccess ( will be renamed prepare) is for not only redirection but also some other cases.
So it should be out of use_built_package?

I have fixed it.

kou · 2022-10-20T04:34:20Z

packages/Rakefile

+    rm_rf(htaccess_path)
+
+    __send__("#{target_namespace}_targets").each do |target|
+      redirect_url = built_package_url(target_namespace, target)
+      redirect_target = target.gsub(@version, "latest")
+      htaccesss_value = "RewriteRule #{redirect_target} #{redirect_url}\n"
+      File.write(htaccess_path, htaccesss_value, mode: "a+")  
+    end


How about opening a file instead of calling File.write() multiple times? I haven't seen the usage (Use FileUtils.rm_f and multiple File.write(mode: "a+") to create multi lines content).

Suggested change

rm_rf(htaccess_path)

__send__("#{target_namespace}_targets").each do |target|

redirect_url = built_package_url(target_namespace, target)

redirect_target = target.gsub(@version, "latest")

htaccesss_value = "RewriteRule #{redirect_target} #{redirect_url}\n"

File.write(htaccess_path, htaccesss_value, mode: "a+")

end

File.open(htaccess_path, "w") do |htaccess|

__send__("#{target_namespace}_targets").each do |target|

redirect_url = built_package_url(target_namespace, target)

redirect_target = target.gsub(@version, "latest")

htaccess.puts("RewriteRule #{redirect_target} #{redirect_url}")

end

end

end

I have fixed it.

HashidaTKS · 2022-10-20T05:32:40Z

@kou @komainu8

Thank you.
I have addressed your comments.

kou · 2022-10-20T05:51:10Z

packages/packages-groonga-org-package-task.rb

@@ -244,6 +247,12 @@ def define_release_tasks
          end
          tasks << "#{target_namespace}:download"
        end
+
+        desc "Prepare for #{target_namespace} packages"


Do we need for here?

@yoshimotoyuk What do you think about this?

I think for is needless in this case.

メモ:

Prepare: 「準備する」というようなニュアンス
Prepare for: （来たるべき何かに）「備える」というようなニュアンス

Or, Prepare for releasing #{target_namespace} packages ?

「アップロード用にパッケージを準備する」タスクだと思っていました。

アップロード用にパッケージを準備する場合は

Prepare #{target_namespace} packages for up-load

になると思います。

"prepare for releasing #{target_namespace} packages"だと　{target_namespace}パッケージをリリースするために準備する。となると思います。

あざっす。
この文脈だとfor up-loadは冗長なのでPrepare #{target_namespace} packagesでいい気がしました。

話を伺って Prepare #{target_namespace} packages だけで良いのではないかと思いましたが、いかがでしょうか？

被りました。Prepare #{target_namespace} packages に変更します。

はい。前後関係を見るに
シンプルにして大丈夫だと思いました

更新しました。

packages/Rakefile

kou · 2022-10-20T05:57:29Z

packages/Rakefile

+      __send__("#{target_namespace}_targets").each do |target|
+        redirect_url = built_package_url(target_namespace, target)
+        redirect_target = target.gsub(@version, "latest")
+        htaccess.puts("RewriteRule #{redirect_target} #{redirect_url}")  


Can we use Redirect instead of RewriteRule here?

I have changed to use RedirectMatch instead of Redirect in order to simplify a path specification in source code.

The new .htaccess is as below.

RedirectMatch ^.+/groonga\-latest\-x64\-vs2022\.zip$ https://github.com/groonga/groonga/releases/download/v12.0.9/groonga-12.0.9-x64-vs2022.zip RedirectMatch ^.+/groonga\-latest\-x64\-vs2022\-with\-vcruntime\.zip$ https://github.com/groonga/groonga/releases/download/v12.0.9/groonga-12.0.9-x64-vs2022-with-vcruntime.zip RedirectMatch ^.+/groonga\-latest\-x64\-vs2019\.zip$ https://github.com/groonga/groonga/releases/download/v12.0.9/groonga-12.0.9-x64-vs2019.zip RedirectMatch ^.+/groonga\-latest\-x64\-vs2019\-with\-vcruntime\.zip$ https://github.com/groonga/groonga/releases/download/v12.0.9/groonga-12.0.9-x64-vs2019-with-vcruntime.zip RedirectMatch ^.+/groonga\-latest\-x86\-vs2019\.zip$ https://github.com/groonga/groonga/releases/download/v12.0.9/groonga-12.0.9-x86-vs2019.zip RedirectMatch ^.+/groonga\-latest\-x86\-vs2019\-with\-vcruntime\.zip$ https://github.com/groonga/groonga/releases/download/v12.0.9/groonga-12.0.9-x86-vs2019-with-vcruntime.zip

Why do we need to use RedirectMatch not Redirect?
It seems that we don't have any pattern (we have only literal).

As a first, the rule below was incorrect because it redirects http://packages.groonga.org/windows/packages/groonga-latest-x64-vs2022.zip to http://packages.groonga.org/windows/packages/https://github.com/groonga/groonga/releases/download/v12.0.8/groonga-12.0.8-x64-vs2022.zip. We need to specify an absolute path for old URL.

RewriteRule groonga-latest-x64-vs2022.zip https://github.com/groonga/groonga/releases/download/v12.0.8/groonga-12.0.8-x64-vs2022.zip

If we use Redirect, the rule will be as below.

Redirect /windows/groonga/groonga-latest-x86-vs2019.zip https://github.com/groonga/groonga/releases/download/v12.0.9/groonga-12.0.9-x86-vs2019.zip

The reason why I used RedirectMatch was just because I wanted to avoid creating /windows/groonga in source code.

How about just prepending /windows/groonga/ instead of using regular expression?
/windows/groonga/ will not be changed.

Or we can use /#{target_namespace}/#{@package}/.

I see. Thanks.

Or we can use /#{target_namespace}/#{@package}/.

I have changed to use it.

kou · 2022-10-21T00:24:42Z

Could you update the description to match the latest behavior?

HashidaTKS · 2022-10-21T00:56:32Z

@kou

I have addressed your comments.
Please re-review this.

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

kou · 2022-10-21T03:30:08Z

Thanks!

HashidaTKS commented Oct 18, 2022

View reviewed changes

HashidaTKS added 2 commits October 19, 2022 12:03

Use .htaccess to download latest groonga

e0bca3a

Fix regexp form "not contain" to "not start with"

2c87903

HashidaTKS force-pushed the redirect-windows-packages-to-github branch from fbc1cc7 to 2c87903 Compare October 19, 2022 03:03

komainu8 reviewed Oct 19, 2022

View reviewed changes

kou changed the title ~~packages windows: redirect to Github Releases to download latest Groonga~~ packages windows: redirect to GitHub Releases to download latest Groonga Oct 20, 2022

kou reviewed Oct 20, 2022

View reviewed changes

HashidaTKS added 4 commits October 20, 2022 12:33

define new task for create .htaccess

4ab8c1f

Revert "Fix regexp form "not contain" to "not start with""

0bc5f7d

This reverts commit 2c87903.

Revert "Use .htaccess to download latest groonga"

434c276

This reverts commit e0bca3a.

Refactor

799df5e

kou reviewed Oct 20, 2022

View reviewed changes

HashidaTKS added 2 commits October 20, 2022 14:23

Use File.open

da8a584

Rename create_htaccess to prepare

55ca339

kou reviewed Oct 20, 2022

View reviewed changes

HashidaTKS added 5 commits October 20, 2022 16:17

Use "Redirect" instead of "RewiteRule"

aed7e5e

Use RedirectMatch instead of Redirect

7c1033b

Refactor download_packages

16ecc6a

Remove needless brackets

3eb60a9

Use "Redirect" instead of "RedirectMatch"

4cfc4f2

Fix a typo

a94f98d

Update packages/Rakefile

fadc356

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

kou merged commit e73a61e into groonga:master Oct 21, 2022

HashidaTKS mentioned this pull request Dec 6, 2022

[WIP] ansible: add htaccess groonga/packages.groonga.org#28

Closed

github-actions bot mentioned this pull request May 22, 2023

nginx: update bundles version 1.23.4 komainu8/groonga#1

Closed

	if (target_namespace == :windows)
	if target_namespace == :windows

packages windows: redirect to GitHub Releases to download latest Groonga #1432

packages windows: redirect to GitHub Releases to download latest Groonga #1432

Conversation

HashidaTKS commented Oct 18, 2022 • edited Loading

Choose a reason for hiding this comment

HashidaTKS Oct 18, 2022 • edited Loading

Choose a reason for hiding this comment

komainu8 Oct 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HashidaTKS commented Oct 19, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

komainu8 commented Oct 19, 2022

kou commented Oct 20, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HashidaTKS Oct 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HashidaTKS Oct 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HashidaTKS commented Oct 20, 2022

Choose a reason for hiding this comment

HashidaTKS Oct 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HashidaTKS Oct 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HashidaTKS Oct 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kou commented Oct 21, 2022

HashidaTKS commented Oct 21, 2022

kou commented Oct 21, 2022

HashidaTKS commented Oct 18, 2022 •

edited

Loading

HashidaTKS Oct 18, 2022 •

edited

Loading

komainu8 Oct 19, 2022 •

edited

Loading

HashidaTKS Oct 20, 2022 •

edited

Loading

HashidaTKS Oct 20, 2022 •

edited

Loading

HashidaTKS Oct 20, 2022 •

edited

Loading

HashidaTKS Oct 20, 2022 •

edited

Loading

HashidaTKS Oct 20, 2022 •

edited

Loading