Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

To host stats.jenkins.io GSoC 2024 project in jenkins-infra #4132

Open
krisstern opened this issue Jun 7, 2024 · 49 comments
Open

To host stats.jenkins.io GSoC 2024 project in jenkins-infra #4132

krisstern opened this issue Jun 7, 2024 · 49 comments
Assignees
@lemeurherve
Labels
helpdesk Infrastructure related issues management in Github stats.jenkins.io
Milestone
infra-team-sync-2...

Comments

@krisstern
Copy link
Member

krisstern commented Jun 7, 2024 •
edited
Loading

Service(s)

Helpdesk, stats.jenkins.io

Summary

We are in the process of redeveloping the frontend for the data presentation of the Jenkins Infra Statitics project, and will need help migrating the GSoC contributor's repo (https://github.com/shlomomdahan/stats.jenkins.io/) from his personal GitHub account to one in the jenkins-infra org.

@lemeurherve @gounthar @shlomomdahan

Reproduction steps

N/A
@krisstern krisstern added the triage Incoming issues that need review label Jun 7, 2024
@jenkins-infra-helpdesk-app jenkins-infra-helpdesk-app bot added helpdesk Infrastructure related issues management in Github stats.jenkins.io labels Jun 7, 2024
@lemeurherve lemeurherve self-assigned this Jun 7, 2024
@lemeurherve lemeurherve removed triage Incoming issues that need review helpdesk Infrastructure related issues management in Github labels Jun 7, 2024
@dduportal dduportal added the triage Incoming issues that need review label Jun 11, 2024
@dduportal dduportal added this to the infra-team-sync-2024-06-18 milestone Jun 11, 2024
@dduportal dduportal removed the triage Incoming issues that need review label Jun 11, 2024
@jenkins-infra-helpdesk-app jenkins-infra-helpdesk-app bot added the helpdesk Infrastructure related issues management in Github label Jun 11, 2024
@krisstern krisstern changed the title To host https://github.com/shlomomdahan/stats2.jenkins.io/ docs in jenkins-infra for GSoC 2024 To host https://github.com/shlomomdahan/stats.jenkins.io/ docs in jenkins-infra for GSoC 2024 Jun 12, 2024
@lemeurherve
Copy link
Member

@shlomomdahan could you initiate the transfer of your repository to https://githubcom/jenkins-infra please?

See https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository#transferring-a-repository-owned-by-your-personal-account

@lemeurherve
Copy link
Member

lemeurherve commented Jun 14, 2024 •
edited
Loading

Action plan

Similar to:

@lemeurherve
Copy link
Member

lemeurherve commented Jun 14, 2024 •
edited
Loading

@krisstern transfered the repository to jenkins-infra org (thanks!).

I've set @krisstern as "maintainer", and https://github.com/orgs/jenkins-infra/teams/core as "admin".

@krisstern
Copy link
Member Author

Thanks @lemeurherve!

This was referenced Jun 14, 2024
Merged
Merged
@dduportal
Copy link
Contributor

Question about Fastly: what is the rationale behind using Fastly in this case (compared to serving the website ourselves)?

@lemeurherve
Copy link
Member

lemeurherve commented Jun 14, 2024 •
edited
Loading

Using it as CDN, same logic as for other websites like jenkins.io, stories.jenkins.io, contributors.jenkins.io, docs.jenkins.io.

The website would be served from a nginx on publick8s at (new.)stats.origin.jenkins.io like the others.

Am I missing something?

@dduportal
Copy link
Contributor

Using it as CDN, same logic as for other websites like jenkins.io, stories.jenkins.io, contributors.jenkins.io, docs.jenkins.io.

The website would be served from a nginx on publick8s at (new.)stats.origin.jenkins.io like the others.

Am I missing something?

For jenkins.io, the rationale has always been because of the amount of requests costs too much in outbound bandwidth. But it was measured.

do we have current stats for stats.jenkins.io ?

the question is related to: do we need a CDN ? If no then what is the point.

the same question I have for stories and contributors but i never had the answer or at least i do not see anything around numbers and facts for it.

@lemeurherve
Copy link
Member

lemeurherve commented Jun 14, 2024 •
edited
Loading

As stats.jenkins.io is currently hosted on GitHub Pages, I'm not sure we can access traffic stats from it.
I found https://github.blog/2014-01-07-introducing-github-traffic-analytics/ but it doesn't seem active anymore.

Then I don't think there's a lot of traffic on these pages, we can start serving it without Fastly then see what are the actual numbers in term of visits.

Removing the Fastly step points and reworking jenkins-infra/azure-net#257

@lemeurherve lemeurherve mentioned this issue Jun 16, 2024
Merged
@lemeurherve
Copy link
Member

FTR, I invited @shlomomdahan as writer on stats.jenkins.io.

Could be revised when a dedicated team is created.

@lemeurherve
Copy link
Member

Opened jenkins-infra/stats.jenkins.io#3 to add a pipeline for building the website on ci.jenkins.io and infra.ci.jenkins.io (pending jenkins-infra/kubernetes-management#5310).

@lemeurherve lemeurherve mentioned this issue Jun 16, 2024
Merged
@krisstern
Copy link
Member Author

Hi @lemeurherve, would it be possible to add @gounthar as a co-maintainer?

@lemeurherve
Copy link
Member

lemeurherve commented Jun 17, 2024 •
edited
Loading

https://github.com/orgs/jenkins-infra/teams/stats-jenkins-io team including you and @gounthar created and set as "maintainer" on https://github.com/jenkins-infra/stats.jenkins.io/.

This was referenced Jun 17, 2024
Merged
Merged
Merged
This was referenced Jun 20, 2024
Merged
Merged
@lemeurherve
Copy link
Member

lemeurherve commented Jun 20, 2024 •
edited
Loading

I had to push an hotfix with the correct STATS_SERVICE_PRINCIPAL_WRITER_CLIENT_ID value to fix production deployment error: https://github.com/jenkins-infra/charts-secrets/commit/4dca10bd26da859393aedff61ec672df0f297464 (private repo)

The CD part is now (almost) ready, https://new.stats.jenkins.io content is updated continuously on every new push on the main branch of the repository.

@lemeurherve
Copy link
Member

lemeurherve commented Jun 20, 2024 •
edited
Loading

After discussing with @krisstern about jenkins-infra/stats.jenkins.io#2 (comment)

this project is a part of the ongoing GSoC 2024 and is not in production yet. There will be many additions to its documentation as the project progress. Currently it is still in its development phase.

I propose to temporarily restrict repository interactions to "prior contributors" (include @jenkins-infra members) on https://github.com/jenkins-infra/stats.jenkins.io:

image

And to (optionally) open a pinned issue in https://github.com/jenkins-infra/stats.jenkins.io to have a visible public status about this restriction.

WDYT?

@MarkEWaite
Copy link

I propose to temporarily restrict repository interactions to "prior contributors"

That sounds fine to me.

This was referenced Jun 20, 2024
Merged
Merged
lemeurherve added a commit to jenkins-infra/azure that referenced this issue Jun 21, 2024
@lemeurherve
chore(updatecli): track `infraci_stats_jenkins_io_fileshare_servicepr…
24d044a 
…incipal_writer` end date (#737)

This PR adds an updatecli manifest to track
`infraci_stats_jenkins_io_fileshare_serviceprincipal_writer` end date.

Proposed PR body:

>
![image](https://github.com/jenkins-infra/azure/assets/91831478/ad015472-2440-466d-b565-0a5c57950010)

Test:
<details><summary>updatecli diff --values updatecli/values.yaml --config
updatecli/updatecli.d/stats.jenkins.io.sp.endate.yaml</summary>

```console
+++++++++++
+ PREPARE +
+++++++++++

Loading Pipeline "updatecli/updatecli.d/stats.jenkins.io.sp.endate.yaml"

SCM repository retrieved: 1


++++++++++++++++++
+ AUTO DISCOVERY +
++++++++++++++++++



++++++++++++
+ PIPELINE +
++++++++++++



##################################################################################
# GENERATE NEW END DATE FOR STATS.JENKINS.IO FILE SHARE SERVICE PRINCIPAL WRITER #
##################################################################################


SOURCES
=======

nextExpiry
----------
The shell 🐚 command "/bin/sh /var/folders/ly/77mv7l9968g068w9_fwzwggw0000gp/T/updatecli/bin/bd1b66faf6e39c527f085eea5b269bba76c890efa7550e4ca84317fcabb6e3f1.sh" ran successfully with the following output:
----
2024-09-18T00:00:00Z
----
✔ shell command executed successfully

currentExpiry
-------------
✔ value "\"2023-09-19T23:00:00Z\"" found for key "$.end_dates.infraci_stats_jenkins_io_fileshare_serviceprincipal_writer" in the yaml file "locals.yaml"


CONDITIONS:
===========

checkIfExpirySoonExpired
------------------------
The shell 🐚 command "/bin/sh /var/folders/ly/77mv7l9968g068w9_fwzwggw0000gp/T/updatecli/bin/56202140e385731a2fc616bc17989ad91eed830fd60ba795da3fdd132df9fd37.sh" ran successfully with the following output:
----
time for update
----
✔ shell condition of type "console/output", passing


TARGETS
========

updateNextExpiry
----------------

**Dry Run enabled**

⚠ - change detected:
        * key "$.end_dates.infraci_stats_jenkins_io_fileshare_serviceprincipal_writer" should be updated from "\"2024-09-19T23:00:00Z\"" to "2024-09-18T00:00:00Z", in file "locals.yaml"


ACTIONS
========


Generate new end date for stats.jenkins.io File Share service principal writer - default
----------------------------------------------------------------------------------------

[Dry Run] An action of kind "github/pullrequest" is expected.

=============================

REPORTS:



⚠ Generate new end date for stats.jenkins.io File Share service principal writer:
        Source:
                ✔ [currentExpiry] Get current `end_date` date
                ✔ [nextExpiry] Prepare next `end_date` date within 3 months
        Condition:
                ✔ [checkIfExpirySoonExpired] 
        Target:
                ⚠ [updateNextExpiry] generate new end date 2024-09-18T00:00:00Z for stats.jenkins.io File Share service principal writer on infra.ci.jenkins.io


Run Summary
===========
Pipeline(s) run:
  * Changed:    1
  * Failed:     0
  * Skipped:    0
  * Succeeded:  0
  * Total:      1
 ```
 
 </details>
 
 Follow-up of:
- #734

Ref:
- #733 (comment)
- jenkins-infra/helpdesk#4132 (comment)
@lemeurherve
Copy link
Member

lemeurherve commented Jun 21, 2024 •
edited
Loading

A temporary interaction limit to prior contributors has been added to https://github.com/jenkins-infra/stats.jenkins.io/ for a month:

image

And an event in Jenkins Infra calendar pointing to this comment to renew the limit or not in a month.

@lemeurherve lemeurherve mentioned this issue Jun 21, 2024
Closed
@krisstern
Copy link
Member Author

@lemeurherve Please add @Vandit1604 to the team with "triage" rights.

@lemeurherve
Copy link
Member

lemeurherve commented Jun 23, 2024 •
edited
Loading

@krisstern added as "triage":

image

@Vandit1604 you should have received an invitation to join @jenkins-infra and accept this role, please note that it will expire in a few days.

@Vandit1604
Copy link

Thanks, @lemeurherve
I have accepted it.

@krisstern
Copy link
Member Author

Hi @lemeurherve, is site preview for the PR's not an option at the moment? I am currently seeing the following:

Screenshot 2024-06-23 at 4 16 36 PM

@lemeurherve
Copy link
Member

lemeurherve commented Jun 23, 2024 •
edited
Loading

@krisstern: as @shlomomdahan doesn't have write permissions on the repo for now, builds on infra.ci.jenkins.io aren't triggered. (Security measure)

We discussed the case with @dduportal and came to the conclusion that we should create a distinct infra.ci.jenkins.io pipeline dedicated to previews that could be triggered without this requirement, while keeping the "production deployment" one restricted.

WIP, incoming soon.

This was referenced Jun 23, 2024
Merged
Merged
@lemeurherve
Copy link
Member

we should create a distinct infra.ci.jenkins.io pipeline dedicated to previews that could be triggered without this requirement, while keeping the "production deployment" one restricted.

WIP, incoming soon.

Should be addressed by:

@lemeurherve
Copy link
Member

lemeurherve commented Jun 24, 2024 •
edited
Loading

Note that in order to implement jenkins-infra/stats.jenkins.io#53 I made the compromise of duplicating "Install dependencies" and "Build" stages between this new pipeline dedicated to previews and the main one.

Currently reworking jenkins-infra/kubernetes-management#5345 after discussing it with @dduportal and @smerle33:

  • Existing stats.jenkins.io job on infra.ci.jenkins.io
  • New stats.jenkins.io_preview job on infra.ci.jenkins.io
    • Needs to review "allowUntrustedChanges" so it does exactly what we want
      • To be developed in a separate help desk issue where this whole process will be extended to all websites built on infra.ci.jenkins.io
    • GitHub checks enabled
    • Will exclude main branch from build as previews don't make sense for it

@lemeurherve
Copy link
Member

lemeurherve commented Jun 24, 2024 •
edited
Loading

Previews are now working on all stats.jenkins.io pull requests, even those from contributors without write permission on the repository:

image

@krisstern
Copy link
Member Author

Thanks @lemeurherve!

@krisstern
Copy link
Member Author

@lemeurherve Could you please let us know what will be the arrangement for the data to be fetched for the project. Currently this appears to be a major blocker. Will data be fetched internally or externally?

@lemeurherve
Copy link
Member

An helpdesk issue should be open to discuss how to put in place a public storage of the processed data from https://github.com/jenkins-infra/infra-statistics so they could be fetched when building https://github.com/jenkins-infra/stats.jenkins.io.

This was referenced Jun 26, 2024
Closed
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lemeurherve lemeurherve

Labels
helpdesk Infrastructure related issues management in Github stats.jenkins.io
Projects
None yet
Milestone
infra-team-sync-2024-07-16
Development

No branches or pull requests
8 participants
@gounthar @MarkEWaite @dduportal @timja @shlomomdahan @krisstern @lemeurherve @Vandit1604