New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

展示打包历史 #794

Closed
lilydjwg opened this Issue Jul 3, 2018 · 55 comments

Comments

Projects
None yet
5 participants
@lilydjwg
Copy link
Member

lilydjwg commented Jul 3, 2018

问题类型 / Type of issues

  • 其它 / other

通过网页展示:

  • 目前所有自动构建包的最后一次打包时间、版本(nvchecker 检测结果)和状态(成功或者失败)
  • 每个包的构建历史时间、版本和状态
  • 「成功指数」:使用滑动平均值来展示最近这个包有多么成功(或者不成功)

RFC: 记录每次打包所花费的时间?

@lilydjwg lilydjwg added the enhancement label Jul 3, 2018

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 3, 2018

好棒,不过 lilac 目前没把数据存到数据库里吧?

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 3, 2018

嗯。不过 lilac 有记 build.log 哦。可以从那里解析出数据存数据库里去。

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 6, 2018

这东西现在在做吗?。我(ˇˍˇ) 想~ 做。。。。还有 build.log 在哪看?

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 6, 2018

还没开始做呢。build.log 位于 ~lilydjwg/.lilac/build.log。

建议的读取方式是拿 inotify 去读取这个文件的更新,存到 sqlite3 数据库里,然后记录一下读到的文件偏移(方便重启后继续)。

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 6, 2018

我读不了那个文件。。。。

分成三部分岂不更好

  1. 解析并发送程序
  2. 后台api服务端
  3. 前端页面

inotify 读取文件并解析,发个服务端,服务端提供标准的api接口,前端使用api并呈现

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 6, 2018

可以读的啊:
tail build.log

不要想着列我家里的东西啦。

是啊,分成三个部分做。不过我希望解析和 Web 后端放一个程序里边,方便管理。

PS: 切到你的用户时 nodejs 报了一堆错呢……

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 6, 2018

日志好简单。。。我还以为多复杂那。。这么简单,解析应该丢后端(写文件时触发事件,丢给服务端)

数据库我推荐PostgreSQL。。。(en,两张表应该就足够了)

日志可以软连接到 /var/log/ 下吗?

PS: 暴露了我在家目录装 nodejs 的事了。。。

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 6, 2018

为什么要软链接到 /var/log?
我懒得维护数据库服务啊喵。

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 6, 2018

额。。第一反应找 lilac 的日志应该在这里

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 7, 2018

百合百合
以这条数据为例[2018-07-07 18:12:17] python-git 20180707.062835 successful after 292s

  • [2018-07-07 18:12:17] time 时间 (具体什么时间?写入时间?)
  • python-git pkgname 包名
  • 20180707.062835 time_ 这是什么时间
  • successful building_status 状态 布尔型?True or False
  • after (舍掉?)
  • 292s building_time 构建时长

两张数据表:

  • building_latest
  • building_log
@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 7, 2018

开头的是日志的写入时间,也就是打包完成的时间(CST)。包名后边那个是 nvchecker 报告的版本号(可能有空格)。结果是「successful」或者「failed」。「after」子句有可能没有的。

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 7, 2018

表名叫「build_xxx」比较好吧。

你大概需要在 building_latest 里记录当前成功了几次失败了几次,方便在总览页面展示。

@a-wing a-wing self-assigned this Jul 7, 2018

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 8, 2018

关于解析还有几个问题

  1. 时间都是 +0800 时区吧?好像是。。问的好像多余。。。
  2. after 是和构建时长一定是同时出现的吗?
  3. 构建时长 的时间的单位一定是秒吗?好像是
@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 8, 2018

  1. 对。我之前说了。
  2. 是。
  3. 是。
@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 8, 2018

今天肝不动了。。。差不多下周日上线吧(做解析数据的功能花时间有点多)

  • 发送的端 shell 脚本。。。
  • 后端 ruby on rails 一把梭(功能还差点)
  • 前端准备用 vue.js 全家桶写那。。。。

https://github.com/a-wing/KISS2U

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 9, 2018

WOW 好快,也有好多文件啊……

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 12, 2018

目前后端完成的功能:(还有没有落下的。。)

  • 提交数据认证, hmac 认证
  • 过滤重复提交日志,已 lilac 日志写入时间为基准
  • 日志解析 (包括版本号有空格的情况)
  • latest 表,successful or failed 次数

没要加功能的我就把他扔到服务器上了。。。

PS: 其实大部分文件是rails的脚手架生成的。。。。还有发送端 shell 部分有没有人想写啊?

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 12, 2018

你在编译机上跑起来试试看?

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 12, 2018

其实我想要个PostgreSQL 的数据库权限喵。还有要用sqlite3 来做数据库?。。。

还有要不要在在编译机上装个bundle。。。还是在我家目录下搞个虚拟的ruby 环境

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 12, 2018

编译机上就没有 PosstgreSQL 的吧。

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 12, 2018

http://build.archlinuxcn.org:22333/packages/

添加了三条测试数据。。。还行

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 12, 2018

还没有前端呀?

building_status true 是什么鬼……注意语义,用「success」和「failure」之类的比较好。另外我觉得它应该叫「building_result」。

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 12, 2018

另外 Cache-Control 可以写成 max-age=60, public

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 12, 2018

前端还没开始写那(有种周日要跳票的感觉)。。。building_result 还行。。。布尔型,从数据库取出来肯定是 T 或 F 啊。转换成「success」和「failure」应该在前端实行啊。

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 12, 2018

不对。你的数据应当在每一个地方都尽力做到自解释,这样无论以后维护还是别人学习都不会弄错。

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 13, 2018

要不 building_result 改成 building_success 值为「true」和「false」 ,这样就清晰了

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 13, 2018

叫 building_successful 比较符合语法。不过 building_ok 比较短?

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 14, 2018

百合百合。。编译机有 nginx 吧。。。

临时看:http://build.archlinuxcn.org:8033/

把我家下的这个目录当静态网站
.service/KISS2UI/

然后应该就可以看到效果了。。。。

或许该缕一缕前端页面的需求了。。

PS:明天不想写代码了。。。明天想出去玩 喵。。。(先上线,再迭代。。)逃

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 14, 2018

哪能直接把 git 仓库放 nginx 呢。另外准备一个目录来 serve 静态文件吧。另外为了安全起见,用户家目录都是 700 的,nginx 进不去。所以我给你建了一个 /srv/packages 目录,你可以把静态文件复制过来。
访问路径目前是 /packages/。

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 14, 2018

git 仓库。。。额。我 travis-ci 加上不用 git 仓库了。。。(<== 暴露安全意识薄弱)

还有他会异步加载 http://build.archlinuxcn.org:22333/packages/ 里的数据。。。然后那个没https。。浏览器会禁止加载(做个代理吧。。。)

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 14, 2018

可是我代理到哪个路径下呢?对了你这个 RoR 没有安全隐患吧?要不要我用单独的用户 + 限制权限来跑?

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 14, 2018

要不前端静态用 package/
RoR 后端用 packages/

RoR 安全隐患。。。。(应该没有。。。)
不觉得麻烦的话就 单独的用户 + 限制权限来跑 (我自己部署RoR 项目的时候都是新建用户来跑的。。。)

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 14, 2018

你就不会 api/packages/ 么……

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 14, 2018

可以啊。。。。。(你都为我想好了路径还来问我。。。)

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 14, 2018

好了。

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 14, 2018

百合百合。。。你忘记把 /api/packages/ 重写成 /packages/ 了。。。。

Started GET "/api/packages/" for <手动打码> at 2018-07-14 22:56:56 +0800
ActionController::RoutingError (No route matches [GET] "/api/packages"): 
@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 14, 2018

好了。

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 16, 2018

前端页面上的排序完全无效啊。

我在想,前端主页面的展示

  • 排序默认最近的最先
  • 可以按包名搜索
  • 可以只显示打包失败者
  • 时间显示为几小时几分钟这样
  • 成功率感觉没太大意义,因为打包次数不一样

然后每个包一个专门的展示历史的页面,就像 Travis-CI 的 build history 那个样子?

啊对了,太老的信息就不要展示了吧?或者过滤一下当前存在于仓库中的包?

@axionl

This comment has been minimized.

Copy link
Member

axionl commented Jul 16, 2018

对于 Pkgver 等若干字段,建议加上 filter 实现,不然排序意义不大。

顺便补上一个 Build history 图:

image

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 16, 2018

前端页面上的排序完全无效啊。(有效啊,1K多条数据有点延时很正常)

  • 包名目前没实现排序(没找着合适的给包名排序的算法)。其实pkgver 排序也比较蛋疼(有些版本号比较奇怪)

  • 成功率也没法实现排序(成功率是动态计算的)不过这个要去掉了

  • 排序默认最近的最先 enen。目前是最新的数据在最下面,反转一下就好

  • 数据搜索:

    • 可以按包名搜索
    • 可以只显示打包失败者
  • 时间显示为几小时几分钟这样。(JavaScript 的控制时间的库很蛋疼。要不定死 +0800 时区?)

  • 然后每个包一个专门的展示历史的页面,就像 Travis-CI 的 build history 那个样子?

  • 太老的信息就不要展示 (要不把build.log 最后 1K 条数据作为基础数据?)

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 16, 2018

@axionl

对于 Pkgver 等若干字段,建议加上 filter 实现,不然排序意义不大。

你说的是搜索的功能吗?

@axionl

This comment has been minimized.

Copy link
Member

axionl commented Jul 16, 2018

@a-wing 搜索功能也不错(但是实际上我表达的是过滤出某个包/组的历史

@yan12125

This comment has been minimized.

Copy link
Member

yan12125 commented Jul 16, 2018

Build Status的按鈕是預計加入顯示build log的功能嗎?

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 16, 2018

@a-wing 那个时间,我说的是打包用时啦。
可以从最后1k条开始导入数据库,但是以后呢?好像过滤掉已经从仓库中删除的包比较合理(那么就需要定理检查并在数据库里标记包状态了)。

@yan12125 好主意。不过得先等 lilac 那边改进日志的记录。

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 16, 2018

@axionl @yan12125 单个包的building log 后端有接口 比如:获取mpv-git 的构建历史

curl https://build.archlinuxcn.org/api/packages/mpv-git

显示历史 build log 是要加的功能(我还想加个统计的图表那。。。)
Build Status的按鈕 我就是借用一下人家写好的样式喵。。。。

@lilydjwg 过滤掉已删除包。。。KISS2U 并没实现这个功能。。。(还要加张表。。。把nvchecker.ini导入进去。。。)

@yan12125

This comment has been minimized.

Copy link
Member

yan12125 commented Jul 16, 2018

单个包的building log 后端有接口 比如:获取mpv-git 的构建历史

啊 我一開始心中想的是makechrootpkg的輸出0.0 常常要登入ssh看log覺得挺麻煩xd

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 16, 2018

@a-wing 不用啊。你可以定期读取软件包列表(读 archlinuxcn.db 好了),然后生成一个文件留着用的。

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 16, 2018

@yan12125 等 lilac 把日志分开记之后,这个就简单了啦。

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 20, 2018

这个编译机上没装啊。。。。

inotifywait.sh: line 11: /usr/bin/inotifywait: No such file or directory

@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 20, 2018

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 21, 2018

@felixonmars

This comment has been minimized.

Copy link
Member

felixonmars commented Jul 21, 2018

Installed.

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Jul 31, 2018

  • 排序默认最近的
  • 数据搜索:
    • 可以按包名搜索(可以用正则)
    • 可以只显示打包失败者 (好像没必要,有排序)
  • 时间显示为几小时几分钟这样。(根据浏览器时区来显示)
  • 然后每个包一个专门的展示历史的页面,就像 Travis-CI 的 build history 那个样子?
  • 太老的信息就不要展示

待完成:

  • UI 太丑。。。
  • 爪机端的支持
  • 还有什么???
@lilydjwg

This comment has been minimized.

Copy link
Member Author

lilydjwg commented Jul 31, 2018

包名不要折行啦。

@a-wing

This comment has been minimized.

Copy link
Member

a-wing commented Aug 5, 2018

都拖了一个月了。。。。该关掉了

kiss2u 地址(最主要的地址):https://build.archlinuxcn.org/packages/

关于 Archlinux CN KISS2U 的介绍:https://a-wing.top/archlinux/2018/08/05/kiss2u.html

@a-wing a-wing closed this Aug 5, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment