feat: add changes stream syncer #970

Merged
merged 2 commits into from Jun 25, 2016

Projects

None yet

4 participants

@fengmk2
Member
fengmk2 commented Jun 25, 2016

fix scope package missing problem

close #908

@fengmk2 fengmk2 added the feat label Jun 25, 2016
@fengmk2 fengmk2 added this to the 2.x milestone Jun 25, 2016
@dead-horse dead-horse was assigned by fengmk2 Jun 25, 2016
@mention-bot

By analyzing the blame information on this pull request, we identified @dead-horse to be a potential reviewer

@fengmk2
Member
fengmk2 commented Jun 25, 2016

image

@codecov-io
codecov-io commented Jun 25, 2016 edited

Current coverage is 88.47%

Merging #970 into master will decrease coverage by 0.16%

@@             master       #970   diff @@
==========================================
  Files            85         85          
  Lines          3291       3296     +5   
  Methods         340        340          
  Messages          0          0          
  Branches        613        615     +2   
==========================================
- Hits           2917       2916     -1   
- Misses          374        380     +6   
  Partials          0          0          

Sunburst

Powered by Codecov. Last updated by ee1e084...2c18f6e

@fengmk2 fengmk2 commented on an outdated diff Jun 25, 2016
config/index.js
@@ -209,6 +197,10 @@ var config = {
// sync devDependencies or not, default is false
syncDevDependencies: false,
+ // changes streaming sync
+ changesStreamingSync: false,
@fengmk2
fengmk2 Jun 25, 2016 Member

只需要在内网开启 changesStreamingSync ,并且设置 handleSyncRegistry 为 tnpm registry 即可。

@dead-horse dead-horse commented on the diff Jun 25, 2016
sync/changes_stream_syncer.js
+ });
+ changes.await = streamAwait;
+ changes.on('data', change => {
+ logger.syncInfo('Get change: %j', change);
+ syncPackage(change);
+ });
+
+ yield changes.await('error');
+};
+
+function syncPackage(change) {
+ const url = `${config.handleSyncRegistry}/${change.id}/sync`;
+ urllib.request(url, {
+ method: 'PUT',
+ dataType: 'json',
+ timeout: 10000,
@dead-horse
dead-horse Jun 25, 2016 Member

这个接口只同步这个模块,不会同步他的依赖吧? 10s 够么?

@fengmk2
fengmk2 Jun 25, 2016 Member

只是调用一下创建同步任务,不关心是否真正同步成功。即使失败,原来的定时同步在10分钟后也会确保同步成功。

@dead-horse
dead-horse Jun 25, 2016 edited Member

记起来了,查询是否成功日志的接口是另外的

@dead-horse dead-horse commented on an outdated diff Jun 25, 2016
sync/changes_stream_syncer.js
+
+function syncPackage(change) {
+ const url = `${config.handleSyncRegistry}/${change.id}/sync`;
+ urllib.request(url, {
+ method: 'PUT',
+ dataType: 'json',
+ timeout: 10000,
+ }, (err, data, res) => {
+ if (err) {
+ logger.syncInfo('%s:%s PUT %s error: %s, retry after 5s',
+ change.seq, change.id, url, err);
+ logger.syncError(err);
+ syncPackage(change);
+ setTimeout(() => {
+ syncPackage(change);
+ }, 5000)
@dead-horse
dead-horse Jun 25, 2016 Member

分号, setTimeout 写一行吧

@dead-horse dead-horse commented on an outdated diff Jun 25, 2016
sync/index.js
@@ -132,6 +119,32 @@ if (config.syncPopular) {
logger.syncInfo('sync popular module disable');
}
+if (config.changesStreamingSync) {
+ const sync = require('./changes_stream_syncer');
+ let syncing = false;
+ const syncFn = co.wrap(function* syncPopular() {
@dead-horse
dead-horse Jun 25, 2016 Member

syncChangeStream

@dead-horse dead-horse commented on an outdated diff Jun 25, 2016
sync/index.js
- debug('start syncing');
- var data;
- var error;
- try {
- data = yield sync();
- } catch (err) {
- error = err;
- error.message += ' (sync package error)';
- logger.syncError(error);
- }
- data && logger.syncInfo(data);
- if (!config.debug) {
- sendMailToAdmin(error, data, new Date());
+if (sync) {
+ // the same time only sync once
+ var syncing = false;
@dead-horse
dead-horse Jun 25, 2016 Member

var 都改成 let 吧,下面也有一个 syncing ,看的容易混

fengmk2 added some commits Jun 25, 2016
@fengmk2 fengmk2 feat: add changes stream syncer
fix scope package missing problem

close #908
95c7a33
@fengmk2 fengmk2 refactor: use let instead of var
2c18f6e
@fengmk2
Member
fengmk2 commented Jun 25, 2016

@dead-horse 都改了

@dead-horse dead-horse merged commit 6485e4b into master Jun 25, 2016

2 of 4 checks passed

Node Security 3 vulnerabilities found
Details
codecov/project 88.47% (-0.17%) compared to ee1e084
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@dead-horse dead-horse deleted the chagne-stream-syncer branch Jun 25, 2016
@dead-horse
Member

+1

@dead-horse
Member

2.11.0

@fengmk2
Member
fengmk2 commented Jun 25, 2016

哈哈,马上被捕获到

image

@fengmk2
Member
fengmk2 commented Jun 25, 2016 edited

'2.11.0': '2016-06-25T13:07:08.649Z', 1s 之后被实时同步


  [2016-06-25 21:07:09.492][12064] Get change: {"seq":2615175,"id":"cnpmjs.org","changes":[{"rev":"293-837113a82174791d334fe1980a4296f8"}]} +6s
  [2016-06-25 21:07:09.526][12064] 2615175:cnpmjs.org sync request sent, log: 
@dead-horse
Member

发布之后不需要手动执行 tnpm sync 了

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment