Node.js Ops First with Pandora.js, Today

GuangWong edited this page Feb 23, 2018 · 1 revision

Node.js is moving fast, along with an extremely large ecosystem.

The primary concern is productivity at first (one language, frontend and backend). Various frameworks and tools boost the ecosystem a lot via enabling startups and large companies.

Nowadays, we realize that having a reliable workflow, from dev to ops, is Key to success. Dev is essential, but Ops take over the long term responsibility for day-to-day business. Be able to operate(monitoring, scaling, trouble-shooting, etc.) smoothly , is vital for business.

We call it ops-first, means that the operation concern, which is the long term concern, should be addressed at the very beginning.

So what is Pandora.js exactly?

Pandora.js is an application manager dedicated to Node.js applications. The first release is built by Taobao Midway team mostly, and involves contributors all over Alibaba group.

Thanks to the dev productivity brought by Node.js, a few lines of code can make a working web application. And Pandora.js can enable it with the operation capabilities, make it manageable, measurable, traceable and highly available.

Yes, this is the goal of Pandora.js, making node.js applications manageable, measurable, traceable and highly available, with little effort. Let's dive into the features:

  1. Manageable
    • Standard management capabilities of applications, processes and basic services (such as middlewares).
    • Graceful online/offline.
    • Inter-process object proxying.
  2. Measurable
    • Be able to measure different aspects of applications.
    • Support tons of metrics types, gauge, counter, meter, histogram, etc.
  3. Traceable
    • Be able to trace the whole execution stack, inspect applications at runtime.
    • Support tons of 3rd party middlewares, such as MySQL, redis, and many more.
    • Compatible with Open-Tracing standard
  4. Highly available
    • Provides mechanism to mock middleware failures, aims to enhance system resilience.
    • Do trouble shooting by remote debugging.

Besides capabilities above, Pandora.js has a built-in realtime dashboard with critical features, such as error logs, slow transactions, and custom metrics.

Community involvements are encouraged. You are more than welcome to

  • Use it, integrate it with your workflow.
  • Build your operation infrastructure upon it.
  • Send pull requests
  • Open issues
  • Star us :)

Btw. Pandora.js is written in TypeScript. TypeScript enthusiasts are welcome to drop by and we can share best practices with each others.

Why use Pandora.js?

We have 2 major Node.js application frameworks(Midway and Egg) and hundreds of Node.js applications in production in Alibaba. Operation and governance capabilities are provided as framework plugins at first. It works.

With the time passing, micro-service architecture becomes widely adopted. People don’t want to use full-stack frameworks to build micro-services. They prefer micro-frameworks or other suitable community solutions to build micro-services, e.g. a schedule push service, a stream pipe service.

So we decoupled the operation and governance capabilities from the application framework, and extract it into a single layer. Node.js applications with any framework upon this layer can have the operation and governance capabilities.

For community, it is easy for you to integrate Pandora.js with your own monitoring system(e.g. built on open-falcon, etc.) or business apm platform(e.g. new-relic, appdynamics, etc.)

The built-in dashboard showcased how to leverage those aggregated data. Features below require launching Node.js applications via Pandora.js only.

Detect slow transactions

It can trace your transactions and detect those slow ones and errors.

undefined

You can drill down to dig out the root causes. Related logs are provided nearby for further analysis.

undefined

Custom metrics

Metrics are vitally important for you to judge whether your Node.js apps are performing well and how you can improve.

Pandora.js provides lots of built-in metrics to measure applications. Other than system indicators and error indicators, we offer a bunch of Node.js specific metrics.

undefined

Custom metrics is another feature worth trying. You can define your own metrics to be recorded, collected and aggregated based on your business requirements.

Here is an example. Suppose you are requesting users’ avatars by their email addresses and you would like to know the success ratio. You can define your metrics as bellow:

const {MetricsClientUtil} = require('dorapan'); //dorapan is the client SDK of pandora.js
const client = MetricsClientUtil.getMetricsClient(); //get an instance of metrics client

// create 2 counters
const successCounter = client.getCounter('custom', 'custom.gravatar.success');
const failCounter = client.getCounter('custom', 'custom.gravatar.fail');

...

const userAvatars = {};
const promises = [];
for(const user of rows) {
  const profileUrl = gravatar.profile_url(user.email);
   promises.push(urllib.request(profileUrl, {
     followRedirect: true, dataType: 'json'
   }).then((res) => {
     if(typeof res.data === 'object') {
	   successCounter.inc(1);
       userAvatars[user.email] = res.data;
     } else {
	   failCounter.inc(1);
	 }
   }));
}
await Promise.all(promises);

By adding some custom counters in your applications, Pandora.js could help to collecte, aggregate, and choose various reporters to expose those data.

Here we can see the result from the built-in dashboard.

undefined

How can we use Pandora.js?

Installation

Prerequisites: Support *nix and MacOSX currently.

  1. Install Pandora.js cli
$ npm i pandora -g
  1. Install Pandora.js dashboard
$ npm i pandora-dashboard -g

Then you can start the dashboard

$ pandora start --name dashboard `pandora-dashboard-dir`

Here pandora-dashboard-dir is a command substitution. It outputs the exact directory where dashboard is installed. Pandora.js launches it as a common Node.js application.

Vist http://127.0.0.1:9081 to have a look at the dashboard and explore it by yourself.

An example application with MySQL CRUD

Sample code can be found here in github, repo. In this project, server.js is the entry file.

Before launch the app, you are suggested to do the init job first.

$ pandora init server.js 
? Which type do you like to generate ? (Use arrow keys)
  fork 
❯ cluster

Here you can choose cluster, by which you can take advantage of multi-core systems capabilities.

A procfile.js is generated after the initialization. It defines the process model for the app. You can customize your process model later as it is an advanced topic.

Now launch the app.

$ pandora start
Starting rest-crud at /xx/xxx/rest-crud
rest-crud started successfully! Run command [ pandora log rest-crud ] to get more information

Check out the dashboard now.

undefined

You can click around to see different types of data. Standard output.

undefined

Trace.

undefined

And Metrics.

undefined

For more information, please checkout the documents here: http://www.midwayjs.org/pandora/en/

What’s next

Pandora.js has been running in production for months. We keep on improving its usabilities and trying to enhance the cli. We are also working hard on the mechanism to mock middleware failures, help applications to improve the failure tolerance.

Stay tuned with our later releases via watching [github.com/midwayjs/pandora]

We would like very much to receive feedbacks from community. Pandora.js has no way being better without community.

Last, but not the least

Yes, we are hiring.

Join us, join Taobao, join the progress of Node.js ecosystem enabling New-Retail ecosystem, like one light enlighten another.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.