Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for OneDev devops platform #423

Merged
merged 6 commits into from Sep 1, 2021
Merged

Add support for OneDev devops platform #423

merged 6 commits into from Sep 1, 2021

Conversation

KN4CK3R
Copy link

@KN4CK3R KN4CK3R commented Jul 2, 2021

fixes #422

This PR adds support for OneDev. As OneDev lacks some needed API endpoints this Pull Request needs to merged first:

grafik

@alexwaibel
Copy link
Collaborator

Thank you for taking up this feature. Could you please add some test cases to cover this new integration? I would also appreciate if you could create a PR on the documentation website repo to cover the new config options and link to it here in this PR.

@KN4CK3R
Copy link
Author

KN4CK3R commented Jul 15, 2021

I hope I used the correct repository for the documentation: eduardoboucas/staticman.net#28
Your link leads to a Github app?

Copy link
Collaborator

@alexwaibel alexwaibel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some small suggestions. Thanks again for submitting this and for your patience with the review. Looking forward to getting this merged

source/lib/OneDev.js Outdated Show resolved Hide resolved
source/lib/OneDev.js Outdated Show resolved Hide resolved
test/unit/lib/OneDev.test.js Outdated Show resolved Hide resolved
@alexwaibel
Copy link
Collaborator

I hope I used the correct repository for the documentation: eduardoboucas/staticman.net#28

Your link leads to a Github app?

Sorry, I linked from mobile and GH must redirect to the app. This is indeed the correct repo. I'll review but wait to approve the docs change till this PR is also ready.

@arturhefczyc
Copy link

Hi,

I am trying to test 1dev support. It is not clear to me however if the implementation supports setting 1dev through env variables.
Documentation suggest that setting through config file is only supported using onedevBaseUrl and so on. Would these work if I set these variables as ENV variables?

This is essential to run Staticman from Docker.

@KN4CK3R
Copy link
Author

KN4CK3R commented Aug 12, 2021

The ENV variables are ONEDEV_BASE_URL, ONEDEV_USERNAME and ONEDEV_PASSWORD.

@arturhefczyc
Copy link

Also, there is staticman section in project's _config.yml file.
What should be setting for endpoint for OneDev?

@KN4CK3R
Copy link
Author

KN4CK3R commented Aug 12, 2021

Have a look at the documentation: https://github.com/eduardoboucas/staticman.net/pull/28/files
The default endpoint is https://code.onedev.io/api but you should use your instance.

@KN4CK3R
Copy link
Author

KN4CK3R commented Aug 13, 2021

I have added the suggestions and mentioned the env variables in the doc PR.

@arturhefczyc
Copy link

arturhefczyc commented Aug 16, 2021

Thank you for relying but I was asking about Staticman endpoint configuration in _config.yml file.

  • For GitHub it is: https://{your Staticman v3 API}/v3/entry/github/
  • For GitLab this is: https://{your Staticman v3 API}/v3/entry/gitlab/

So, is there a corresponding https://{your Staticman v3 API}/v3/entry/onedev/ endpoint?

@arturhefczyc
Copy link

Looks like there is onedev endpoint after all.

However, and unfortunately I cannot get it working. After switching my Staticman setup from GitLab (working) to my own OneDev installation it no longer works.

I am trying to test it from command line using curl:

curl -X POST http://staticman.host:8880/v3/entry/onedev/kobit/myrepo/master/comments \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-raw 'fields%5Bname%5D=Richard&fields%5Bmessage%5D=This+is+a+test+comment.&fields%5Bemail%5D=user@domain.com'

Such a call works with my GitLab setup, however, when I change setup to my OneDev installation I am consistently getting the error:

{"success":false,"rawError":"RequestError: connect ECONNREFUSED 127.0.0.1:443"}

I spent some time to dig into this and found lots of references related to a similar error coming from using got package. Like this:
https://forums.meteor.com/t/solved-problems-with-using-the-got-http-package/54297

There is more on the internet.

Looking at Staticman source code I can see that implementation for GitHub and GitLab do not use got package. This is only used by OneDev implementation. Hence the problem happens only for OneDev.

Unfortunately I have no idea on how to fix it or even implement some workaround.

So any hints on how to get it working would be very much appreciated.

I can provide more details on my environment if necessary.

@KN4CK3R
Copy link
Author

KN4CK3R commented Aug 17, 2021

Does your OneDev instance use https? I used OneDev in a docker container with http for my tests.

@arturhefczyc
Copy link

Yes, my OneDev runs over HTTPS. It is a bit more complex though.
I run OneDev in Docker as well with plain HTTP but I also have nginx proxy manager in front which takes care of SSL and certificates.

@arturhefczyc
Copy link

I reconfigured Staticman to have it connect directly to 1dev over http. This time I got a different error:

{"success":false,"rawError":"HTTPError: Response code 404 (Not Found)"}%

However, the error does not have any details, therefore I have no idea what was not found.

@KN4CK3R
Copy link
Author

KN4CK3R commented Aug 18, 2021

I have tested it again with https://code.onedev.io and got the same ECONNREFUSED error. I played with the dependencies and the mailgun-js dependency is the reason for the error. It uses proxy-agent which eventually uses agent-base which causes the error.

Workaround:

const http = require('http');
const https = require('https');
const urlToOptions = require('got/dist/source/core/utils/url-to-options').default;

export default class OneDev extends GitService {
  constructor(options = {}) {
    super(options.username, options.repository, options.branch);

    this.api = got.extend({
      prefixUrl: config.get('onedevBaseUrl'),
      username: config.get('onedevUsername'),
      password: config.get('onedevPassword'),
      responseType: 'json',
      request: (url, options, callback) => {
        if (url.protocol === 'https:') {
            return https.request({...options, ...urlToOptions(url)}, callback);
        }
        return http.request({...options, ...urlToOptions(url)}, callback);
      }
    });
  }

Or migrate from mailgun-js to the official mailgun.js library. //cc @alexwaibel

@KN4CK3R
Copy link
Author

KN4CK3R commented Aug 18, 2021

For your 404 error: You need to specify the repository id instead of the repository name in the url. The OneDev api sadly does not use the names. Here is how I tested it:

  1. Create a repository in OneDev:
    https://code.onedev.io/projects/test-staticman
  2. Push the staticman.yml
  3. Get the id of the repository:
    I don't know if the id is visible in the ui. I used the /api/projects endpoint which lists all projects. In this case 250.
  4. Create the staticman config:
{
  "onedevBaseUrl": "https://code.onedev.io/api",
  "onedevUsername": "KN4CK3R",
  "onedevPassword": "password or token",
  "rsaPrivateKey": "-----BEGIN RSA PRIVATE KEY---- ... -----END RSA PRIVATE KEY-----",
  "port": 3000
}
  1. Execute a POST request to http://localhost:3000/v3/entry/onedev/kn4ck3r/250/master/comments
    The username is not needed and can be anything. 250 is the project id.

Result:
https://code.onedev.io/projects/test-staticman/pulls/3
grafik

@KN4CK3R
Copy link
Author

KN4CK3R commented Aug 18, 2021

I created a second PR to replace the mailgun-js dependency: #431

@arturhefczyc
Copy link

arturhefczyc commented Aug 18, 2021

Hi,

I merged the PR #431 into my staticman repository: https://1dev.just-4.dev/projects/staticman

Both errors went away but I am not sure if I moved further because now staticman fails with the following error:

Successfully compiled 26 files with Babel (962ms).
Staticman API running on port 8880
(node:561797) UnhandledPromiseRejectionWarning: TypeError: _got.default.extend is not a function
    at new OneDev (/home/kobit/projects/staticman/build/lib/OneDev.js:23:29)
    at _default (/home/kobit/projects/staticman/build/lib/GitServiceFactory.js:22:14)
    at /home/kobit/projects/staticman/build/lib/Staticman.js:60:55
    at new Staticman (/home/kobit/projects/staticman/build/lib/Staticman.js:71:7)
    at _default (/home/kobit/projects/staticman/build/controllers/process.js:141:27)
    at Layer.handle [as handle_request] (/home/kobit/projects/staticman/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/kobit/projects/staticman/node_modules/express/lib/router/route.js:137:13)
    at /home/kobit/projects/staticman/build/server.js:244:14
    at Layer.handle [as handle_request] (/home/kobit/projects/staticman/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/kobit/projects/staticman/node_modules/express/lib/router/route.js:137:13)
    at /home/kobit/projects/staticman/build/server.js:223:14
    at Layer.handle [as handle_request] (/home/kobit/projects/staticman/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/kobit/projects/staticman/node_modules/express/lib/router/route.js:137:13)
    at /home/kobit/projects/staticman/build/server.js:208:14
    at Layer.handle [as handle_request] (/home/kobit/projects/staticman/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/kobit/projects/staticman/node_modules/express/lib/router/route.js:137:13)
    at module.exports.<anonymous> (/home/kobit/projects/staticman/node_modules/express-brute/index.js:142:36)
    at module.exports.MemoryStore.set (/home/kobit/projects/staticman/node_modules/express-brute/lib/MemoryStore.js:28:35)
    at module.exports.<anonymous> (/home/kobit/projects/staticman/node_modules/express-brute/index.js:127:17)
    at module.exports.MemoryStore.get (/home/kobit/projects/staticman/node_modules/express-brute/lib/MemoryStore.js:38:35)
    at module.exports.<anonymous> (/home/kobit/projects/staticman/node_modules/express-brute/index.js:82:15)
    at keyFunc (/home/kobit/projects/staticman/node_modules/express-brute/index.js:39:41)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:561797) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:561797) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

And I am running the staticman from a command line to test it like this:

NODE_ENV=production npm start

My config/production.json:

{
  "onedevPassword": "onedev-user-token",
  "onedevBaseUrl": "https://1dev.just-4.dev/api",
  "onedevUsername": "kobit",
  "rsaPrivateKey": "-----BEGIN RSA PRIVATE KEY----- key -----END RSA PRIVATE KEY-----",
  "port": 8880
}

I would appreciate if you have any suggestions.

@KN4CK3R
Copy link
Author

KN4CK3R commented Aug 19, 2021

Somehow you are missing got in your dependencies. Run npm i got. After that it worked for me again.
And you are not using the latest commit from this PR.

@arturhefczyc
Copy link

Thank you. With suggestions from your last comment I was able to run it successfully from command line and it all worked.

However, it seems like applying all changes from both PR results in Staticman not working within Docker container.

As Staticman compiles source files at startup time, it needs development dependencies which are not automatically available. It seems that older version of node and modules might have worked differently.

Also, the current version requires that configuration file exists, even if it is empty, otherwise it will not start.

So, this is something the project maintainer has to be aware of, that accepting this PR will break the docker installation.

I managed to get it working with the following Dockerfile and a minor change in packages.json file to put all development dependencies into the production dependencies. It feels more like a workaround rather than a proper solution.

In any case, it is all working for me. If there is anything I need to do to release the bounty for the implementation, please let me know.

@KN4CK3R
Copy link
Author

KN4CK3R commented Aug 20, 2021

Hm, strange. Both PRs do not change how Staticman should be build but I haven't tested that.

In any case, it is all working for me. If there is anything I need to do to release the bounty for the implementation, please let me know.

You just need to close your issue #422. 👍

@KN4CK3R
Copy link
Author

KN4CK3R commented Aug 20, 2021

I see the problem but it's not from my PRs.
The start command was changed in #396. It starts a build now instead of just executing:
ba5b96d#diff-7ae45ad102eab3b6d7e7896acd08c427a9b25b346470d7bc6507b6481575d519R11

@arturhefczyc
Copy link

Ok, issue is closed.

@alexwaibel
Copy link
Collaborator

There's one additional docs change I'd like before we merge this PR. I want to merge both at the same time.
eduardoboucas/staticman.net#28 (review)

I merged the mailgun dependency upgrade, but now there's a merge conflict on this PR that needs resolved as well. Really close to getting this merged, thanks.

@alexwaibel alexwaibel merged commit a644286 into eduardoboucas:dev Sep 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants