Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

URL shortener #123

Closed
1 of 9 tasks
Logic-gate opened this issue Aug 27, 2017 · 24 comments
Closed
1 of 9 tasks

URL shortener #123

Logic-gate opened this issue Aug 27, 2017 · 24 comments

Comments

@Logic-gate
Copy link

Logic-gate commented Aug 27, 2017

Issue Label:

  • Bug
  • New feature
  • Enhancement
  • New component

Description:
I have created a url shortener at ki.tc. I made it with CLI in mind.
Example Usage:

Create a link:

 curl -i -H "Content-Type: application/json" -X POST -d '{"url": "http://google.com"}' http://ki.tc/

Response:

{
  "url_short": {
    "_id": "b24a138a7f1bce6c493bc", 
    "admin_link": "http://ki.tc/url_shortner/b24a138a7f1bce6c493bc", 
    "link": "http://ki.tc/d93da", 
    "time": "Sun, 27 Aug 2017 18:35:16 GMT", 
    "url": "http://google.com"
  }
}

Admin Access:

curl -i -H "Content-Type: application/json" -X GET http://ki.tc/url_shortner/b24a138a7f1bce6c493bc

Response:

{
  "_id": "b24a138a7f1bce6c493bc", 
  "url_build": {
    "_id": "b24a138a7f1bce6c493bc", 
    "admin_link": "http://ki.tc/url_shortner/b24a138a7f1bce6c493bc", 
    "link": "http://ki.tc/d93da", 
    "time": "Sun, 27 Aug 2017 18:35:16 GMT", 
    "url": "http://google.com"
  }, 
  "url_id": 194249668
}

Admin access updated(after someone visits the link). Updated with every Access attempt.

{
  "2017-08-27 18:39:19": {
    "165914": {
      "access_id": "ae3c63d61f35", 
      "access_time": "Sun, 27 Aug 2017 18:39:19 GMT", 
      "ip_address": "X.X.X.X", 
      "user_agent": "Mozilla/5.0 (X11; Linux x86_64)"
    }
  }, 
  "_id": "b24a138a7f1bce6c493bc", 
  "url_build": {
    "_id": "b24a138a7f1bce6c493bc", 
    "admin_link": "http://ki.tc/url_shortner/b24a138a7f1bce6c493bc", 
    "link": "http://ki.tc/d93da", 
    "time": "Sun, 27 Aug 2017 18:35:16 GMT", 
    "url": "http://google.com"
  }, 
  "url_id": 194249668
}


Delete Link(DELETE METHOD | admin link)

curl -i -H "Content-Type: application/json" -X DELETE http://ki.tc/url_shortner/b24a138a7f1bce6c493bc

Response:

{
  "result": true
}


After finishing, I realised that I spelled shortener wrong shortner. Will stick with it. 😢

If its a bug make sure to include this section.

OS and OS version:

  • Mac
  • Linux 32 Bit
  • Linux 64 Bit
  • Windows 32 Bit
  • Windows 64 Bit

OS Version:

@alexanderepstein
Copy link
Owner

I will look into adding this as a feature to the short tool. What Im thinking is I will take in the link as an argument and first try to find the true link behind the shortened link. If these two links are the same I will assume the user actually wanted to shorten the provided link and then it will make this api request and return the shortened link. Hows that sound?

@Logic-gate
Copy link
Author

Hey,

Why not just pass arguments for shortening and expanding.

short -e http://tinyurl.com/jhkjh78 --> to expand
short -s https://codepen.io/khadkamhn/pen/VvxwKL?depth=everything&limit=all&order=popularity&page=10&q=landing&show_forks=false --> to shorten
short --admin http://ki.tc/url_shortner/b24a138a7f1bce6c493bc --> admin access
OR 
short --admin b24a138a7f1bce6c493bc --> admin access by passing _id
short -d http://ki.tc/url_shortner/b24a138a7f1bce6c493bc --> to delete
OR
short -d b24a138a7f1bce6c493bc --> delete by passing _id

Or something to that affect. Because, checking if a link will resolve to its actual destination will not always work. Take youtube https://youtu.be/aStGnWN-i9o this is a shortened link; let's assume I want to shorten it again, if I were to check if it resolves to itself, it will fail. Case in point:

{
  "/short": {
    "destination": "https://www.youtube.com/watch?v=aStGnWN-i9o&feature=youtu.be", 
    "short_url": "https://youtu.be/aStGnWN-i9o"
  }
}

I think this is a better approach, thoughts?

@Logic-gate
Copy link
Author

UPDATE

Since I started the issue, I have made significant changes that improve the overall functionality of ki.tc.

Link creation remains the same. However, admin access now includes fingerprinting from fingerprintjs2.


  "_id": "ab51c4e9ffd3901577973", 
  "ba728db72668": {
    "access_time": "Sun, 10 Sep 2017 10:09:01 GMT", 
    "browser": "chrome", 
    "browser_version": 60, 
    "details": [
      [
        "user_agent ", 
        " XX "
      ], 
      [
        " language ", 
        " XX"
      ], 
      [
        " color_depth ", 
        " XX"
      ], 
      [
        " pixel_ratio ", 
        " XX"
      ], 
      [
        " hardware_concurrency ", 
        " XX "
      ], 
      [
        " resolution ", 
        " XX, XX "
      ], 
      [
        " available_resolution ", 
        " XX,XX "
      ], 
      [
        " timezone_offset ", 
        " XX "
      ], 
      [
        " session_storage ", 
        " XX "
      ], 
      [
        " local_storage ", 
        " XX "
      ], 
      [
        " indexed_db ", 
        " XX "
      ], 
      [
        " open_database ", 
        " XX "
      ], 
      [
        " cpu_class ", 
        " XX "
      ], 
      [
        " navigator_platform ", 
        " XX "
      ], 
      [
        " do_not_track ", 
        " XX "
      ], 
      [
        " regular_plugins ", 
        " XX "
      ], 
      [
        " canvas ", 
        " XX "
      ], 
      [
        " webgl ", 
        " XX "
      ], 
      [
        " adblock ", 
        " XX "
      ], 
      [
        " has_lied_languages ", 
        " XX "
      ], 
      [
        " has_lied_resolution ", 
        " XX "
      ], 
      [
        " has_lied_os ", 
        " XX "
      ], 
      [
        " has_lied_browser ", 
        " XX "
      ], 
      [
        " touch_support ", 
        " XX "
      ], 
      [
        " js_fonts ", 
        " XX "
      ]
    ], 
    "fp": "XX", 
    "ip_address": "XX.XX.XX.XX", 
    "ip_info": {
      "as": "XX", 
      "city": "XX", 
      "country": "XX", 
      "countryCode": "XX", 
      "isp": "XX", 
      "lat": XX, 
      "lon": XX, 
      "org": "XX", 
      "query": "XX", 
      "region": "XX", 
      "regionName": "XX", 
      "status": "XX", 
      "timezone": "XX", 
      "zip": ""
    }, 
    "platform": "linux", 
    "user_agent": "XX"
  }, 
  "url_build": {
    "_id": "ab51c4e9ffd3901577973", 
    "admin_link": "http://ki.tc/url_shortner/ab51c4e9ffd3901577973", 
    "link": "http://ki.tc/18a1f", 
    "time": "Sun, 10 Sep 2017 10:08:49 GMT", 
    "url": "http://www.google.com"
  }, 
  "url_id": 16721477
}

@alexanderepstein
Copy link
Owner

Sorry I havent responded in a little, I have just started school and was busy working on cryptowatch

Good point about using options I did not account for the possibility of a case like that. When this is implemented that is how it will be done.

Can u further explain the use of this admin link? Why would someone want to use this etc

@Logic-gate
Copy link
Author

Hey,

So the admin link is used to track or rather log usage. Again, this is a url shortener with logging capabilities.

Note, it relies heavily on JS; I used the same method t.co uses in twitter, where the page loads, then redirects. The initial loading part takes advantage of fingerprintjs2.

@alexanderepstein
Copy link
Owner

Sorry I haven't really done anything with this I spent a few hours trying to implement and ran into some errors with how the quotations needed to be escaped when the command is used within a script. It may take a while (since school has started up again and my open source work has been harder to keep up with) but I am definitely looking to implement this.

@stale
Copy link

stale bot commented Nov 3, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the Stale label Nov 3, 2017
@alexanderepstein
Copy link
Owner

Sorry not I've done this yet bot I will do this by the weekend

@stale stale bot removed the Stale label Nov 3, 2017
@alexanderepstein
Copy link
Owner

So i got everything here to work but the issue seems to be coming from your api, whenver I click on a shortened url all that ends up happening is it decodes the url and adds it to after the ki.tc and gives me a page which says the url was not found ex.

short -s google.com
=====================================================================
Original URL:  google.com
Shortened URL: http://ki.tc/c697f
=====================================================================

Clicking on the shortened url it turn the website into http://ki.tc/google.com and then Im greeted with a webpage that says URL Not found!

For now I will add this to a dev branch but it will not be added to mainstream until this works, afterwards I will look into adding the admin link

@Logic-gate
Copy link
Author

Hey,
I am looking into the issue; I cant reproduce it.

The curl argument for creating a shortened link is

  curl -i -H "Content-Type: application/json" -X POST -d '{"url": "http://google.com"}' http://ki.tc/

it should, and I tested it multiple times, return a functional shortened link

 HTTP/1.1 201 CREATED
Date: Wed, 08 Nov 2017 19:38:34 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 244
Content-Type: application/json

{
  "url_short": {
    "_id": "sadasdasdasd", 
    "admin_link": "http://ki.tc/url_shortner/asdasdasdasdasdasddasd", 
    "link": "http://ki.tc/fb241", 
    "time": "Wed, 08 Nov 2017 19:38:34 GMT", 
    "url": "http://google.com"
  }
}

Try it http://ki.tc/fb241

I think you might have shortend http://ki.tc/google.com, I'll look into it more this evening.

Cheers,

@alexanderepstein
Copy link
Owner

alexanderepstein commented Nov 8, 2017

I have run my script verbosely and the curl call looks like this

curl -A curl -s -i -H 'Content-Type: application/json' -X POST -d '{"url": "google.com"}' http://ki.tc/

The response looks like this

'HTTP/1.1 201 CREATED
Date: Wed, 08 Nov 2017 20:08:46 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 237
Content-Type: application/json

{
  "url_short": {
    "_id": "58ade014577a9190a6da8", 
    "admin_link": "http://ki.tc/url_shortner/58ade014577a9190a6da8", 
    "link": "http://ki.tc/20434", 
    "time": "Wed, 08 Nov 2017 20:08:46 GMT", 
    "url": "google.com"
  }
}'

Going to the link http://ki.tc/20434 brings me to the page http://ki.tc/google.com

The link you provided to check works, not sure why this issue is occurring when I do it from my script

@Logic-gate
Copy link
Author

Ahaaaa, figured it out.

you're posting google.com as oppose to http://google.com; i never wrote a handler for a striped url. should be a quick fix from my part. My bad...

@alexanderepstein
Copy link
Owner

Ah fair enough, let me know when this is good to go and I will make sure it will be ready to go in my script for the next release 😄

@Logic-gate
Copy link
Author

Hey,

Fixed!!

I opted for adding a url validator(Thanks to your indirect bug report).

    curl -i -H "Content-Type: application/json" -X POST -d '{"url": "asdasdasdasd"}' http://ki.tc/
HTTP/1.1 400 BAD REQUEST
Date: Wed, 08 Nov 2017 22:36:04 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 119
Connection: close
Content-Type: application/json

{
  "error": "malformed url: \"asdasdasdasd\". url must start with a scheme specifier(http://, https://, ftp://...)"

and


curl -i -H "Content-Type: application/json" -X POST -d '{"url": "google.com"}' http://ki.tc/
HTTP/1.1 400 BAD REQUEST
Date: Wed, 08 Nov 2017 22:34:33 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 117
Connection: close
Content-Type: application/json

{
  "error": "malformed url: \"google.com\". url must start with a scheme specifier(http://, https://, ftp://...)"
}

I could have added a regex that simply checks if google.com starts with a scheme, but then I ran into the issue of domains.

Anyway, if this quick fix is something you could work with thumbs-up, if not...let me know..

Cheers,

@alexanderepstein
Copy link
Owner

I implemented shortening the url by checking to see if the user enters http:// or https:// in the beginning and if not I prepend it.

@stale
Copy link

stale bot commented Dec 4, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the Stale label Dec 4, 2017
@vitalkanev
Copy link

And if I add http:// or https://, this will not work

Here is what I did
vitalkanev@vitalkanev-HP-Artful:~$ short -s https://<some some some some url>.xyz
'=====================================================================
Original URL:  https://
Shortened URL: 
====================================================================='
vitalkanev@vitalkanev-HP-Artful:~$ short -s http://example.com
'=====================================================================
Original URL:  http://
Shortened URL: 
====================================================================='
vitalkanev@vitalkanev-HP-Artful:~$ short -v
Version 1.20.0

@stale stale bot removed the Stale label Dec 4, 2017
@alexanderepstein
Copy link
Owner

@vitalkanev can you run one of those as bash -x short -s http://example.com

@vitalkanev
Copy link

HTTPS output
vitalkanev@vitalkanev-HP-Artful:~/clones$ bash -x short -s https://<...>.xyz
+ currentVersion=1.20.0
+ configuredClient=
+ configuredPython=
++ uname
+ [[ Linux != \D\a\r\w\i\n ]]
++ uname
+ [[ Linux != \D\a\r\w\i\n ]]
+ getConfiguredPython
+ command -v python2
+ command -v python
+ configuredPython=python
+ getConfiguredClient
+ command -v curl
+ configuredClient=curl
+ getopts e:s:uvh opt
+ case "$opt" in
+ shorten=true
+ inputURL=https://<...>.xyz
+ getopts e:s:uvh opt
+ [[ '' == \t\r\u\e ]]
+ [[ 2 == 0 ]]
+ [[ 2 == \1 ]]
+ [[ '' == \t\r\u\e ]]
+ [[ true == \t\r\u\e ]]
+ [[ curl != \c\u\r\l ]]
+ checkInternet
+ httpGet github.com
+ shortenURL https://<...>.xyz
++ echo https://<...>.xyz
++ grep -Eo '^[h]ttp[s]?://'
+ newURL=https://
+ [[ https:// == '' ]]
++ curl -A curl -s -i -H 'Content-Type: application/json' -X POST -d '{"url": "https://"}' http://ki.tc/
+ response='HTTP/1.1 400 BAD REQUEST
Date: Tue, 05 Dec 2017 12:21:53 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 115
Connection: close
Content-Type: application/json

{
  "error": "malformed url: \"https://\". url must start with a scheme specifier(http://, https://, ftp://...)"
}'
++ echo HTTP/1.1 400 BAD $'REQUEST\r' Date: Tue, 05 Dec 2017 12:21:53 $'GMT\r' S' Content-Length: $'115\r' Connection: $'close\r' Content-Type: $'application/json\r' $'\r' '{' '"error":' '"malformed' url: '\"https://\".' url must start with a scheme 'specifier(http://,' https://, 'ftp://...)"' '}'
++ tail -n 1
++ grep -Eo 'http://ki.tc/[0-9a-zA-Z]*'
+ returnedURL=
+ printShortenedResults
+ cat
'=====================================================================
Original URL:  https://
Shortened URL: 
====================================================================='
HTTP output
vitalkanev@vitalkanev-HP-Artful:~/clones$ bash -x short -s http://example.com
+ currentVersion=1.20.0
+ configuredClient=
+ configuredPython=
++ uname
+ [[ Linux != \D\a\r\w\i\n ]]
++ uname
+ [[ Linux != \D\a\r\w\i\n ]]
+ getConfiguredPython
+ command -v python2
+ command -v python
+ configuredPython=python
+ getConfiguredClient
+ command -v curl
+ configuredClient=curl
+ getopts e:s:uvh opt
+ case "$opt" in
+ shorten=true
+ inputURL=http://example.com
+ getopts e:s:uvh opt
+ [[ '' == \t\r\u\e ]]
+ [[ 2 == 0 ]]
+ [[ 2 == \1 ]]
+ [[ '' == \t\r\u\e ]]
+ [[ true == \t\r\u\e ]]
+ [[ curl != \c\u\r\l ]]
+ checkInternet
+ httpGet github.com
+ shortenURL http://example.com
++ echo http://example.com
++ grep -Eo '^[h]ttp[s]?://'
+ newURL=http://
+ [[ http:// == '' ]]
++ curl -A curl -s -i -H 'Content-Type: application/json' -X POST -d '{"url": "http://"}' http://ki.tc/
+ response='HTTP/1.1 400 BAD REQUEST
Date: Tue, 05 Dec 2017 12:26:03 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 114
Connection: close
Content-Type: application/json

{
  "error": "malformed url: \"http://\". url must start with a scheme specifier(http://, https://, ftp://...)"
}'
++ echo HTTP/1.1 400 BAD $'REQUEST\r' Date: Tue, 05 Dec 2017 12:26:03 $'GMT\r' S' Content-Length: $'114\r' Connection: $'close\r' Content-Type: $'application/json\r' $'\r' '{' '"error":' '"malformed' url: '\"http://\".' url must start with a scheme 'specifier(http://,' https://, 'ftp://...)"' '}'
++ tail -n 1
++ grep -Eo 'http://ki.tc/[0-9a-zA-Z]*'
+ returnedURL=
+ printShortenedResults
+ cat
=====================================================================
Original URL:  http://
Shortened URL: 
=====================================================================

@Logic-gate
Copy link
Author

Hey,

It seems that for some reason it's passing an empty url

curl -A curl -s -i -H 'Content-Type: application/json' -X POST -d '{"url": "http://"}' http://ki.tc/

if I am not mistaken, short -s example.com should work. I am assuming that the regex only checks if there is no scheme specifier,

if [[ $newURL == "" ]]; then newURL="http://"$1; fi

and if it actually finds one, it assigns only the specifier
response=$(curl -A curl -s -i -H "Content-Type: application/json" -X POST -d "{\"url\": \"$newURL\"}" http://ki.tc/ || exit 1)

I am bad at bash, so I cant do it Adding a condition for when there is a specifier should fix the issues.

Also, this whole thing was due to the addition of a validation function from my part on ki.tc. It only accepts correct scheme specifiers.

@stale
Copy link

stale bot commented Dec 30, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the Stale label Dec 30, 2017
@stale
Copy link

stale bot commented Jan 4, 2018

This issue has been automatically closed because it has been inactive for 30 days.

@stale stale bot closed this as completed Jan 4, 2018
@alexanderepstein
Copy link
Owner

I fixed the url shortener, I am having success expanding urls such as tiny urls but I cannot expand urls that were shortened by your api it just returns a destination identical to the source

@Logic-gate
Copy link
Author

Hey, the shortening url feature relies heavily on js, in fact it's dependent on it. After all it is a fingerprinting api. The url expansion api is a simple redirect follower; it wont follow redirects from the shortening api because there won't be any redirects unless js is detected.

It's a shortcoming from my part, not adding non-js redirects for the shortening api.

I will amend it however; in the meantime, curl any shortened url from ki.tc, you'll see the js dump.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants