New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Payload in array #1974

Closed
guori12321 opened this Issue Mar 24, 2014 · 7 comments

Comments

Projects
None yet
2 participants
@guori12321

guori12321 commented Mar 24, 2014

When I studied the HTTP Post in the Google Flight, I found the Request Payload is like the following:

[,[[,"aa","[,2,"S"]","18563594632275",5]],[,[[,"b_al","no:73"],[,"b_ahr","no:s"],[,"b_am","aa"],[,"b_qu","0"],[,"b_qc","1"]]]]

However, Python does allow None in array, that means, var = [None, 'something'] is right in Python, but var = [ , 'something'] is wrong. What's more, when I post something via lib requests, there is an error :

TypeError: 'NoneType' object is not iterable

So, could you please give me some suggestions of how to deal with the Payload? I searched about it, but in other cases the Payload is a list rather than a array like [, 'something'].

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Mar 25, 2014

Member

Uh, can I see the body as it went over the wire? Because that payload doesn't look like anything I've ever seen before.

Member

Lukasa commented Mar 25, 2014

Uh, can I see the body as it went over the wire? Because that payload doesn't look like anything I've ever seen before.

@guori12321

This comment has been minimized.

Show comment
Hide comment
@guori12321

guori12321 Mar 25, 2014

  1. For instance,
    https://www.google.fr/flights/#search;f=CFE;t=ORY,CDG;d=2014-04-10;r=2014-04-14.
    When you input the city name (Paris in this example), you will get the
    following.
    2.
  2. Request URL:
    https://www.google.fr/flights/rpc
  3. Request Method:
    POST
  4. Status Code:
    200 OK
  5. Request Headers
    1. :host:
      www.google.fr

    2. :method:
      POST

    3. :path:
      /flights/rpc

    4. :scheme:
      https

    5. :version:
      HTTP/1.1

    6. accept:
      /

    7. accept-encoding:
      gzip,deflate,sdch

    8. accept-language:
      en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4

    9. content-length:
      332

    10. content-type:
      application/json; charset=UTF-8

    11. cookie:
      PREF=ID=bdc670c680013c1d:U=19abbcd0a8a83450:LD=en:TM=1394271727:LM=1394271727:S=QmfR-IjoUHauV5zE;
      NID=67=amO5W8p5BQ4fVLAxx-0BDwEn7HfIMxgmu7qBOUmcDxNIO1kSHXy7zTuzJq4lVTrCL3LeReO6Zt3L6FDexx6A-KkhoN1ssm2yhDhJqCupoP1C03GzgyotwrL4kchf8ZyQ4XDIOyCETdEjgbhutwcyoub7vg5P0xIz-ZMFDeEOaTvHS4TosDwnByJPQJP586F0Lab2HGXMuxgqfVXj-Y1L-vJoE0zYgkoNlVj-SCrKpoFS0Rd7mEIHnjBE58_M;
      HSID=AI16cWqyIltev68Uf; SSID=AlsAFvEjOeGI2t5dK;
      APISID=xY5RyNYIKLf8WiOV/AyRDjN2kPNIj0LPun;
      SAPISID=ciFaHzHJIXPA1jk3/AxCRCO5Y6MZJZWYgC;
      SID=DQAAAGIBAABrDfdaH1R_Yn793rLBvpfcnAKmXxfQrx4-NCpxKVau462gHkkEk-O9efQjJ5it9p5brTq9JvvbmLycn_VQeXz3vYlhCUkSAcE6S-CtnIW9IwSYVG8ymnj14JR4YDfT3E-JgY3sZAyZII4AqD3kAkb0t1SNp75wg_m36xFOayDD3D048PCcd6ziA3XUN7G-9H9cY-TqvK2NAyKxT83ab9pnZ7d_ke1nzbWpmTLq5MCmfE5eSfj31HXHEC_N7djFEeiTojdZG0WZmBgmGU38lYzo2Zsqe_hqQ_lZY5Q_FXTLsAxJBzSd-EiOP1OGGGEpceTOURtJfoAhTz5dl_c-eOVRQoZNrBoJLgY77sgKJQ52060A1vzH-oxmfRKaJFlDCmkl7Oqtlu6VxTbAcq8LaK67FIEftgfAVdg2dQX9o7s6RglTqGdVT31bCr53sSQ3bmlfHjGgIeHS3Lu3bJ3h99UESxi-nM8xnSad9mNWvEnccw;
      S=travel-flights=7TxbQr450yFEt8Fvj_9-mA

    12. origin:
      https://www.google.fr

    13. referer:
      https://www.google.fr/flights/

    14. user-agent:
      Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36
      (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36

    15. x-client-data:
      CPm1yQEIlLbJAQiitskBCKm2yQEIxLbJAQi5iMoBCNiIygE=

    16. x-gwt-cctoken:

      ADS25WNRLxnpbQVKxuqSMqiVqxn6s93mt3519cz4gVBbRfMoGZw2GbNb4S87PRNL65-ea7L6gTpcQOZcUjVw9flmw9tBfHmhvCreoXDJoP-2pZZGcp66BFFbMFtQArEPlRAfKzC-wcjQjPA02FnPR7sXYuPhgI4WqyBwtVvMlxQ8kOUozKZMMqfyNWGS4ev15eKrhA

    17. x-gwt-module-base:
      https://www.google.fr/flights/static/

    18. x-gwt-permutation:
      D847595CBA5616E47BFE440BE5D950B6

    19. Request Payload
      1.
      [,[[,"tb","[,[,[[,["CFE"],["ORY","CDG"],"2014-04-10"],[,["ORY","CDG"],["CFE"],"2014-04-14"]]]]","1846721392154327",11]],[,[[,"b_lr","15:45"],[,"b_al","fs:86"],[,"b_ahr","fs:s"],[,"b_lr","13:207"],[,"b_lr","19:0"],[,"b_lr","13:208"],[,"b_am","tb"],[,"b_pe","4F56A1E74FAE0.AD54F0A.299D"],[,"b_qu","1"],[,"b_qc","1"]]]]

  6. Response Headersview source
    1. alternate-protocol:
      443:quic
    2. cache-control:
      no-cache, no-store, max-age=0, must-revalidate
    3. content-encoding:
      gzip
    4. content-length:
      724
    5. content-type:
      application/json; charset=utf-8
    6. date:
      Tue, 25 Mar 2014 08:27:13 GMT
    7. expires:
      Fri, 01 Jan 1990 00:00:00 GMT
    8. pragma:
      no-cache
    9. server:
      GSE
    10. status:
      200 OK
    11. version:
      HTTP/1.1
    12. x-content-type-options:
      nosniff
    13. x-frame-options:
      SAMEORIGIN
    14. x-xss-protection:
      1; mode=block

2014-03-25 8:12 GMT+01:00 Cory Benfield notifications@github.com:

Uh, can I see the body as it went over the wire? Because that payload
doesn't look like anything I've ever seen before.

Reply to this email directly or view it on GitHubhttps://github.com//issues/1974#issuecomment-38536270
.

Guo Rui
Senior Undergraduate Student
School of Software
Harbin Institute of Technology

guori12321 commented Mar 25, 2014

  1. For instance,
    https://www.google.fr/flights/#search;f=CFE;t=ORY,CDG;d=2014-04-10;r=2014-04-14.
    When you input the city name (Paris in this example), you will get the
    following.
    2.
  2. Request URL:
    https://www.google.fr/flights/rpc
  3. Request Method:
    POST
  4. Status Code:
    200 OK
  5. Request Headers
    1. :host:
      www.google.fr

    2. :method:
      POST

    3. :path:
      /flights/rpc

    4. :scheme:
      https

    5. :version:
      HTTP/1.1

    6. accept:
      /

    7. accept-encoding:
      gzip,deflate,sdch

    8. accept-language:
      en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4

    9. content-length:
      332

    10. content-type:
      application/json; charset=UTF-8

    11. cookie:
      PREF=ID=bdc670c680013c1d:U=19abbcd0a8a83450:LD=en:TM=1394271727:LM=1394271727:S=QmfR-IjoUHauV5zE;
      NID=67=amO5W8p5BQ4fVLAxx-0BDwEn7HfIMxgmu7qBOUmcDxNIO1kSHXy7zTuzJq4lVTrCL3LeReO6Zt3L6FDexx6A-KkhoN1ssm2yhDhJqCupoP1C03GzgyotwrL4kchf8ZyQ4XDIOyCETdEjgbhutwcyoub7vg5P0xIz-ZMFDeEOaTvHS4TosDwnByJPQJP586F0Lab2HGXMuxgqfVXj-Y1L-vJoE0zYgkoNlVj-SCrKpoFS0Rd7mEIHnjBE58_M;
      HSID=AI16cWqyIltev68Uf; SSID=AlsAFvEjOeGI2t5dK;
      APISID=xY5RyNYIKLf8WiOV/AyRDjN2kPNIj0LPun;
      SAPISID=ciFaHzHJIXPA1jk3/AxCRCO5Y6MZJZWYgC;
      SID=DQAAAGIBAABrDfdaH1R_Yn793rLBvpfcnAKmXxfQrx4-NCpxKVau462gHkkEk-O9efQjJ5it9p5brTq9JvvbmLycn_VQeXz3vYlhCUkSAcE6S-CtnIW9IwSYVG8ymnj14JR4YDfT3E-JgY3sZAyZII4AqD3kAkb0t1SNp75wg_m36xFOayDD3D048PCcd6ziA3XUN7G-9H9cY-TqvK2NAyKxT83ab9pnZ7d_ke1nzbWpmTLq5MCmfE5eSfj31HXHEC_N7djFEeiTojdZG0WZmBgmGU38lYzo2Zsqe_hqQ_lZY5Q_FXTLsAxJBzSd-EiOP1OGGGEpceTOURtJfoAhTz5dl_c-eOVRQoZNrBoJLgY77sgKJQ52060A1vzH-oxmfRKaJFlDCmkl7Oqtlu6VxTbAcq8LaK67FIEftgfAVdg2dQX9o7s6RglTqGdVT31bCr53sSQ3bmlfHjGgIeHS3Lu3bJ3h99UESxi-nM8xnSad9mNWvEnccw;
      S=travel-flights=7TxbQr450yFEt8Fvj_9-mA

    12. origin:
      https://www.google.fr

    13. referer:
      https://www.google.fr/flights/

    14. user-agent:
      Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36
      (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36

    15. x-client-data:
      CPm1yQEIlLbJAQiitskBCKm2yQEIxLbJAQi5iMoBCNiIygE=

    16. x-gwt-cctoken:

      ADS25WNRLxnpbQVKxuqSMqiVqxn6s93mt3519cz4gVBbRfMoGZw2GbNb4S87PRNL65-ea7L6gTpcQOZcUjVw9flmw9tBfHmhvCreoXDJoP-2pZZGcp66BFFbMFtQArEPlRAfKzC-wcjQjPA02FnPR7sXYuPhgI4WqyBwtVvMlxQ8kOUozKZMMqfyNWGS4ev15eKrhA

    17. x-gwt-module-base:
      https://www.google.fr/flights/static/

    18. x-gwt-permutation:
      D847595CBA5616E47BFE440BE5D950B6

    19. Request Payload
      1.
      [,[[,"tb","[,[,[[,["CFE"],["ORY","CDG"],"2014-04-10"],[,["ORY","CDG"],["CFE"],"2014-04-14"]]]]","1846721392154327",11]],[,[[,"b_lr","15:45"],[,"b_al","fs:86"],[,"b_ahr","fs:s"],[,"b_lr","13:207"],[,"b_lr","19:0"],[,"b_lr","13:208"],[,"b_am","tb"],[,"b_pe","4F56A1E74FAE0.AD54F0A.299D"],[,"b_qu","1"],[,"b_qc","1"]]]]

  6. Response Headersview source
    1. alternate-protocol:
      443:quic
    2. cache-control:
      no-cache, no-store, max-age=0, must-revalidate
    3. content-encoding:
      gzip
    4. content-length:
      724
    5. content-type:
      application/json; charset=utf-8
    6. date:
      Tue, 25 Mar 2014 08:27:13 GMT
    7. expires:
      Fri, 01 Jan 1990 00:00:00 GMT
    8. pragma:
      no-cache
    9. server:
      GSE
    10. status:
      200 OK
    11. version:
      HTTP/1.1
    12. x-content-type-options:
      nosniff
    13. x-frame-options:
      SAMEORIGIN
    14. x-xss-protection:
      1; mode=block

2014-03-25 8:12 GMT+01:00 Cory Benfield notifications@github.com:

Uh, can I see the body as it went over the wire? Because that payload
doesn't look like anything I've ever seen before.

Reply to this email directly or view it on GitHubhttps://github.com//issues/1974#issuecomment-38536270
.

Guo Rui
Senior Undergraduate Student
School of Software
Harbin Institute of Technology

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Mar 25, 2014

Member

So, that request body appears to claim to be JSON, though I'm sure it's not valid JSON as Python fails to parse it. This is not really anything Requests can help you with: you'll need to create that string yourself. You won't even be able to use Python's JSON module to build it as Python has this silly idea that JSON actually needs to match the JSON spec.

Sorry we can't be more helpful, but I strongly suggest you write a very strongly worded letter to Google.fr to tell them to sort their mess out.

Member

Lukasa commented Mar 25, 2014

So, that request body appears to claim to be JSON, though I'm sure it's not valid JSON as Python fails to parse it. This is not really anything Requests can help you with: you'll need to create that string yourself. You won't even be able to use Python's JSON module to build it as Python has this silly idea that JSON actually needs to match the JSON spec.

Sorry we can't be more helpful, but I strongly suggest you write a very strongly worded letter to Google.fr to tell them to sort their mess out.

@Lukasa Lukasa closed this Mar 25, 2014

@guori12321

This comment has been minimized.

Show comment
Hide comment
@guori12321

guori12321 Mar 25, 2014

I'm not sure how to create a string to do those things...I'm new to Web
tech...Can you show me some examples in such cases? I searched the
documents and didn't find what I want, so I open an issue in Github...

2014-03-25 10:11 GMT+01:00 Cory Benfield notifications@github.com:

So, that request body appears to claim to be JSON, though I'm sure it's
not valid JSON as Python fails to parse it. This is not really anything
Requests can help you with: you'll need to create that string yourself. You
won't even be able to use Python's JSON module to build it as Python has
this silly idea that JSON actually needs to match the JSON spec.

Sorry we can't be more helpful, but I strongly suggest you write a very
strongly worded letter to Google.fr to tell them to sort their mess out.

Reply to this email directly or view it on GitHubhttps://github.com//issues/1974#issuecomment-38543308
.

Guo Rui
Senior Undergraduate Student
School of Software
Harbin Institute of Technology

guori12321 commented Mar 25, 2014

I'm not sure how to create a string to do those things...I'm new to Web
tech...Can you show me some examples in such cases? I searched the
documents and didn't find what I want, so I open an issue in Github...

2014-03-25 10:11 GMT+01:00 Cory Benfield notifications@github.com:

So, that request body appears to claim to be JSON, though I'm sure it's
not valid JSON as Python fails to parse it. This is not really anything
Requests can help you with: you'll need to create that string yourself. You
won't even be able to use Python's JSON module to build it as Python has
this silly idea that JSON actually needs to match the JSON spec.

Sorry we can't be more helpful, but I strongly suggest you write a very
strongly worded letter to Google.fr to tell them to sort their mess out.

Reply to this email directly or view it on GitHubhttps://github.com//issues/1974#issuecomment-38543308
.

Guo Rui
Senior Undergraduate Student
School of Software
Harbin Institute of Technology

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Mar 25, 2014

Member

Unfortunately, there's very limited help I can provide without simply writing your code for you. I'm sorry that this was your first foray into web technologies because Google have screwed you. They've invented their own crazy system that doesn't interoperate with anything. You'll literally need to build a plain string with no particular structure and put your own values into it.

Member

Lukasa commented Mar 25, 2014

Unfortunately, there's very limited help I can provide without simply writing your code for you. I'm sorry that this was your first foray into web technologies because Google have screwed you. They've invented their own crazy system that doesn't interoperate with anything. You'll literally need to build a plain string with no particular structure and put your own values into it.

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Mar 25, 2014

Member

For anyone who ends up looking at this mess in the future, I wrote an open letter.

Member

Lukasa commented Mar 25, 2014

For anyone who ends up looking at this mess in the future, I wrote an open letter.

@guori12321

This comment has been minimized.

Show comment
Hide comment
@guori12321

guori12321 Mar 25, 2014

I got it...Perhaps I can try nodeJS to develop the crawler later. Thanks
again!

2014-03-25 10:32 GMT+01:00 Cory Benfield notifications@github.com:

For anyone who ends up looking at this mess in the future, I wrote an
open letter https://gist.github.com/Lukasa/9758031.

Reply to this email directly or view it on GitHubhttps://github.com//issues/1974#issuecomment-38544888
.

Guo Rui
Senior Undergraduate Student
School of Software
Harbin Institute of Technology

guori12321 commented Mar 25, 2014

I got it...Perhaps I can try nodeJS to develop the crawler later. Thanks
again!

2014-03-25 10:32 GMT+01:00 Cory Benfield notifications@github.com:

For anyone who ends up looking at this mess in the future, I wrote an
open letter https://gist.github.com/Lukasa/9758031.

Reply to this email directly or view it on GitHubhttps://github.com//issues/1974#issuecomment-38544888
.

Guo Rui
Senior Undergraduate Student
School of Software
Harbin Institute of Technology

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment